Abstract: Usually modular inverse operation becomes bottlenecked in realizing the public key cryptosystem. The method commonly used leads to a lot of division operation. The modular inverse operation by extended Euclidean algorithm and the binary extended Euclidean algorithm are analyzed, and the improved modular inverse operation by binary extended Euclidean algorithm is presented in this paper. The result shows that the new algorithm runs faster than the old one under the given testing environment. Furthermore it has much more feasibility in hardware realization.