Abstract: In the digital measurement comparison of periodic signal, there exists a quantization phase step error that depends on the frequency relationship of the two comparison periodical signals, besides the A/D quantization error. The cause and relation of the two quantization errors are analyzed in this paper. And a new method which can reduce quantization error is then presented by utilizing the relationship of two errors, the resolution stability with influence of quantization error, and the border technology in digital processing. The experiments show that measurement resolution can reach higher level using an A/D of 10 bit. Compared with the A/D resolution, the result can improve one magnitude. This method can play an important role in precision measurement and control about frequency, phase and other periodical parameters.