Abstract: Classic R-D optimized algorithms, which use the objective image quality as distortion assessment standard, do not consider the effect produced by human visual system (HVS). This paper proposes an H.264 encode mode decision algorithm based on just-noticeable distortion (JND) model. The R-D values in coding process are calculated by using modified DCT-based JND model, incorporating the spatio-temporal contrast sensitivity function, luminance adaptation, and intra- and inter-band contrast masking. The redundant coefficients which can't be noticeable are removed to optimize the subjective quality. The simulation results show that under same subjective quality, the proposed algorithm can obviously reduce the production of bit-rate and hence improve the coding efficiency.