Abstract: Aiming at the problem that Mean Shift tracking algorithm cannot track well in complicated background, a Mean Shift algorithm based on fuzzy background weighting is proposed. It introduces a fuzzy membership function based on difference, which makes use of the difference between target model and background model in order to represents each pixel contribution to target exact description, and improves target description accuracy. At the same time, the original scale increment and decrement method is improved by using background information for adapting to target scale changing. Experimental results show that the proposed algorithm solves the problem of small-scale wandering and tracking hysteresis of the scale increment and decrement method to a certain extent, and improves the robustness of Mean Shift algorithm under complex background disturbances.