Abstract: A variable structure dynamic model of gear driving system is established based on backlash hysteresis model and friction aggregation model. The influence of backlash and friction is compensated based on a compound controller formed by radial basis function (RBF) neural network and sliding mode. The switching plus of sliding mode controller can be adjusted by RBF neural network, which can reduce the buffeting of sliding mode control. The feasibility of this method is validated by simulation results.