Abstract: In recent years, object recognition modeling techniques based on deep learning face new challenges such as insufficient annotated samples, low interpretability of models, and insufficient stability. All these challenges limit the possibility of deep learning to solve more complex and abstract problems. Constructing intelligent model jointly driven by knowledge and data is an important way to break through the existing bottleneck. This paper presents a classification standard of model constructing methods according to the way of introduction of external experience and cognitive knowledge during model constructing, including modeling methods based on explicit knowledge, modeling methods based on implicit knowledge and modeling methods based on fusion knowledge. Then, following the proposed classification standard, the explorations in each class of methods about solving the problems of few samples and model interpretability are reviewed. Subsequently, taking advantage of different model constructing methods, a future model constructing method jointly driven by knowledge and data is proposed. The proposed method can effectively solve the problem of model construction under the condition of few samples and improve the interpretability of the model by decoupling knowledge modeling and data modeling and taking unsupervised and weak supervised training as the core training patterns. Finally, some research issues which need further study as well as future research directions are drawn in conclusion for promote the object recognition model constructing.