Abstract: Computer vision algorithms for individual tasks such as object recognition, detection and segmentation have shown impressive results in the recent years. The next challenge is to integrate all these algorithms and address the problem of scene understanding. A new higher order conditional random field (CRF) model is proposed to get semantic segmentation and object detection simultaneously. Specifically, the proposed higher order CRF model consists of low-order potentials and improved detector potentials. To avoid wrong recognition caused by the confidence given by the initial detector, the first-and-second-order pooling and logistic regression are adopted to improve the detector potential. Experimental results show that the proposed model achieves significant improvement over the baseline methods on MSRC 21-class and PASCAL VOC 2007 datasets.