Abstract: We present a novel on-line algorithm for target segmentation and tracking in video. Superpixels, which are abstracted in every frame, are treated as data points in this paper. The object in current frame is represented as sparse linear combination of dictionary templates, which are generated based on the segmentation result in the previous frame. Then the algorithm capitalizes on the inherent low-rank structure of representation that are learned jointly. A low-rank sparse matrix optimal solution results in the construction of the trimap. At last, a simple energy minimization solution is adopted in segmented stage, leading to a binary pixel-wise segmentation. Experiments demonstrate that our approach is effective.