The Convolution Neural Network(CNN) in Deep Learning has a strong anti-jamming ability for image translation, rota- tion and other transformations. Compared with the traditional vehicle recognition technology, it can extract deeper and richer image information. Based on the VGGNet structure and simulating the order of human eyes' perception of vehicle characteristics, this pa- per designs a hierarchical retrieval system for vehicle image database. Firstly, a CNN which can recognize eight kinds of colors is constructed and trained to recognize the color of the target vehicle. Then, SIFT and LBP features are combined to match and retrieve the same color candidate vehicle database. The hierarchical retrieval mode of the system can effectively reduce the scope of retrieval and improve the efficiency of retrieval. The fusion of multi features can also guarantee the extraction of enough image information and ensure the accuracy of retrieval.