Line of sight detection and tracking is a real-time and accurate recording of the movement of the eyeball when people ob- serve scene information, and has a wide application prospect in the fields of psychology, education and so on. Most tracking technology needs to wear eye movement instrument and other professional equipment to achieve with high cost and inconvenient operation. This paper uses face alignment algorithm based on HOG face detection and ERT algorithm regression tree to mark face features, es- tablish mask to separate eyes, detect pupil contour adaptively, and calculate pupil position and line of sight direction. The tracking system reduces the requirement of hardware equipment and promotes the application of line of sight tracking system.