Human activity recognition is the important basis of video surveillance system. In this paper, a new activity recognition method is proposed based on the high-level codebook. We construct a code-word graph based on the mutual information of lowlevel code-words, and then partition the graph into different groups, which discover the high-level code-words patterns. Experimental result shows that the proposed method can effective recognize human activities.