In the digital era, the problem of data transmission security has become increasingly prominent, and traditional encryption methods have certain limitations. In order to improve the security of data transmission, this paper proposes an encryption and decryption method based on AES algorithm. Firstly, AES algorithm is used to generate data key, which is efficient, secure and widely used. During encryption, byte substitution and column are mixed, the former is nonlinear byte substitution through S-box, and the latter is linear transformation of data matrix column to enhance the degree of data confusion. At the same time, the artificial bee colony algorithm is used to optimize the confusion sequence composed of data and key, and give full play to its global search and adaptive ability to improve the encryption security and complexity. Experiments are carried out under different transmission bandwidths. The results show that compared with the existing methods, this method can resist brute force attack more effectively, the number of successful attacks is significantly reduced, and the data transmission security is guaranteed. After applying this method, the encryption and decryption time is shorter and the memory usage is reduced.