Random quantum circuit holds a particular interest due to the non-trivial growth in complexity of the state with the increase in circuit depth. Various metrics have been proposed to characterize this quantum complexity, such as conventional entanglement entropy and linear cross-entropy benchmarks (XEB). As both of these methods require formidable amounts of quantum data or even large-scale classical simulations for estimation, the need for an efficient alternative approach arises to detect complexity given a limited amount of data. We turn to the notion of K-design, which invites one to look into higher moments of the bit string distribution. While the rigorous estimation of the moments would also require large amounts of data, it ought to be possible to learn partial information relevant for higher moments, in principle. Using the self-attention mechanism to learn aspects of higher moments, we design and train quantum attention networks (QAN) to learn increasing complexity. We show that QAN can learn past anti-concentration on simulated data. Application of QAN on experimental data shows how QAN can learn in the presence of real noise. Our results show an exciting prospect of using QAN to learn quantum complexity beyond the limits of manually specified functions.