The alignment problem for artificial superintelligence presents difficult challenges to our mathematical and philosophical understanding of agentic systems. In this talk I will give a brief overview of what I see as the core difficulties, and the core requirements for the theoretical framework that we need to address the difficulties. These requirements lead to an emerging theory that I call "computational agent theory". As a taste of this emerging theory I will discuss compositional learning: an idea that aims to bridge the gap between machine learning theory and the empirical abilities of deep learning. This idea already produced novel algorithms for deterministic sequence prediction, inspired by stringology and combinatorics on words.