This talk discusses the approximation of two-layer compositions $f(x)=g(\phi(x))$ via deep networks with ReLU activation, where $\phi$ is a geometrically intuitive, dimensionality reducing feature map. We'll focus on two intuitive and practically relevant choices for $\phi$: the projection onto a low-dimensional embedded submanifold and a distance to a collection of low-dimensional sets. This will cover the capacity of neural networks for both regression and classification models. Since $\phi$ encapsulates all nonlinear features that are material to the function $f$, this suggests that deep nets are faithful to an intrinsic dimension governed by $f$ rather than the complexity of the domain / data on which $f$ is defined. In particular, the prevalent model of approximating functions on low-dimensional manifolds can be relaxed to include significant off-manifold noise by using functions of this type, with $\phi$ representing an orthogonal projection onto the same manifold. We'll also discuss connections of this work to manifold autoencoders and data generation. This is joint work with Timo Klock, Rongjie Lai, Stefan Schonsheck, and Scott Mahan, and supported by NSF grants DMS-1819222 and DMS-2012266.