We consider a stochastic volatility model where the dynamics of the volatility are described by linear functions of the (time extended) signature of a primary underlying process, which is supposed to be some multidimensional continuous semimartingale. Under the additional assumption that this primary process is of polynomial type, we obtain closed form expressions for the VIX squared, exploiting the fact that the truncated signature of a polynomial process is again a polynomial process. Adding to such a primary process the Brownian motion driving the stock price, allows then to express both the log-price and the VIX squared as linear functions of the signature of the corresponding augmented process. This feature can then be efficiently used for pricing and calibration purposes. Indeed, as the signature samples can be easily precomputed, the calibration task can be split into an offline sampling and a standard optimization. We also propose a Fourier pricing approach for both VIX and SPX options exploiting that the signature of the augmented primary semimartingale is an infinite dimensional affine process. For both the SPX and VIX options we obtain highly accurate calibration results, showing that this model class allows to solve the joint calibration problem without adding jumps or rough volatility.