Interpretable Control Systems

This work was done under the guidance and direction of Alice Rigg. Alice is a passionate researcher with a strong intuition for interpretability. I am fortunate to have had the opportunity to learn from her.

related works and background#

This post builds on prior work done by Rigg, et al. (2024), which demonstrated that replacing standard MLP activations with bilinear layers makes a model’s computations expressible in terms of weights.

bilinear layers#

With bilinear layers, every logit expands into a clean quadratic form that we can interpret directly. The hidden step is:

Where $\odot$ is elementwise multiplication. Instead of introducing nonlinearity, a bilinear layer multiplies two different linear projections of the state. Each coordinate of $h$ is in the form

This already encodes a pairwise interaction between weighted sums of the input features.

When these hidden features are combined in the output head

Expanding one logit $z_a(x)$ explicitly:

This can be simplified neatly as

where $B_a$ is the quadratic interaction matrix, $\beta_a$ is the linear term, and $c_a$ is the constant.

interaction matrices#

The interaction matrix $B_a$ tells us how pairs of features $(x_i, x_j)$ work together to influence logit $a$. Only the symmetric portion of $B_a$ contributes to the quadratic form*, so we analyze

With this, we can directly see which pairs of state features the policy uses. $\\$ *See section 3 of the original paper for a detailed explanation.

modes and decomposition#

Because $S_a$ is real and symmetric, the spectral theorem guarantees that it can be decomposed into eigenvalues and eigenvectors:

Where each $v_i$ is an eigenvector and $\lambda_i$ is its corresponding eigenvalue. This makes the quadratic form easy to interpret:

Each term here corresponds to a mode - a pattern of input features.
The eigenvalue $\lambda_i$ tells us how strongly that pattern influences the logit:

• If $\lambda_i > 0$, the mode amplifies the score when present.
• If $\lambda_i < 0$, the mode suppresses it.

Generally, there will only be a few modes with large magnitudes, meaning the policy can often be summarized into a handful of dominant patterns.