Archives

Reverse-engineering a one-layer, one-head, attention-only transformer that computes the maximum of a sequence, read straight off the weights: the QK circuit attends to the biggest token, the OV circuit copies it, and both read a single size direction in embedding space. Plus the min variant, which is the same circuit flipped.

A short introduction.