Level 5 Checkpoint — Prove Causal Attention

Complete this checkpoint after L5.8 — Causal Masking. You should be able to trace one query row without relying on a high-level attention API.

1. Roles

For one query and three positions, state which vectors:

• produce compatibility scores;
• carry the information that is ultimately mixed.

Explain why swapping key and value roles is a semantic bug even when their shapes happen to match.

2. Dot products and scaling

Choose a 2D query and three 2D keys. Compute the three dot products by hand and divide each by sqrt(2). Explain what scaling changes and what it does not change.

3. Mask before normalization

For a length-4 sequence, draw the legal keys for query positions 0 through 3. At position 2, key 3 must be illegal.

Explain why setting its logit to negative infinity before softmax produces probability zero while still normalizing the legal positions.

4. Weighted values

Use a simple legal weight row such as [0.7, 0.3, 0.0, 0.0] and two-dimensional values. Compute the weighted mixture and explain how the answer changes when the first two weights trade places.

5. Executable causal evidence

Run the Lab below. Change one future-mask entry and measure future attention mass.

Loading lab…

Quick Check

1. Which vectors create attention scores?

Values and token stringsQueries and keysKeys and losses

2. Why divide by `sqrt(head_dim)`?

To change sequence lengthTo keep dot-product magnitudes in a useful softmax rangeTo remove the causal rule

3. What should a legal softmax row sum to?

Head dimensionVocabulary sizeApproximately 1

4. What must always be true in causal decoder attention?

All values are equalEvery raw score is positiveFuture attention probability is zero

0 of 4 questions answered.

Check your reasoning after you try

• Queries and keys create compatibility scores; values carry the information that is mixed.
• Dividing by sqrt(head_dim) changes score scale, not which raw dot product is larger.
• A causal mask must remove future positions before softmax so their normalized probability is zero.
• A row such as [0.7, 0.3, 0, 0] produces a weighted sum of the first two value vectors. Swapping the first two weights changes which value contributes more.
• In the Lab, any positive attention mass on a future token is evidence that the causal boundary failed.

Pass condition

You are ready for L5.9 — Multi-Head Attention when you can explain the complete path query/key → score → scale → mask → softmax → weighted values and diagnose where future leakage could enter.