Padding, Truncation, and Masks

Goal

Turn unequal token sequences into a rectangular batch, build a validity mask, and explain exactly which positions are real text, filler, or discarded by truncation.

Take a real encoded sequence [2, 8, 3] and a target length of 5:

IDs:  [2, 8, 3, 0, 0]
mask: [1, 1, 1, 0, 0]

The shape is now convenient for batching, but positions 3 and 4 did not become text. They are filler. The mask preserves that boundary for later computation.

Truncation is different: if seven real tokens must fit in five slots, two real positions are deliberately removed. Padding is shape management; truncation is information loss.

Batches need a rectangular shape

Neural-network batches are usually stored as rectangular tensors. But text sequences have different lengths:

A: [5, 8, 2]
B: [7, 4, 9, 3, 6]

To place them in one (B,T) tensor, the shorter row can be padded:

A: [5, 8, 2, 0, 0]
B: [7, 4, 9, 3, 6]

If token ID 0 is the padding token, the model still needs a separate signal telling it that the final two positions in A are not real content.

An attention mask can encode that distinction:

A mask: [1, 1, 1, 0, 0]
B mask: [1, 1, 1, 1, 1]

The exact mask convention can vary by library, so never infer semantics from the numbers alone. Check the API contract.

Padding and truncation solve different problems

Padding extends a shorter sequence up to a chosen batch length.

Truncation removes tokens when a sequence is longer than the allowed length.

Those operations are not interchangeable.

If a context limit is 5 and you receive 8 tokens, no amount of padding helps; you must choose which 5-token region to keep or use another long-context strategy.

That choice can remove important information. For a document classifier, truncating the end may remove the conclusion. For a chat model, truncating the wrong side may remove the most recent user request.

Mask bugs can look numerically valid

A padding mask with the wrong polarity can still have the correct shape.

For example, [1,1,1,0,0] and [0,0,0,1,1] are both length 5. Only one matches the intended convention.

That is why shape tests are necessary but not sufficient. A useful behavioral test changes only padded positions and checks that the output for real tokens does not change when those padded values should be ignored.

Predict

A 7-token sequence is truncated to a maximum of 5 positions. What is the maximum number of real token positions that remain?

5712

Keep IDs and masks side by side

Pad [2, 9, 3] to length 6 by hand, then write the mask. Next, choose a truncation rule for a longer sequence and state which end or region loses information.

Run the starter once, then complete the TODO that pads the kept IDs and builds the aligned validity mask. The short-sequence checks should fail before your change and pass afterward. Then try short, exact-length, and too-long inputs; change only the target length and observe both arrays.

Loading lab…

A useful failure is a padded sequence with a mask of all ones. The tensor shape is valid, but later computation is told that filler positions are real. This is why “the code ran” is weak evidence.

Quick Check

1. What should a validity mask usually contain at a padding position?

1, because padding still occupies a tensor slotThe pad token ID itself, because masks reuse vocabulary IDs0, under this lesson's valid=1/pad=0 convention

2. What is the key difference between padding and truncation?

Padding removes old tokens while truncation adds mask-only slotsPadding adds filler; truncation removes real positionsBoth preserve all original tokens and differ only in mask polarity

3. Why can a correct shape still hide a bug?

Any correct-size mask is semantically correct if its dtype matchesThe mask may mark filler as real or real tokens as fillerA mask bug must change tensor rank before it can affect attention

0 of 3 questions answered.

Transfer the idea

Compare tail truncation, head truncation, and sliding windows for a document where important evidence can appear anywhere. Explain what each strategy might lose.

Key Takeaways

• Padding creates rectangular shapes with non-content filler.
• Masks preserve the boundary between real and padded positions.
• Truncation discards real information by design.
• Debug token IDs and masks together, not as separate artifacts.

Next Lesson

Next, each retained token ID will select a learned embedding vector—the first stage where the model receives learnable numerical features.

References

• Kudo and Richardson, SentencePiece.
• Hugging Face Transformers, Padding and truncation.