Tokenizer Evaluation Workshop

Goal

Compare tokenizers under identical evaluation conditions, inspect slice-level failures, and justify a tokenizer choice with reproducible evidence and explicit limitations.

There is no useful answer to “Which tokenizer is best?” without a target text distribution and criteria. A small whole-word tokenizer may be compact on familiar English and terrible on rare names. A byte tokenizer may preserve every UTF-8 string but consume more positions. The comparison becomes meaningful only when both see the same inputs and measurements.

Before running the workshop, write two predictions: which tokenizer will have lower unknown rate, and which will use fewer positions on familiar words.

Build an evaluation scorecard instead of choosing one favorite example

A tokenizer comparison should include several text slices:

ordinary prose
names and numbers
URLs
source code
emoji
multilingual text
domain-specific terms

For each slice, record whether text round-trips correctly, token count, unexpected fallback behavior, and surprising boundaries. One average token count can hide a serious failure in an important slice.

Encode/decode round trips test a basic invariant

If the tokenizer is supposed to preserve text exactly, check:

decode(encode(text)) == text

A failure here is more fundamental than a slightly inefficient segmentation.

After round-trip correctness is established, compare efficiency and domain behavior.

The best tokenizer depends on the intended model and data

A tokenizer with the lowest token count on English prose might perform poorly on code or Korean. State the decision relative to target domains, vocabulary/model constraints, context-window cost, coverage requirements, and reproducibility of the measurement rather than declaring a universal winner.

Predict

What makes a comparison between two tokenizers fair?

Use the same source strings, slices, and metrics for bothGive each only the examples it handles bestCompare vocabulary size and ignore produced sequences

Build a scorecard from evidence

Run the notebook workshop and record, per slice:

• token count or compression behavior;
• unknown fallback where applicable;
• round-trip/coverage evidence;
• the worst concrete example;
• the exact tokenizer/configuration revision.

Loading lab…

Then remove one difficult slice and see how the conclusion changes. This demonstrates a general evaluation lesson: a precise table can still be misleading if its test distribution is incomplete.

Quick Check

1. Why inspect worst-case examples beside averages?

Worst cases alone define performance for every userAverages can hide severe failures concentrated in one sliceAverages already weight rare severe failures enough, so examples are only cosmetic

2. What should remain fixed when comparing candidates?

Evaluation text, slice definitions, and metricsEach tokenizer should use the text domain where it performs bestOnly the metric name; candidate tokenizers may use different evaluation text

3. What makes the result reproducible?

A fixed random seed even if tokenizer files or input text changeThe final average metric without the underlying examplesDeterministic inputs/code plus recorded tokenizer files and metrics

0 of 3 questions answered.

Make a decision, not a universal claim

Choose a tokenizer for a fictional multilingual chat app using at least two measured criteria. Name one limitation the current evaluation does not answer, such as latency, compatibility with an existing model, or a missing domain slice.

Key Takeaways

• Tokenizer selection is an evaluation problem, not a prestige ranking.
• Compare candidates on identical, representative text slices.
• Use aggregate metrics together with concrete failures.
• Record tokenizer files/settings so conclusions are reproducible.
• State tradeoffs and remaining uncertainty explicitly.

Next Lesson

Complete the Tokenizer Workbench Level Project. Level 5 will use these token, mask, embedding, and position representations to construct attention.

References

• Kudo and Richardson, SentencePiece.