Index listing of all 1 items tagged with #tokenization.
Models do not read words; they read integers produced by a tokenizer. What that step does, and why token counts behave differently from character counts.