Characters are stable: every tool counts the same ones. Words are spans between whitespace, a convenience for people.
Tokens are the pieces a model consumes. Their boundaries come from a fixed table and vary by model, so measure them.
Text | Characters | Words | Fixed 3-wide chunks |
|---|---|---|---|
| 11 | 2 | 4 (3 + 3 + 3 + 2) |
| 5 | 1 | 2 (3 + 2) |
empty string | 0 | 0 | 0 |
The calc block counts hello world by hand. The program below checks the same arithmetic on fixed strings.
text: "hello world"
characters = 5 + 1 + 5 = 11
words = 2 ("hello", "world")
3-wide chunks = ceil(11 / 3) = 4, pieces 3 + 3 + 3 + 2 = 11
check: 4 chunks cover 11 characters with no remainder lostCost and fit are counted in tokens, not words or characters. A word budget can miss the token budget by a wide margin on code.
Chunking here is illustrative only. Real splitters use learned pieces, and the rate varies by model, so measure it.
import sys
TEXTS = ["hello world", "assign wdata = 8'hff;", "token"]
CHUNK = 3
def count_words(s):
return len(s.split())
def count_chunks(n, size):
return (n + size - 1) // size
bad = 0
for s in TEXTS:
n_chars = len(s)
n_words = count_words(s)
n_chunks = count_chunks(n_chars, CHUNK)
if n_chunks * CHUNK < n_chars:
bad += 1
if n_chunks > 0 and (n_chunks - 1) * CHUNK >= n_chars:
bad += 1
if count_words("hello world") != 2:
bad += 1
if count_words("") != 0:
bad += 1
if count_chunks(11, 3) != 4:
bad += 1
if len("hello world") != 11:
bad += 1
if not (count_chunks(len("assign wdata = 8'hff;"), 3) > count_words("assign wdata = 8'hff;")):
bad += 1
if bad != 0:
print("FAIL: " + str(bad) + " checks wrong")
sys.exit(1)
print("PASS: 3 texts counted with chunk size 3; hello world is 11 chars and 2 words")hello world has 2 words. How many 3-wide chunks does it fill, and why is neither number a token count?4 chunks (3 + 3 + 3 + 2). Words and fixed chunks are human rules, while tokens follow a learned table that varies by model.
assign wdata = 8'hff;. What does that comparison prove?It proves the units differ on code input: 7 three-wide chunks against 4 whitespace words for that fixed string.