Tokenizer in GenAI Models

What is a Tokenizer?

Think of a tokenizer as a text-to-numbers converter.

LLMs can’t read plain human language like “Hello, how are you?”. They only understand numbers. So before anything else, the text must be broken down into smaller parts, and then converted into numerical IDs. That’s exactly what a tokenizer does.

Example:

Input sentence:

"playing football"

A tokenizer might break this into:

['play', 'ing', 'football']

Then it converts each part into a unique ID:

[1234, 567, 7890] (just sample IDs)

These numbers are what get passed into the model.

📘 What is Vocabulary?

A vocabulary is the list of all tokens the model knows.

Think of it as a dictionary. Each word (or subword or character) the tokenizer can recognize is stored in this vocabulary, and each one has a unique ID.

• In GPT-3, the vocabulary size is about 50,000 tokens.
• It includes whole words, parts of words (like “un”, “ing”), symbols, and even emojis.

If a word isn’t in the vocabulary, the tokenizer breaks it down into smaller known parts.

🔗 Tokenizer vs. Embedding – What’s the Difference?

This is a common point of confusion, so here’s a side-by-side comparison to make it crystal clear:

In simple terms:

• The tokenizer gets the data ready.
• The embedding gives each token a meaning-rich numerical representation.
• This vector is what the neural network understands and works with during training or inference.

Tokenizer ID Comparison with tokenizer.json

import json
import requests
from transformers import AutoTokenizer

# Step 1: Load tokenizer from Hugging Face
model_name = "distilbert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Step 2: Tokenize a sample sentence
text = "Playing football is fun!"
encoded = tokenizer(text)

print("📌 Input Text:", text)
print("🔤 Tokens:", tokenizer.convert_ids_to_tokens(encoded['input_ids']))
print("🔢 Token IDs:", encoded['input_ids'])

# Step 3: Download tokenizer.json from Hugging Face model repository
tokenizer_json_url = f"https://huggingface.co/{model_name}/resolve/main/tokenizer.json"
response = requests.get(tokenizer_json_url)
tokenizer_json = response.json()

# Step 4: Extract vocabulary from tokenizer.json
vocab_from_json = tokenizer_json["model"]["vocab"]

# Step 5: Compare token IDs manually
print("\n🔍 Token ID Mapping from tokenizer.json:")
for token in tokenizer.convert_ids_to_tokens(encoded['input_ids']):
    token_id = vocab_from_json.get(token, "Not found")
    print(f"  Token: '{token}' → ID: {token_id}")