Tokens
The basic units that AI models use to process text, roughly corresponding to word fragments or characters.
What Tokens Actually Are
AI models don't read words like humans do. They break text into smaller pieces called tokens. A token might be a whole word, part of a word, or punctuation.
Examples:
- "Hello" = 1 token
- "extraordinary" = might be 2-3 tokens
- "AI" = 1 token
- Code often uses more tokens than English prose
Rule of thumb: 1 token is roughly 4 characters or 0.75 words.
Why Tokens Matter
Pricing: Most AI APIs charge per token. More tokens = higher cost.
Context limits: Models have maximum token limits for input + output combined.
Speed: More tokens = slower responses.
Counting Tokens
Before using an AI tool heavily, estimate your token usage:
- 1,000 words ≈ 1,300 tokens
- A typical email ≈ 200-400 tokens
- A code file ≈ varies wildly based on language
Most AI tools show token counts in their interfaces or APIs.
Token Economy Tips
Be concise. Shorter prompts cost less and often work better.
Batch wisely. Sometimes one long request is cheaper than many short ones.
Watch for bloat. AI responses can be verbose. Ask for concise answers when you don't need detail.