AI Engineering

Understanding Tokens in LLMs

by
Tokens in LLMs explained - the word unbelievable split into three tokens: un, believ and able

The hidden currency behind every AI conversation — explained from scratch A beginner-friendly deep dive, with real examples from Ollama. A token is the small chunk of text an LLM actually reads and writes — usually part of a word, not a whole word. As a rough rule, 1 token ≈ 4 characters ≈ three-quarters … Read more

Deep Dive: How Machines “Understand” Text Without Exact Matching (Learning Exercise)

by

I recently completed a Proof of Concept (PoC) to understand the internal mechanics of how machines process language. The goal was to learn how data transforms from human words into mathematical numbers and back. This method is the foundation for modern search, allowing a system to find answers based on meaning rather than just matching … Read more