Before text is sent to the LLM for generation, it is tokenized. Tokens are how the model sees the input — single characters, words, parts of words, or other segments of text or code. Each model does this step differently. For example, GPT models use Byte Pair Encoding (BPE).
The Problems with Tokenization in LLMs
I am not yet sure how this post will end up helping me... but I found it really interesting. Nice breakdown, Matt.