Before text is sent to the LLM for generation, it is tokenized. Tokens are how the model sees the input — single characters, words, parts of words, or other segments of text or code. Each model does this step differently. For example, GPT models use Byte Pair Encoding (BPE).
I am not yet sure how this post will end up helping me... but I found it really interesting. Nice breakdown, Matt.