Tokenization

The rule-driven conversion of text into the token sequence consumed by a model or feature pipeline.