Natural language processing

What it is and why it exists

Natural language processing (NLP) studies how computers process human language. An engineering task must turn an ambiguous language problem into explicit inputs, outputs, and scoring rules. NLP doesn’t promise that a program understands text like a person; it requires verifiable behavior on a defined task.

Natural language is hard to hand directly to an ordinary program. The same meaning has many forms, and a word’s meaning changes with context; spelling, punctuation, mixed languages, and domain terms expand the input space further. An NLP pipeline compresses that variation into structures a model can process through normalization, tokenization, representation learning, or feature extraction.

You meet NLP in search, spam detection, support routing, sentiment classification, named entity recognition, machine translation, and summarization. Large language models (LLMs) are NLP systems, but NLP isn’t limited to generative models. A rule-based entity extractor, TF-IDF classifier, or small encoder may be the right production system.

The task determines the output

Common tasks differ by output shape. Text classification selects labels for a complete text; sequence labeling assigns a label to each token; span extraction returns start and end positions in the source; retrieval ranks candidate documents; generation produces a new token sequence. Choose the shape before deciding how to annotate data, design the model interface, or score results.

A classifier answers “which queue owns this ticket,” not “what does it mean.” A named entity recognizer can mark a company name without resolving that company’s relationship to an order. Compressing a product requirement into “understand user messages” leaves both the training data and acceptance criteria without boundaries.

Language data is more than a string

Text has visible content, an encoding, a normalization form, and boundaries. What a user sees as one character may contain several Unicode code points; one model token may be only part of a word. Character offsets, byte offsets, UTF-16 code-unit offsets, and token positions are different coordinate systems.

That difference affects annotation and interfaces. A span predicted against normalized text cannot be used to slice the source text directly. If a browser, Python service, and model tokenizer use different offset units, the predicted words may be right while the highlights appear in the wrong place.

How it works

A maintainable NLP system is a chain of contracted transformations, and it starts before the model call. Each step states its input representation, output representation, information loss, and the route back to source data.

1. Write the task contract

The task contract states at least the input unit, allowed outputs, abstention behavior, and success metrics. Classification needs a fixed label set and multi-label rules; span tasks need inclusive or exclusive end-position semantics; generation needs maximum length, citation requirements, and refusal conditions. If labels overlap, revise the annotation guide first. A different model rarely repairs an incoherent target.

Define business costs at the same time. Misrouting a refund request to general support has a different cost from routing it to a security queue, so one accuracy number may be inadequate. Release thresholds and human review paths belong in the task contract, not hidden inside model code.

2. Preserve source text and derive normalized views

The source text supports audits, display, and span mapping, so store it unchanged. Search or matching can use a derived view with Unicode normalization, case folding, or controlled whitespace processing. Each transformation can change content or length; record its order and version.

More normalization isn’t automatically better. Removing punctuation erases question marks, emoji, and sentence boundaries; deleting stop words can make “not approved” resemble “approved”; lowercasing removes signals from proper names. Validate each step on target data before keeping it.

3. Find boundaries and build representations

Tokenization turns text into a token sequence. Whitespace splitting does not work for languages without explicit word boundaries, and it mishandles contractions, punctuation, and combining characters. Modern models usually use subword or byte-level tokens, so one visible word may become several tokens.

Traditional models convert tokens into counts, n-grams, or TF-IDF features. Neural networks commonly map token ids to embeddings and then create contextual representations. Both routes need the vocabulary or tokenizer used during training. Replacing that component during inference silently changes the model’s input.

4. Train, decode, and evaluate

Training adjusts model parameters against an annotated corpus . Inference first produces scores or probabilities; decoding rules turn those values into the labels, spans, rankings, or text the product needs. Thresholds, label maps, and conflict rules are decoding logic and belong in version control and tests.

Evaluation needs an aggregate view and meaningful data slices. Language, text length, source channel, time period, and minority classes can reveal failures hidden by averages. Start with a simple baseline to verify the data and metrics, then compare more complex models. If a baseline looks implausibly good, inspect duplicates, label leakage, and the split strategy first.

Examples

The next four examples use only Python’s standard library. They move from text views to sparse features, a classification baseline, and per-class evaluation. Their small scale keeps every transformation and number inspectable; it does not imitate a full training platform.

Derive a matching view

NFKC converts the full-width Latin letters and digit to compatibility forms, while casefold() creates a case-insensitive matching view. The program does not overwrite original, so it can still display exactly what the user submitted. A production system should also record the normalization policy in the model or index version.

This regular expression illustrates boundaries; it is not a general tokenizer. It groups consecutive Unicode letters or digits and separates punctuation, but lacks enough rules for Chinese words, emoji sequences, or model subwords. In an application, use a tokenizer designed for the language and the model’s training procedure.

Compute TF-IDF features

Term frequency increases the weight of a word repeated within one text, while inverse document frequency discounts words shared by many documents. The example uses the smoothed form log((1 + N) / (1 + df)) + 1 and normalizes term frequency by document length. The formula and parameters must remain identical to the training pipeline.

import math
import re
from collections import Counter


def tokenize(text: str) -> list[str]:
    return re.findall(r"[^\W_]+", text.casefold())


documents = [
    "refund invoice payment",
    "invoice refund delayed",
    "application crashes after update",
    "update causes login error",
]
document_frequency = Counter(
    token for document in documents for token in set(tokenize(document))
)


def tfidf(text: str) -> dict[str, float]:
    counts = Counter(tokenize(text))
    total = sum(counts.values())
    return {
        token: round(
            count / total
            * (math.log((1 + len(documents)) / (1 + document_frequency[token])) + 1),
            3,
        )
        for token, count in sorted(counts.items())
    }


print(tfidf("refund payment refund"))

{'payment': 0.639, 'refund': 1.007}

refund occurs twice in the text being processed, so its normalized term frequency is higher; payment appears in fewer training documents, so its inverse document frequency is higher. The final weight depends on both parts and cannot be explained by raw occurrence count alone.

A real vectorizer must preserve the training vocabulary, feature order, and IDF values. Refitting IDF on the test set lets the test distribution participate in feature construction and leaks data. The inference path also needs the training policy for ignoring, hashing, or representing out-of-vocabulary words.

Train an inspectable classification baseline

Multinomial naive Bayes is a useful baseline for token counts. Its conditional-independence assumption is strong, but it quickly exposes problems in labels, splits, and features. The example uses add-one smoothing so a word unseen in one class does not turn the whole probability product into zero.

import math
import re
from collections import Counter, defaultdict


def tokenize(text: str) -> list[str]:
    return re.findall(r"[^\W_]+", text.casefold())


training = [
    ("billing", "refund invoice payment"),
    ("billing", "invoice refund delayed"),
    ("technical", "application crashes after update"),
    ("technical", "update causes login error"),
]
document_counts = Counter(label for label, _ in training)
token_counts: dict[str, Counter[str]] = defaultdict(Counter)
vocabulary: set[str] = set()

for label, text in training:
    words = tokenize(text)
    token_counts[label].update(words)
    vocabulary.update(words)


def predict(text: str) -> tuple[str, dict[str, float]]:
    scores: dict[str, float] = {}
    for label in sorted(document_counts):
        score = math.log(document_counts[label] / len(training))
        denominator = sum(token_counts[label].values()) + len(vocabulary)
        for token in tokenize(text):
            score += math.log((token_counts[label][token] + 1) / denominator)
        scores[label] = round(score, 3)
    return max(scores, key=scores.get), scores


for ticket in ["refund still delayed", "application login error"]:
    print(ticket, "->", predict(ticket))

refund still delayed -> ('billing', {'billing': -7.401, 'technical': -9.526})
application login error -> ('technical', {'billing': -9.193, 'technical': -7.447})

Log probabilities replace multiplication of many small probabilities with addition and avoid numerical underflow. Each class has two training documents, so the priors are equal; token likelihoods determine the final labels. The function returns per-class scores so tests can inspect the boundary instead of seeing only a label.

Four training examples do not establish production quality. The baseline provides a complete, deterministic path that is easy to diagnose. A more complex model should beat it reliably on a fixed test set and slices before its added latency and operational cost are justified.

Catch majority-only predictions with per-class metrics

Accuracy can be dominated by the common class. The predictor below labels every item as billing and still gets four items right, but misses every technical case. Macro F1 gives each class equal weight, so it exposes that failure.

from collections import Counter


gold = ["billing", "billing", "billing", "billing", "technical", "technical"]
predicted = ["billing", "billing", "billing", "billing", "billing", "billing"]
labels = sorted(set(gold) | set(predicted))
confusion = Counter(zip(gold, predicted))
f1_scores = []

for label in labels:
    true_positive = confusion[label, label]
    false_positive = sum(confusion[other, label] for other in labels if other != label)
    false_negative = sum(confusion[label, other] for other in labels if other != label)
    precision = true_positive / (true_positive + false_positive) if true_positive else 0.0
    recall = true_positive / (true_positive + false_negative) if true_positive else 0.0
    f1 = 2 * precision * recall / (precision + recall) if precision + recall else 0.0
    f1_scores.append(f1)
    print(f"{label}: precision={precision:.3f} recall={recall:.3f} f1={f1:.3f}")

accuracy = sum(actual == guess for actual, guess in zip(gold, predicted)) / len(gold)
macro_f1 = sum(f1_scores) / len(f1_scores)
print(f"accuracy={accuracy:.3f} macro_f1={macro_f1:.3f}")

billing: precision=0.667 recall=1.000 f1=0.800
technical: precision=0.000 recall=0.000 f1=0.000
accuracy=0.667 macro_f1=0.400

technical is never predicted, so the example defines its precision and recall as zero. Evaluation libraries may use a parameter to choose a warning, zero, or another behavior. Pin that choice so an upgrade cannot silently change metric semantics. Inspect examples in the confusion matrix too, because the same F1 can come from different errors.

Pitfalls

Further reading

• Unicode normalization forms
• Unicode text segmentation
• Python 3.14 unicodedata documentation
• Python 3.14 regular expression documentation
• Hugging Face tokenizer summary