# Regular expressions

Source: https://codewiki.com/foundations/regular-expressions/

> - **what**: A regular expression is a small pattern program that searches text and can expose the parts it matched.
> - **trap**: Matching a shape does not prove its meaning, and ambiguous repetition can turn a short pattern into expensive work on hostile input.
> - **fix**: State the text contract, require full consumption for validation, escape dynamic literals, bound input, and test near misses as well as successful cases.

## What it is and why it exists

A regular expression, usually shortened to regex, describes a set of text sequences. A regex engine tries that pattern against an input string and reports whether and where it matches. Parenthesized groups can also expose selected substrings for later code.

Regexes are concise because one pattern combines literal text with operators for choice, repetition, boundaries, and character classes. They are useful when the text has a local, regular shape: an identifier, a log prefix, a token inside a document, or a mechanical rewrite. The same density that makes a pattern convenient also makes an unstated contract easy to hide.

Four common jobs use the same engine but need different success criteria:

| Job | Successful result | Question to ask |
| --- | --- | --- |
| Validation | The whole input has the permitted shape | Did the match consume every code unit? |
| Search | At least one relevant substring was found | Which occurrence and which flags apply? |
| Capture | Named parts were returned with the match | Can an optional group be absent? |
| Replacement | Matches were transformed without changing other text | Is replacement text literal or interpreted? |

A capturing group records the substring consumed by one parenthesized part. Named groups such as `(?<year>\d{4})` make the result easier to review than numbered groups in a changing pattern. Use a noncapturing group `(?:...)` when parentheses exist only to control alternation or repetition.

A regex recognizes shape, not domain truth. A month fragment can restrict text to `01` through `12`, but a pattern that accepts `2025-02-29` has not checked the calendar. Parse and validate semantic rules in ordinary code after the regex establishes a safe, useful shape.

Regular-expression syntax is a dialect, not one universal language. This topic uses ECMAScript regexes on Node 24. A construct from PCRE, Python, Java, or a linear-time engine may be missing or have different escaping, Unicode, anchoring, and replacement behavior.

## How it works

The engine receives a pattern, flags, an input string, and usually a starting position. It looks for a path through the pattern whose operations can consume input in order. On success it returns the overall span plus any captures; on failure the calling API returns `null` or `false`.

Read a pattern from the outside inward. First identify anchors and flags, then top-level alternatives, then groups, and finally the quantifiers attached to individual atoms. This order reveals whether the pattern searches a substring or claims to validate the whole input.

### Atoms, choices, and quantities

An atom consumes one unit or asserts a position. Literal `A`, character class `[A-Z]`, escape `\d`, wildcard `.`, and a parenthesized group are common atoms. An assertion such as `^`, `$`, `\b`, or lookaround checks a position without consuming text.

Quantifiers attach to the atom immediately before them:

| Form | Meaning |
| --- | --- |
| `x?` | zero or one `x` |
| `x*` | zero or more `x` values |
| `x+` | one or more `x` values |
| `x{2,5}` | between two and five `x` values |
| `x+?` | one or more, preferring fewer at first |

Greedy and lazy describe which candidate length the engine tries first, not a safety guarantee. A greedy quantifier can give characters back when the remainder fails. A lazy quantifier can expand repeatedly for the same reason.

Alternation `left|right` tries alternatives in source order at a candidate start. Parentheses define its reach: `cat|dog food` differs from `(?:cat|dog) food`. When alternatives overlap, their order can change the captured result even if both forms report a match.

### Captures and match results

The overall match is result element `0`. Numbered captures follow opening-parenthesis order, while named captures are available through `match.groups`. A group that participates several times under a quantifier normally retains only its last captured substring.

Optional groups can produce `undefined`. Downstream code must not assume every declared group participated. If code only needs to test existence, make structural groups noncapturing so capture results stay small and intentional.

Backreferences such as `\1` or `\k<name>` ask later input to equal an earlier capture. They are different from replacement references such as `$1` and `$<name>`, which are interpreted after matching by replacement APIs. Mixing the two contexts is a common review error.

### Flags and state

Flags change the language and the search procedure. `u` enables Unicode-aware parsing and code-point handling for several constructs. `i` ignores case, `m` changes line-anchor behavior, `s` lets dot match line terminators, `g` iterates globally, and `y` requires a match exactly at `lastIndex`.

The `g` and `y` flags make a `RegExp` object stateful through `lastIndex`. Calls to `exec()` or `test()` can advance or reset that property. Reusing one global object as a validator can therefore make identical valid inputs alternate between `true` and `false`.

Choose the API from the result you need:

| API | Useful result | State concern |
| --- | --- | --- |
| `pattern.test(text)` | one Boolean | mutates `lastIndex` with `g` or `y` |
| `pattern.exec(text)` | one match and captures | mutates `lastIndex` with `g` or `y` |
| `text.matchAll(pattern)` | iterator of all detailed matches | requires a global regex |
| `text.replace(pattern, value)` | transformed string | callback avoids replacement-template ambiguity |

### Escaping at two layers

A regex literal such as `/\d+/u` is parsed by JavaScript as regex syntax directly. The string passed to `new RegExp("\\d+", "u")` is parsed once as a JavaScript string and then again as regex syntax. The doubled backslash belongs to the string layer, not to a different regex meaning.

Dynamic text is data, not regex source. On Node 24, `RegExp.escape(userText)` turns an arbitrary literal fragment into source that can safely be embedded in a larger pattern. Escaping only dots and stars by hand misses punctuation and context-sensitive cases.

### Unicode contracts

JavaScript strings use UTF-16 code units. With `u`, dot and several regex operations treat a surrogate pair for one code point as one item, but they still do not identify a user-perceived grapheme cluster. A combining sequence or joined emoji can contain several code points.

The shorthand `\d` remains ASCII digits in ECMAScript, even with `u`. If the contract permits decimal digits from other scripts, use `\p{Decimal_Number}` with `u` and decide whether later numeric parsing supports them. If the contract is an ASCII protocol token, `\d` may be exactly right.

Unicode property escapes such as `\p{Script=Greek}` state character intent more clearly than large copied ranges. They require Unicode-aware mode and still need tests for combining marks, normalization forms, and mixed-script input. Regex matching does not normalize text automatically.

## Examples

These examples progress from full-input validation to search, capture-driven replacement, and measured adversarial behavior. Every output below comes from executing the shown file with Node `v24.14.0`.

### Validating and parsing an order reference

The first pattern matches a prefix and captures two named fields. The function separately requires the matched span to equal the full input, so a valid-looking prefix followed by extra text is rejected.

<!-- quick -->

```javascript
// file: parse_reference.js
const referencePattern = /^(?<region>[A-Z]{2})-(?<number>\d{6})/u;

function parseReference(input) {
  const match = referencePattern.exec(input);
  if (!match || match[0].length !== input.length) return null;

  return match.groups;
}

const samples = [
  "FR-004219",
  "fr-004219",
  "FR-4219",
  "FR-004219\n",
];

for (const sample of samples) {
  const parsed = parseReference(sample);
  const result = parsed
    ? `region=${parsed.region}, number=${parsed.number}`
    : "invalid";
  console.log(`${JSON.stringify(sample)} => ${result}`);
}
```

```text
"FR-004219" => region=FR, number=004219
"fr-004219" => invalid
"FR-4219" => invalid
"FR-004219\n" => invalid
```


<!-- /quick -->

Keeping the number as text preserves leading zeroes. If the application later turns it into a number, that conversion is a separate contract. The regex does not assert that the referenced order exists.

JavaScript's `$` assertion can also match before a final line terminator, so `^...$` alone is easy to overstate as absolute full consumption. Comparing the overall match with the original input makes the intended boundary explicit. Another design can reject line terminators first and use a carefully documented anchor policy.

### Finding alerts with named captures

Here `m` makes `^` and `$` operate at line boundaries, while `g` finds every matching line. `matchAll()` returns match indices and named groups without a manual `exec()` loop.

```javascript
// file: find_alerts.js
const log = [
  "2026-09-04T10:30:00Z [INFO] worker started",
  "2026-09-04T10:31:08Z [WARN] queue depth is 42",
  "2026-09-04T10:31:11Z [ERROR] payment timed out",
].join("\n");

const alertPattern =
  /^(?<time>\S+) \[(?<level>WARN|ERROR)\] (?<message>.+)$/gmu;

for (const match of log.matchAll(alertPattern)) {
  const { time, level, message } = match.groups;
  console.log(`${level} at ${time}: ${message}`);
  console.log(`  match starts at index ${match.index}`);
}
```

```text
WARN at 2026-09-04T10:31:08Z: queue depth is 42
  match starts at index 43
ERROR at 2026-09-04T10:31:11Z: payment timed out
  match starts at index 89
```

The pattern recognizes only the log envelope needed by this task. `\S+` is not a timestamp validator, and `.+` deliberately leaves message interpretation to later code. Narrow each field only when the consumer has a real rule for it.

The indices are UTF-16 code-unit offsets because that is how JavaScript indexes strings. They are suitable for `slice()` on the same string, but not automatically for byte offsets in a UTF-8 file or columns displayed to a user.

### Replacing matches with a callback

This example finds two formatted 16-digit sequences and uses named captures to preserve only their first and last groups. A callback constructs the replacement explicitly, avoiding special `$` sequences inside a replacement template.

```javascript
// file: redact_cards.js
const note = [
  "primary=4111 1111 1111 1111",
  "backup=5555-4444-3333-2222",
  "reference=20260904",
].join("; ");

const cardPattern =
  /(?<!\d)(?<first>\d{4})[ -]?\d{4}[ -]?\d{4}[ -]?(?<last>\d{4})(?!\d)/gu;

let replacements = 0;
const redacted = note.replace(cardPattern, (...arguments_) => {
  const groups = arguments_.at(-1);
  replacements += 1;
  return `${groups.first}-••••-••••-${groups.last}`;
});

console.log(redacted);
console.log(`replacements=${replacements}`);
```

```text
primary=4111-••••-••••-1111; backup=5555-••••-••••-2222; reference=20260904
replacements=2
```

Lookbehind and lookahead assert digit boundaries without consuming neighboring text. The eight-digit reference remains unchanged because it cannot satisfy the full 16-digit shape. The callback receives the named-groups object as its final argument when the pattern has named captures.

This is a formatting demonstration, not a complete payment-data control. Real systems should avoid receiving unnecessary card data, apply storage and logging policy before text reaches general logs, and test every supported input format. Redaction after logging is too late.

### Measuring ambiguous repetition

The nested pattern can partition the same run of `a` characters in many ways before the final `!` proves failure. The bounded pattern states the actual sample policy directly: one through 24 `a` characters and nothing else.

```javascript
// file: measure_backtracking.js
const nestedPattern = /^(a+)+$/u;
const boundedPattern = /^a{1,24}$/u;

for (const size of [12, 16, 20, 24]) {
  const adversarial = `${"a".repeat(size)}!`;
  const started = performance.now();
  const nestedMatch = nestedPattern.test(adversarial);
  const elapsed = performance.now() - started;
  const boundedMatch = boundedPattern.test(adversarial);

  console.log(
    `${adversarial.length} chars: nested=${nestedMatch} ` +
      `${elapsed.toFixed(3)}ms, bounded=${boundedMatch}`,
  );
}
```

```text
13 chars: nested=false 0.156ms, bounded=false
17 chars: nested=false 0.335ms, bounded=false
21 chars: nested=false 4.683ms, bounded=false
25 chars: nested=false 73.597ms, bounded=false
```

These are observations from one Node 24 run on the review machine, not portable benchmark constants. The important evidence is the failure case and the rapidly growing work as four characters are added. A production test should enforce a generous machine-specific budget without asserting these exact milliseconds.

The bounded expression also changes the accepted language by enforcing a 24-character policy. If arbitrary length is a true requirement, use a construction or engine with a defensible worst-case bound instead of hiding a cap. Never run an intentionally dangerous benchmark at unbounded sizes.

## Pitfalls

### Treating shape as meaning

> **Pitfall:** A date, email address, URL, or identifier can match a plausible regex and still be invalid in its domain. Increasing pattern size to encode every semantic rule often makes the contract harder to inspect.

**Fix:** use the regex for a documented lexical boundary, then parse with the domain API and apply semantic checks. Test a shape-valid but meaning-invalid value, such as a nonexistent calendar date, so the two stages cannot be confused.

### Validating only a substring

> **Pitfall:** `test()` succeeds when any permitted substring matches unless the pattern and calling code demand full consumption. In JavaScript, `$` can match before a final line terminator, and `m` deliberately changes anchors to line boundaries.

**Fix:** compare the overall match span with the entire input or use an explicitly reviewed absolute-boundary construction. Keep validation regexes free of `g` and `y`, and add cases with leading text, trailing text, and a trailing newline.

### Confusing source with literal data

> **Pitfall:** Interpolating a tenant name, file extension, or search term directly into `new RegExp()` lets punctuation alter grouping, repetition, or alternatives. Handwritten backslash replacement is easy to get wrong across two parsing layers.

**Fix:** keep fixed patterns as regex literals. When dynamic composition is necessary, pass literal fragments through `RegExp.escape()` on Node 24, keep trusted regex source separate from data, and test punctuation such as `.`, `-`, `(`, `]`, and `\`.

### Assuming one meaning of character

> **Pitfall:** Dot, `\d`, string offsets, Unicode property escapes, and user-perceived characters use different units or sets. Adding `u` fixes several code-point behaviors but does not make dot consume a whole grapheme cluster or make `\d` match every decimal script.

**Fix:** name the unit and repertoire in the requirement: ASCII digit, Unicode decimal number, code point, grapheme, byte, or UTF-16 code unit. Use Unicode property escapes and `Intl.Segmenter` where those contracts call for them, with normalization handled as a separate policy.

### Writing ambiguous repeated alternatives

> **Pitfall:** Nested quantifiers and overlapping alternatives can create many equivalent ways to consume a prefix. A near match that fails late may force a backtracking engine to revisit those choices, making attacker-controlled input a denial-of-service risk.

**Fix:** remove ambiguous nesting, factor shared prefixes, and set explicit input and repetition bounds. Exercise long near misses under a deadline in an isolated test process; for exposed, complex patterns, consider a parser or an engine with a suitable worst-case guarantee.

### Trusting global-regex state

> **Pitfall:** A shared regex with `g` or `y` carries `lastIndex` between `test()` and `exec()` calls. Code that first tests and then executes may skip the desired match, while concurrent-looking consumers can interfere through the same object.

**Fix:** omit stateful flags for single validation, use `matchAll()` for complete iteration, or control one explicit `exec()` loop and its zero-length behavior. Never call `test()` merely as a preflight for the `exec()` that needs the captures.

<!-- deep -->

## Backtracking, state, and bounded work

Most practical regex engines expose a leftmost-first search policy. They scan candidate starts from left to right; at a start, ordered alternatives and quantifier preferences decide which path is attempted first. That policy explains why two patterns that accept the same strings can return different captures.

### Choice points

A quantifier or alternation can create a choice point. If later pattern elements fail, a backtracking engine restores an earlier position and tries another choice. This is backtracking; it is normal matching behavior, not automatically a bug.

Trouble starts when many paths consume the same prefix. In `^(a+)+$`, both the inner and outer `+` decide how to partition one run of `a`. Appending a forbidden `!` makes the engine reject only after exploring many partitions that all looked viable earlier.

Factoring shared prefixes reduces choices. Replacing broad `.*` regions with a delimiter-aware class such as `[^,]*` can also make intent clearer, but no mechanical rewrite proves safety for every surrounding pattern. Review the whole expression and its engine.

### What the measurement establishes

The executed example produced these observations:

| Input length | Nested-pattern failure | Bounded-pattern result |
| --- | --- | --- |
| 13 code units | 0.156 ms | `false` |
| 17 code units | 0.335 ms | `false` |
| 21 code units | 4.683 ms | `false` |
| 25 code units | 73.597 ms | `false` |

The table establishes behavior for one engine version, machine, pattern, and input family. It does not establish a universal threshold or a precise complexity class for every regex. JIT warm-up, engine optimizations, CPU load, and different strings can change the numbers.

A useful regression test records a generous upper budget and runs in a worker or child process that the harness can terminate. A JavaScript regex call itself offers no portable mid-match cancellation hook. A timeout checked only after synchronous matching returns cannot interrupt the work that already blocked the event loop.

### Bounds belong to the contract

An input-size limit is not a substitute for pattern review, but it turns an unbounded risk into a capacity decision. Apply the limit before matching, in the same unit named by the surrounding protocol. A byte limit at ingress and a code-unit limit inside JavaScript solve different problems and may both be needed.

Bound individual repetitions when the domain already has a maximum. An order number with exactly six ASCII digits should say `{6}`, not `+`. A field capped at 64 code units should not use `.*` and rely on later truncation.

Pattern ownership matters too. A fixed, reviewed pattern over bounded input has a different threat model from a pattern supplied by an administrator, and both differ from raw user-supplied regex source. Escaping a fragment makes it literal; it does not make an intentionally supplied regex safe to execute.

### Stateful iteration

With `g` or `y`, successful `exec()` updates `lastIndex` to the end of the match. Failure resets it to zero. A loop that can return a zero-length match must still guarantee progress, or it can repeatedly observe the same position in APIs where advancement is manual.

`matchAll()` packages global iteration and yields each detailed result. It is a good default when all matches and captures are needed. For tokenizers, sticky `y` can express “the next token must start here,” but the caller must own `lastIndex` and report an error when no token begins at that position.

Do not share mutable iteration state as hidden module configuration. Construct the regex near the operation, clone it when separate cursors are intentional, or use an API whose iteration state is local to the returned iterator. Tests should interleave two scans to expose accidental sharing.

### Capture boundaries

Captures are part of the program's output schema. Give business-relevant fields names, keep grouping-only parentheses noncapturing, and document optional groups as nullable. Changing parentheses in a pattern can otherwise silently renumber every later `$1` or `match[1]` consumer.

Captures store substrings and offsets from the original JavaScript string. They do not parse numbers, canonicalize Unicode, decode escapes, or validate referential integrity. Perform those transformations explicitly and preserve original text when diagnostics or audit needs it.

Replacement callbacks make the schema visible as function arguments and return literal replacement text. Replacement strings have their own mini-language for `$&`, `$1`, `$<name>`, and related tokens. If replacement text contains untrusted dollar signs, a callback returning that text avoids interpreting it as a template.

### A test corpus that earns confidence

Start with examples that should match and examples that differ at one boundary. Include empty input, shortest and longest permitted values, one value just beyond each limit, leading and trailing junk, line terminators, and optional-group absence. This checks the accepted language rather than only the happy path.

Add Unicode cases derived from the declared character contract: supplementary code points, combining sequences, mixed scripts, non-ASCII decimal digits, and canonically equivalent forms where relevant. Do not add every Unicode curiosity to every pattern; select cases that could falsify the stated policy.

Finally, build near misses from every repeated or overlapping region and increase their size only within a safe test cap. Record the runtime and hardware context for measurements. Keep these adversarial cases in regression tests whenever the pattern guards an exposed request path.

<!-- /deep -->

[Checkpoint: foundations/regular-expressions](https://codewiki.com/foundations/regular-expressions/#checkpoint)

## Further reading

- [ECMAScript specification: RegExp objects](https://tc39.es/ecma262/multipage/text-processing.html#sec-regexp-regular-expression-objects)
- [MDN: Regular expressions guide](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_expressions)
- [MDN: `RegExp` reference](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/RegExp)
- [MDN: Unicode character class escapes](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Regular_expressions/Unicode_character_class_escape)
- [OWASP: Regular expression denial of service](https://owasp.org/www-community/attacks/Regular_expression_Denial_of_Service_-_ReDoS)