# List comprehensions

Source: https://codewiki.com/python/list-comprehensions/

> - **what**: A list comprehension visits an iterable in order, evaluates an expression for each accepted item, and immediately builds a new list.
> - **trap**: A trailing `if` drops items, while a leading `X if condition else Y` chooses a result; confusing them causes invalid syntax or wrong data.
> - **fix**: Use a comprehension when there are few clauses and each input produces at most one result; use a loop for side effects, complex branches, or step-by-step debugging.

## What it is and why it exists

A list comprehension is a form of list display.
It writes "iterate, optionally filter, calculate, and append to a new list" as one expression, with the basic form `[expression for item in iterable if condition]`.
The result is a new list object even when the input isn't a list.

The input only needs to be an iterable, such as a list, tuple, `range`, dictionary view, or generator.
Each item that passes the condition causes the leading expression to be evaluated once, and the result is placed in traversal order.
Rejected items don't enter the result, and the leading expression isn't evaluated for them.

The syntax keeps a simple mapping and filter from being scattered across several statements.
An ordinary loop first creates an empty list, then needs a `for`, a condition, and `append()`; a comprehension keeps that data rule together.
You should still be able to answer two questions quickly: where does the input come from, and which values enter the result?

You'll see list comprehensions in data cleaning, API response shaping, test fixtures, and parameter combinations.
They fit when the task naturally reads as "produce zero or one output for each input."
If one input may append several different results, or the body updates outside state, an ordinary loop is usually more direct.

A list comprehension builds only a list.
Dictionary and set comprehensions and generator expressions reuse a similar clause structure, but differ in result type, duplicate handling, and evaluation timing.
Calling all of these forms “list comprehensions” hides behavior that matters.

## How it works

The result expression comes first, followed by at least one `for` clause and then zero or more `for` or `if` clauses.
Execution doesn't start with the visually leftmost expression; the `for` and `if` clauses expand from left to right.
The result expression is evaluated and appended once only when execution reaches the innermost point.

The following two forms have the same logic. Clause order in the comprehension matches the outside-to-inside order of the expanded statements:

`[transform(item) for item in source if keep(item)]`

The loop visits, filters, and finally transforms and appends:

`for item in source` → `if keep(item)` → `result.append(transform(item))`

That order affects correctness.
A filter can prevent the result expression from running, so a dangerous or costly transformation can be limited to valid input.
Conversely, if both the filter and result call the same function, it runs twice for accepted items; stateful calls may even produce inconsistent answers.

Multiple `for` clauses are nested loops.
`[pair for left in lefts for right in rights]` holds one `left`, traverses all of `rights`, and only then advances to the next `left`.
Later clauses can use names bound by earlier clauses; swapping the clauses may reorder the output or reference a name before it exists.

A trailing `if` is a filter clause and has no `else`. A conditional expression occupies the result position and has the form `when_true if condition else when_false`, so each input can produce one of two results. Both forms may appear together: the trailing condition decides whether to keep an input, then the leading condition decides what an accepted input produces.

A list comprehension consumes its input immediately and constructs the complete result. It fits when you need to index, traverse repeatedly, or modify that result later. If the values go straight into a one-time consumer such as `sum()`, `any()`, or `all()`, a generator expression can avoid constructing an intermediate list.

### Read outward from `for`

When a comprehension spans several lines, don't infer execution from the visual order of the result expression. Find the first `for`, indent each later clause in order, and finally treat the leftmost expression as the argument to the innermost `append()`. The same reading method works for list, set, and dictionary comprehensions.

Check a comprehension in this order:

1. Find the input after the leftmost `for` and identify the values it provides.
2. Record the iteration target's unpacking shape, such as `for key, value in mapping.items()`.
3. Apply later `for` and `if` clauses from left to right, checking that every name is already bound.
4. Evaluate the result expression and verify that it runs once for every path that reaches it.
5. Use the delimiters to identify the result container, then check whether the caller relies on order, duplicates, or repeated consumption.

Each part has one job in the basic form:

| Part | Example | Job |
| --- | --- | --- |
| Result expression | `normalize(item)` | Create the value to append |
| Iteration clause | `for item in source` | Bind each input value in turn |
| Filter clause | `if item.active` | Decide whether this iteration reaches the result expression |
| Conditional expression | `x if condition else y` | Choose a result for an accepted input |

The syntax allows consecutive `if` clauses, but they are simply nested filters. `[x for x in values if ready(x) if valid(x)]` has the same acceptance rule as joining those conditions with `and`, including left-to-right short-circuiting. Separate clauses can make a dependency clearer when the first condition excludes values that the second condition can't handle.

### Choose by result cardinality

When every input produces exactly one output, you need only a transformation expression and one `for`. Add a filter when each input produces zero or one output. Add another `for` when one input produces several outputs, but only if the nested relationship is simple and stable enough to read in compressed form.

A comprehension becomes awkward when branches append different numbers of results. Grouping, accumulation, early exit, and per-item recovery all need statement-level control flow and shouldn't be forced into one expression. The test isn't whether the code can be written as a comprehension; it is whether its expansion still has one clear data path.

## Examples

These four examples progressively add filtering, a conditional expression, multiple iteration clauses, and an assignment expression. Each program runs independently, and the output shown below came from the local `python3` runtime.

### Transform and filter

Payment status is the filter, and receipt text is the transformation. An unpaid order doesn't evaluate the formatting expression and doesn't appear in the new list.

<!-- quick -->

```python
# file: paid_receipts.py
orders = [
    {"id": "A100", "total": 82.5, "paid": True},
    {"id": "A101", "total": 19.0, "paid": False},
    {"id": "A102", "total": 120.0, "paid": True},
]

receipts = [
    f'{order["id"]}: ${order["total"]:.2f}'
    for order in orders
    if order["paid"]
]

print(receipts)
```

```text
['A100: $82.50', 'A102: $120.00']
```

<!-- /quick -->

The comprehension preserves the source order. It doesn't modify `orders`, although result items aren't always isolated from input objects; this expression creates new strings, so there is no shared mutable object here.

When expanded, `for order in orders` is outermost, `if order["paid"]` sits inside it, and formatting plus append happen last. Reverse-expanding a more complicated comprehension this way usually makes its execution order obvious.

### Conditional expression and filter clause

This example filters out `None` and negative values, then discounts prices of at least `20`. The leading `if ... else ...` chooses one result for every retained price, while the trailing `if` determines whether to retain a price.

```python
# file: normalize_prices.py
prices = [12, 0, None, 27, -3, 40]

normalized = [
    round(price * 0.9, 2) if price >= 20 else float(price)
    for price in prices
    if price is not None and price >= 0
]

print(normalized)
```

```text
[12.0, 0.0, 24.3, 36.0]
```

`None` and `-3` are rejected, so they don't reach numeric comparison or transformation. A simple `if price` would be wrong because truth-value filtering would also drop the valid zero; the code checks missing data and the accepted range explicitly.

A conditional expression always needs `else`. If the requirement is only to retain high prices, write `[price for price in prices if price is not None and price >= 20]`. Use the conditional expression when every valid price remains but requires a different transformation.

### Multiple `for` clauses

Multiple clauses can flatten one level of nested data. This code visits departments, then members of the current department, and finally filters active members; the second `for` can use `department`, which the first one bound.

```python
# file: active_members.py
departments = [
    {
        "name": "sales",
        "members": [
            {"email": "ADA@EXAMPLE.COM", "active": True},
            {"email": "LIN@EXAMPLE.COM", "active": False},
        ],
    },
    {
        "name": "support",
        "members": [
            {"email": "KAI@EXAMPLE.COM", "active": True},
        ],
    },
]

addresses = [
    f'{department["name"]}:{member["email"].lower()}'
    for department in departments
    for member in department["members"]
    if member["active"]
]

print(addresses)
```

```text
['sales:ada@example.com', 'support:kai@example.com']
```

Department order determines the first part of the output order, and member order within each department determines the rest. A comprehension neither sorts nor deduplicates automatically; if the requirements call for either, choose `sorted()` or a set explicitly and check whether that changes the business meaning.

Two `for` clauses are already near the limit of what most readers can parse at a glance. If you also need several filters, exception handling, or intermediate logging, expand the code into named loops; fewer lines aren't a reason by themselves to use a comprehension.

### Compute once

When filtering needs a converted value and the result needs it too, calling the parser twice wastes work. An assignment expression, `:=`, can retain the value in the filter, but its scope rule must be understood.

```python
# file: parse_scores_once.py
def parse_score(raw):
    try:
        return int(raw)
    except ValueError:
        return None


raw_scores = ["18", "skip", " 27 ", "-3", "42"]

scores = [
    score
    for raw in raw_scores
    if (score := parse_score(raw)) is not None and score >= 0
]

print(scores)
print(f"last parsed: {score}")
```

```text
[18, 27, 42]
last parsed: 42
```

Every raw string calls `parse_score()` once. The condition excludes `None` first, so short-circuit evaluation compares `score >= 0` only after parsing succeeds; a negative number parses but fails the range condition.

`score` still exists after the comprehension because an assignment expression in a comprehension binds its target in the containing scope. That is a deliberate special rule, so don't assume this name behaves like a non-leaking temporary. If the binding surprises a reader, an ordinary loop or small helper is clearer.

## Pitfalls

### Hiding side effects in the result expression

> **Pitfall:** `[send(message) for message in messages]` builds a list even when the caller cares only about `send()` side effects. If the function returns `None`, it also leaves behind a useless list of `None` values.
>
> **Fix:** Use an ordinary `for` loop so the side effect and failure point are visible. Use a list comprehension only when the result list is required, and test exception propagation.

A comprehension doesn't turn side effects into a batch transaction. If a call fails halfway through, earlier calls may already have taken effect even though assignment of the list never completes. An explicit loop expresses ownership better when you need rollback, retries, or per-item error records.

### Writing a filter as a conditional expression

> **Pitfall:** `[value if valid(value) for value in values]` is invalid syntax because a conditional expression in the result position is missing its `else`. Models borrowing syntax from other languages often generate this wrong ordering.
>
> **Fix:** To discard invalid items, write `[value for value in values if valid(value)]`. To keep every item but select a result, write `[value if valid(value) else fallback for value in values]`.

It helps to name the operation in plain language: "filter" maps to a trailing clause, while "replace" or "label" maps to a leading conditional expression. When both appear, verify that filtered inputs never execute the result expression.

### Repeating work in the filter and transformation

> **Pitfall:** `[parse(raw) for raw in rows if parse(raw) is not None]` calls `parse()` twice for every accepted item. Besides the extra work, a function that reads a cache, counter, or clock may return different results between calls.
>
> **Fix:** Use a parenthesized `:=` to retain a simple, readable intermediate value, or separate parsing, validation, and append in an ordinary loop. Don't introduce a surprising binding just to save a line.

Check exception semantics too. If the parser raises, the filter doesn't skip that item; it terminates the whole comprehension. For per-item recovery, have a helper return an explicit result type or use a narrowly scoped `try` block in a loop.

### Losing the domain order in nesting

> **Pitfall:** The order of multiple `for` clauses controls traversal order and name visibility. Generated code may swap inner and outer clauses, causing different ordering, different combinations, or a reference before binding.
>
> **Fix:** Expand the comprehension into nested loops and write down the input cardinality and dependent names at each level. If more than two levels still need explanation, keep the expanded loop and give intermediates domain names.

Don't confuse a nested comprehension with multiple `for` clauses. `[[transform(x) for x in row] for row in matrix]` preserves rows, while `[transform(x) for row in matrix for x in row]` flattens one level. Different delimiter shapes produce different result shapes.

### Materializing a list for one consumer

> **Pitfall:** `sum([price * quantity for price, quantity in lines])` constructs the complete list before summing it. For a large input, that intermediate list may have no other use.
>
> **Fix:** For one-time consumption, write `sum(price * quantity for price, quantity in lines)`. Keep a list comprehension when you need indexing, repeated traversal, or mutation, and don't claim either form is faster without measuring real data on the target runtime.

A generator expression isn't automatically better. It is usually single-use, errors may move to iteration time, and retaining it may extend an input resource's lifetime. Choose the boundary from how the result is consumed, not just from input size.

### Creating deferred functions inside a comprehension

> **Pitfall:** `[lambda: item for item in items]` creates functions that read `item` later. When they run after the comprehension, they normally all see the same final binding rather than a snapshot from each iteration.
>
> **Fix:** If each iteration needs its own value, use `lambda item=item: item` to bind a default at function creation, or call a named factory that accepts `item`. Prefer a factory when callbacks capture more state because it makes ownership visible.

An immediately evaluated result expression doesn't have this timing gap; the problem appears when the result itself is a deferred function. Run every callback after the comprehension in tests and use at least two distinct inputs, because one element can't expose shared binding.

This is a closure late-binding issue, not a list-copying problem. Don't copy the whole input object automatically; first decide whether the callback should retain the object reference or one immutable field value from creation time.

<!-- deep -->

## Scope and evaluation boundaries

The iteration target of a comprehension has a separate implicit scope. If an outer `item` already exists, `[item * 2 for item in values]` doesn't overwrite that outer binding. This differs from an ordinary `for item in values`, after which `item` remains bound to the final iteration value.

The iterable expression in the leftmost `for` is the exception: it is evaluated directly in the enclosing scope, then passed into the comprehension. Later `for` clauses, all filters, and the result expression follow the comprehension's scope semantics, so they can depend on iteration targets already bound to their left. This is why later clauses see earlier names while the enclosing scope doesn't see those iteration targets.

Assignment expressions follow a different special rule. Inside a list, set, or dictionary comprehension or a generator expression, a `:=` target binds in the scope containing the comprehension; an existing `global` or `nonlocal` declaration is honored. That is why `score` remains readable after the earlier example.

The assignment target can't have the same name as any iteration variable in the containing comprehension. `[item := item + 1 for item in values]` raises `SyntaxError` because one name can't be both a comprehension-local iteration target and an enclosing binding. Assignment expressions are also prohibited in any comprehension iterable expression after `in`.

These are language semantics, not promises about a particular bytecode shape. Python can change its implementation while preserving observable behavior such as non-leaking iteration names and clause order. Debug by expanding the source and checking results before inferring portable rules from one CPython version's disassembly.

### Exceptions and partial evaluation

Assignment to the list variable occurs only after the entire comprehension succeeds, but result and filter functions run one item at a time. If one raises, the temporary list being built on the right isn't assigned to the left-hand name; external side effects that already happened aren't rolled back. This is the deeper reason to keep side effects out of comprehensions.

Short-circuit rules still apply. `if parsed is not None and parsed >= 0` doesn't evaluate the second comparison when the first one is false, which avoids comparing `None` with an integer. Consecutive filters behave like nested `and` conditions, but splitting them into several `if` clauses doesn't cache a repeated subexpression.

### Mutable elements remain shared

A new list isn't a deep copy. `[record for record in records]` creates a new outer list but reuses each `record` object, so later dictionary mutation is visible through both lists. You get new element objects only when the result expression creates them, as it does for a new string, number, or container.

Ask two separate questions when reviewing generated code: "Is the outer container new?" and "Are inner objects shared?" If records need copying, state the copy policy explicitly, such as `record.copy()` for shallow dictionaries. For more complex nested ownership, define which nodes require isolation before choosing a copying operation.

### Test comprehension invariants

A comprehension is short, but one ordinary input isn't enough for a test. Traversal, filtering, and transformation are compressed together, so boundary bugs often appear as a missing item, reordered output, repeated call, or wrong nesting shape. Effective tests target those observable behaviors directly.

- Empty input should produce an empty list without calling the result expression.
- Input containing accepted and rejected items should verify call count and output order.
- Domains where `0`, an empty string, and `None` differ should cover them separately to catch an incorrect truth-value filter.
- Nested input should contain at least two outer items with different inner lengths to expose clause-order and flattening mistakes.

If the transformation can fail, add a failing value in the middle of the input. Confirm the exception type and propagation point, and check for side effects already produced before failure. For code using `:=`, also assert that the transformation is called once per input.

Property tests can fit pure comprehensions too. For example, a filter-only result can't be longer than its input, and its order should be a subsequence of input order. Assert such properties only when they belong to the requirements, because set conversion or explicit sorting changes them.

### Unpacking targets and failure points

The target after `for` may use sequence unpacking. `[price * quantity for price, quantity in lines]` requires every item to unpack into exactly two values; a wrong-length item raises `ValueError` before the filter or result expression is reached. A filter can't rescue an unpacking operation that has already failed.

Iterating a dictionary produces keys by default. Generated code that writes `[key for key, value in mapping]` usually tries to unpack each key into two values instead of receiving key-value pairs. Call `mapping.items()` explicitly when both are required.

`zip()` can provide unpacked tuples from several inputs, but it stops when the shortest input is exhausted by default. If unequal lengths indicate bad data, silent truncation hides missing items; in Python 3.14, `zip(left, right, strict=True)` raises `ValueError` for a mismatch. The data contract should decide whether strictness is appropriate.

Unpacking, filtering, and result-expression failures happen at different stages. One failure sample for each stage locates the expected exception boundary and can reveal generated code that validates too late.

### Input iterator ownership

A list comprehension consumes an input iterator from its current position to exhaustion. If the caller later reads that same iterator, it sees only what remains, often nothing; the comprehension doesn't copy an iterator. For one-shot streams, the interface should make consumption ownership clear.

When an exception interrupts the comprehension, the input iterator has usually advanced. Retrying the same comprehension may resume partway through rather than replaying the original data. For repeatable retries, use a source that can be recreated, or deliberately materialize a stable snapshot after accepting its memory cost.

Mutating the source container during traversal is also hard to reason about. Changing dictionary size commonly raises `RuntimeError`, while list mutation can skip or repeat items. To remove items, construct a new list and replace the old binding after completion; don't modify the traversed container inside the result expression.

The same ownership rule applies to file lines, database cursors, and generators. Passing them to a comprehension means the current call advances them immediately; if a resource must close after failure, put the whole consumption inside its context manager.

## Similar syntax, different containers

A list comprehension, `[expression for ...]`, immediately creates an ordered list that permits duplicates. A set comprehension, `{expression for ...}`, creates a set and removes duplicates; code shouldn't depend on its iteration or display order. A dictionary comprehension, `{key: value for ...}`, needs both expressions, and a later value replaces an earlier one when keys collide.

A generator expression, `(expression for ...)`, yields values on demand instead of immediately constructing a full list. It fits single-pass streaming, but the same generated results can't be traversed again as a list can. Decide whether to replace a list comprehension by checking the caller's access pattern, error timing, and resource lifetime.

These forms share the left-to-right `for` and `if` clause model but not container semantics. Merely changing `[]` to `{}` or `()` can alter duplicates, order, evaluation timing, and repeatability, so treat such an edit as a behavior change in review.

<!-- /deep -->

[Checkpoint: python/list-comprehensions](https://codewiki.com/python/list-comprehensions/#checkpoint)

## Further reading

- [Python tutorial: List Comprehensions](https://docs.python.org/3.14/tutorial/datastructures.html#list-comprehensions)
- [Python language reference: list, set, and dictionary displays](https://docs.python.org/3.14/reference/expressions.html#displays-for-lists-sets-and-dictionaries)
- [PEP 202: List Comprehensions](https://peps.python.org/pep-0202/)
- [PEP 572: Scope of the target](https://peps.python.org/pep-0572/#scope-of-the-target)