# map, filter and reduce

Source: https://codewiki.com/python/map-filter-reduce/

> - **what**: `map()` transforms each item, `filter()` keeps items that satisfy a condition, and `functools.reduce()` combines items from left to right into one result.
> - **trap**: `map()` and `filter()` return single-pass iterators; `filter(None, data)` also removes every false value, not just `None`.
> - **fix**: Make consumption explicit and call `list()` only when you need a list; give `reduce()` an initializer matching the result type, and define a length contract for multi-input `map()`.

## What it is and why it exists

`map()`, `filter()`, and `reduce()` are higher-order functions: they accept another function that describes the operation to perform on data. Each has a distinct role: `map()` changes items, `filter()` decides which items remain, and `reduce()` folds a series of items into one result. They operate on an iterable, so the input can be a list, generator, file object, or anything else that implements the iteration protocol.

These tools compose behavior directly when a transformation or predicate already exists. For example, `map(str.strip, lines)` is more compact than another loop whose only job is to call `strip()`, while `filter(is_valid, records)` keeps the rule in a named function. They don't replace every loop: a regular `for` loop is usually clearer when the operation needs several branches, exception recovery, or multiple side effects.

`map()` and `filter()` return an iterator, not a list. They use lazy evaluation: creating the object doesn't call the transformation or predicate; work starts when a consumer asks for the next item. This permits item-by-item processing, but it also means exceptions, logging, and side effects occur during consumption.

`reduce()` lives in `functools`; it isn't a built-in. It accepts a binary function, passes that function the current accumulated result and the next input, then uses the returned value in the next step. Because it returns one final value, it fits reductions that genuinely have one accumulator and no more specific built-in operation.

In Python, a list comprehension or generator expression often expresses the same work. Choose by result shape and readability: an existing callable suits `map()` or `filter()`, a short expression suits a comprehension, and named reductions such as `sum()`, `any()`, and `max()` are usually more direct than general-purpose `reduce()`.

## How it works

`map(function, iterable, *iterables)` pulls one item from every input on each step, then calls `function`. With one input, the function receives one argument; with three inputs, it must receive three. Python 3.14 also provides the `strict` keyword argument, which controls whether unequal multi-input lengths stop silently or raise an error.

`filter(function, iterable)` pulls an item from its single input and applies truth-value testing to the function's result. It yields the original item when the result is true and keeps pulling when it is false. `filter()` doesn't yield the predicate result or modify the item it keeps.

`filter(None, iterable)` is a special form. It tests each item itself, so it discards `None`, `False`, numeric zero, empty strings, and empty containers. It fits a contract that says “keep only truthy values,” not a data model in which `None` is the only missing-value marker.

`reduce(function, iterable, initial)` works as a state transition: determine the accumulator, then repeatedly calculate `accumulator = function(accumulator, item)`. If `initial` is present, it is the first accumulator and the return value for empty input. If it is omitted, the first input becomes the accumulator and empty input raises `TypeError`.

The important differences are the consumption model and output shape:

| Operation | What the user function receives | What it returns | Empty input |
| --- | --- | --- | --- |
| `map()` | One item from each input | A lazy iterator of results | Yields nothing |
| `filter()` | One item | A lazy iterator of retained original items | Yields nothing |
| `reduce()` | The accumulator and next item | One final value | Depends on whether an initializer exists |

### Pull-driven execution

Downstream consumers drive `map()` and `filter()`. Calling `next()`, entering a `for` loop, constructing `list()`, or passing them to `sum()` pulls items. For a pipeline shaped as `filter(predicate, map(transform, source))`, each requested output repeats “pull source, transform, test” until it finds a retained item or exhausts the input.

This mechanism doesn't cache results already yielded. After a `map` or `filter` object has been traversed, a second traversal sees only the remaining items, usually none. If repeated reads are required, materialize a list at a clear ownership boundary instead of letting several callers compete for one iterator.

Laziness also moves the failure point. Even if a transformation will raise on the third input, creating the `map` object succeeds; the exception appears only after two items have been consumed and a consumer requests the third. Tests must consume through the target branch rather than merely assert that iterator construction succeeded.

### Multi-input `map()`

By default, multi-input `map()` ends when the shortest input is exhausted. This resembles the default length rule of `zip()` and fits an explicit contract that permits extra data to be ignored. When unequal lengths indicate corruption, Python 3.14 code should use `strict=True` and tests must actually consume the iterator because the length error is also lazy.

`strict=True` doesn't read every input up front to compare lengths. It still processes item by item and raises `ValueError` only when one input ends while another still has an item. Infinite and single-pass inputs can therefore still participate in `map()`, but the caller needs a boundary that handles consumption-time errors.

### The accumulator contract

The reduction function's return type must be suitable as the next call's left argument. If the initializer is a dictionary, every invocation should return a dictionary representing the next state; if the first invocation returns `None`, the next one receives `None`. This is why putting `dict.update()` directly in a lambda often fails: it mutates the dictionary in place and returns `None`.

The initializer also defines empty-input behavior and the result type. Counts usually begin with integer `0`, while field collection might begin with an empty tuple or dictionary. Python 3.14 permits `reduce(function, data, initial=seed)`; older versions accept only a third positional argument, so code supporting older runtimes still commonly writes `reduce(function, data, seed)`.

## Examples

These four examples progress through transformation and selection, lazy consumption, multi-input pairing, and an explicit reduction. Every output comes from local Python 3.12.13; the examples use behavior that also applies to the verified Python 3.14 target.

### Transforming paid orders

The input records store money in cents so this example doesn't introduce decimal-rounding concerns. The predicate retains paid orders, then the transformation creates display strings.

<!-- quick -->

```python
# file: paid_receipts.py
orders = [
    {"id": "A-100", "status": "paid", "cents": 1990},
    {"id": "A-101", "status": "pending", "cents": 1250},
    {"id": "A-102", "status": "paid", "cents": 800},
]


def is_paid(order):
    return order["status"] == "paid"


def to_receipt(order):
    return f'{order["id"]}: {order["cents"]} cents'


paid_orders = filter(is_paid, orders)
receipts = list(map(to_receipt, paid_orders))

print(*receipts, sep="\n")
```

```text
A-100: 1990 cents
A-102: 800 cents
```


<!-- /quick -->

`filter()` passes through the original order dictionaries; only `map()` turns them into strings. `list()` marks the materialization point explicitly. If the downstream code only writes each receipt to a file, it can iterate the corresponding `map` object directly instead of building a list first.

The order also states the business intent: pending orders never enter the formatting function. If transformation itself validates records and may produce a missing-value marker, define that marker's type explicitly instead of casually adding `filter(None, ...)`.

### Observing a lazy pipeline

Both the transformation and predicate print a call trace. The first `next()` pulls only as far as the first error event; the later `list()` consumes the remaining input.

```python
# file: lazy_events.py
event_lines = [
    "INFO:started",
    "ERROR:disk full",
    "WARNING:retrying",
    "ERROR:timeout",
]


def parse_event(line):
    print(f"parse {line}")
    level, message = line.split(":", 1)
    return {"level": level, "message": message}


def is_error(event):
    print(f'test {event["level"]}')
    return event["level"] == "ERROR"


errors = filter(is_error, map(parse_event, event_lines))
print("pipeline ready")
print(next(errors))
print(list(errors))
print(list(errors))
```

```text
pipeline ready
parse INFO:started
test INFO
parse ERROR:disk full
test ERROR
{'level': 'ERROR', 'message': 'disk full'}
parse WARNING:retrying
test WARNING
parse ERROR:timeout
test ERROR
[{'level': 'ERROR', 'message': 'timeout'}]
[]
```

`pipeline ready` prints before either function runs, proving that constructing the pipeline did no work. Inputs before the first matching item still require processing, so one `next()` can trigger several transformations and predicate calls. The final `list(errors)` is empty because the same iterator is exhausted.

The trace also shows that the consumer controls logging order. If generated code returns the iterator to another layer, transformation exceptions and side effects cross the original function boundary; the caller must know that it has taken ownership of consumption.

### Pairing multiple inputs

`operator.mul` is already a binary function, so it can be passed directly to `map()`. The inputs have different lengths, and default mode therefore produces only two results.

```python
# file: line_totals.py
from operator import mul

unit_prices = [1250, 800, 500]
quantities = [2, 3]

line_totals = list(map(mul, unit_prices, quantities))

print(line_totals)
print(unit_prices[len(quantities):])
```

```text
[2500, 2400]
[500]
```

The `[500]` in the output is the unpaired input, not a result from `map()`. If it represents a missing quantity, Python 3.14 code should use `map(mul, unit_prices, quantities, strict=True)` instead. Passing that result to `list()` raises `ValueError`, making the inconsistent data fail explicitly.

Don't mistake “parallel inputs” here for concurrent execution. `map()` calls the function one item at a time in the calling thread. Work that needs threads, processes, or asynchronous concurrency belongs to APIs for the corresponding execution model.

### Defining a reduction with an initializer

This reduction accumulates payment statuses into new dictionaries. The initial empty dictionary both handles empty input and makes the first left argument the same type as later ones.

```python
# file: status_counts.py
from functools import reduce

statuses = ["paid", "paid", "refunded", "paid"]


def add_status(counts, status):
    return {
        **counts,
        status: counts.get(status, 0) + 1,
    }


counts = reduce(add_status, statuses, {})
empty_counts = reduce(add_status, [], {})

print(counts)
print(empty_counts)
```

```text
{'paid': 3, 'refunded': 1}
{}
```

`add_status()` returns the complete accumulator needed by the next step and doesn't depend on mutable external state. This example deliberately demonstrates a general reduction. In production, a named `for` loop is usually easier to inspect and extend when grouping requires complex rules, several fields, or in-place updates.

If the task is only to total amounts, write `sum(amounts, start=0)` rather than `reduce(lambda left, right: left + right, amounts, 0)`. The named operation communicates intent and its empty-input rule, so readers don't have to derive the binary function's meaning.

## Pitfalls

### Treating a lazy iterator as a reusable container

> **Pitfall:** Debugging code calls `list(result)`, then application code iterates the same `map` or `filter` object; the second consumer gets only what remains. Checking that the object was constructed also doesn't trigger exceptions in the transformation function.

**Fix:** give one layer ownership of consumption. Materialize once and share the list when repeated reads are required. For streaming, pass the iterator and test partial consumption, complete consumption, and consumption-time exceptions.

### Using `filter(None, ...)` to remove missing values

> **Pitfall:** `filter(None, values)` removes every false value, including legitimate `0`, `False`, `""`, and empty containers. Generated cleaning code often collapses “missing” and “business value is zero” into one condition.

**Fix:** write the exact predicate when only a missing marker should go, such as `filter(lambda value: value is not None, values)`. Define whether the field permits an empty string or zero before deciding that truth-value testing matches the contract.

### Ignoring unequal multi-input lengths

> **Pitfall:** By default, `map(function, left, right)` stops when the shorter input ends, with no notice about the longer input's tail. When the columns must correspond row for row, this turns a missing row into silent data loss.

**Fix:** use `strict=True` on Python 3.14 and consume the pipeline so the check runs. When unequal lengths are permitted, state the fill rule and consider `itertools.zip_longest()` instead of making readers infer the default behavior.

### Omitting an initializer that the contract needs

> **Pitfall:** `reduce()` without an initializer raises `TypeError` on empty input and uses the first item directly as the accumulator. When the item type and result type differ, the reduction function may not expose the mismatch until the second item arrives.

**Fix:** define empty-input behavior from the domain and provide a seed of the result type. When no sensible identity exists, don't invent a default. Reject empty input first or choose an interface that can express “no result.”

### Returning the result of an in-place method

> **Pitfall:** `reduce(lambda acc, item: acc.update(item), mappings, {})` returns `None` on its first step, so the next step can't continue. The same problem appears with lambdas that return `list.append()`, because these in-place methods conventionally return `None`.

**Fix:** a reduction function must explicitly return the accumulator for the next step. If the design intentionally mutates one container repeatedly, a normal loop usually exposes the mutation and return boundary more clearly than hiding the side effect in `reduce()`.

### Hiding a named operation behind general reduction

> **Pitfall:** Reimplementing `sum()`, `min()`, `max()`, `any()`, `all()`, or `str.join()` with `reduce()` forces readers to parse a custom binary function. Repeatedly joining lists with `left + right` also hides the data movement inside one lambda.

**Fix:** prefer the built-in that names the result. Use `itertools.accumulate()` when you need every intermediate total. Keep `reduce()` only when the accumulator is genuinely custom and the binary transition remains easy to explain.

<!-- deep -->

## Choosing the right form

`map()` and a comprehension can both express item-wise transformation, but they emphasize different information. `map(normalize, records)` puts an already named transformation in view, while `[record.name for record in records]` keeps a short expression beside the iteration structure. When the result should remain lazy, replace the brackets with a generator expression instead of nesting lambdas just to obtain laziness.

Likewise, `filter(is_active, users)` works well with a reusable named predicate, while `[user for user in users if user.active]` suits a short condition. Putting selection and transformation into one comprehension is often easier to read in execution order than `map(lambda ..., filter(lambda ..., data))`. This is a readability and result-shape choice, not a reason to repeat unmeasured speed claims.

When the logic must record rejection reasons, the single “keep or discard” result from `filter()` is usually insufficient. A regular loop can collect accepted items and errors together and handle exceptions beside the relevant branch. Don't preserve a functional appearance by making a predicate mutate an external list; that splits output between the explicit iterator and a hidden side-effect channel.

Look for a domain name before choosing a reduction. Use `sum()` for numeric totals, `any()` for existence, `all()` for universal conditions, and `separator.join(parts)` for strings. These functions define clear output shapes, and some have specialized short-circuit or empty-input behavior that a mechanical rewrite to `reduce()` doesn't necessarily preserve.

`itertools.accumulate()` follows a similar binary accumulation idea but has a different output. `reduce()` returns only the final value, whereas `accumulate()` lazily yields intermediate values for balance histories, running maxima, or progressive state. Ask whether the caller needs a final summary or the complete trajectory before choosing the interface.

### Boundaries between transformation, selection, and reduction

A transformation should map each input to one output. When transformation can fail, it can raise a precise exception or return a result object carrying status; don't make `None` mean both parse failure and a valid value. A layer that understands the domain contract should decide whether failed results are then discarded.

A predicate's result undergoes truth-value testing and needn't have the exact type `bool`. A nonempty string can therefore retain an item, but returning an explicit Boolean expression makes type and intent easier to review. For third-party objects with special truth rules, also confirm whether truth testing can raise or be ambiguous.

A reduction function defines both the state type and the state transition. A useful review notation is `(Accumulator, Item) -> Accumulator`. If one branch returns another type, failure may wait until the next step, so tests should cover every branch rather than inspect only the final example.

### Ownership contracts for pipeline APIs

A function that returns a lazy iterator also transfers consumption responsibility to its caller. Its documentation should say that the result is single-pass, when the underlying input closes, and whether errors occur during construction or iteration. A return type of only `Iterable` can hide single-pass semantics; `Iterator` is often more precise when that is what the implementation returns.

When a function converts the result to a list internally, that function owns complete consumption and the exception boundary. The call is simpler, but all work must finish before it returns. Choosing an iterator or container isn't a local syntax preference; it is an API promise about timing, ownership, and failure location.

Common return shapes have distinct contracts:

| Return shape | What the caller receives | Suitable contract |
| --- | --- | --- |
| `Iterator[T]` | Single-pass values produced on demand | The caller owns consumption |
| `Iterable[T]` | An object from which an iterator can be obtained | Repeatability needs separate documentation |
| `list[T]` | A fully materialized result | Processing finishes before return |
| One accumulated value | Final state after reduction | Processing finishes at the call site |

Don't return a lazy object that depends on an already closed resource. If a function creates `map()` inside `with open(...)` and the caller iterates after the context exits, the first read may reach a closed file. Either materialize inside the context or put resource lifetime inside a generator that owns opening and closing.

Consumption ownership also determines who can retry. A partly consumed file or generator may not restart, and calling `list(iterator)` again simply continues from the current position. If retry requires a full replay, retain a factory that can recreate the input or a stable data source rather than only the current iterator.

Trace a pipeline boundary in this order during review:

1. Mark who creates the original iterable and whether it is repeatable.
2. Mark which functions and input references each `map()` and `filter()` retains.
3. Find the first `next()`, `list()`, loop, or reduction that starts consumption.
4. State who handles remaining input after partial consumption, failure, or early exit.

This record exposes more than the final type alone. Two functions may both declare `Iterable[T]`; one returns a fresh list, while the other returns a single-pass iterator tied to an open cursor. They impose completely different lifetime duties on the caller.

### Testing boundary inputs

Tests for `map()` and `filter()` should cover empty input, one item, and an item that makes the user function fail. A laziness test should also separate construction from consumption: first prove construction performs no call, then check what `next()` and full traversal invoke. That verifies real timing instead of only the final collection.

Multi-input `map()` additionally needs equal lengths, a shorter left input, and a shorter right input. Default mode should prove that truncation is intentional; strict mode should prove that consumption reaches a `ValueError` at the boundary. Testing only two equal lists says nothing about real data drift.

`reduce()` needs empty, single-item, and multi-item inputs. With an initializer, assert the empty result; without one, confirm that failure on empty input is the public contract. For a mutable accumulator, run the reduction twice and check whether the two results accidentally share state.

Choose test data that makes transformed versus original and retained versus removed values visibly different. All-positive, nonzero numbers won't expose misuse of `filter(None, ...)`, and equal input lengths won't expose default truncation. Boundary fixtures should target semantic risks instead of repeating the happy path.

### Side effects and observable order

Python preserves input iteration order, and `map()` and `filter()` call user functions in that order. Laziness doesn't mean out of order; it means the consumer advances the call time. If a user function logs, modifies a database, or sends a message, the number and location of consumption operations change external behavior.

Writing `list(map(send, messages))` solely for side effects builds an unused result list and hides the real purpose behind a transformation interface. A regular loop clearly says “perform these actions in order” and leaves room for per-item retries and error handling. `map()` is best when its return values matter; it isn't a task scheduler.

When two consumers alternate calls to the same iterator, they divide items according to call order; each doesn't receive its own copy. If both components need the entire dataset, pass a reusable container or create independent iterators. `itertools.tee()` has its own buffering and ownership semantics, so confirm that those semantics fit the input size and consumption pattern first.

## Edge semantics in Python 3.14

Python 3.14 adds the `strict` parameter to `map()`. The default `False` keeps the established shortest-input behavior; with `True`, any input ending early triggers `ValueError` during iteration. This is a data-alignment contract, not a concurrency or type-checking switch.

Strict mode is most useful for data that must correspond one to one, such as quantities and prices or column names and field values. It can't prove that values at each position correspond semantically; it proves only that all inputs end together. Identifiers, schema validation, or domain constraints still have to establish content alignment.

Python 3.14 also permits the `reduce()` initializer as the keyword argument `initial=`. A keyword can highlight the seed's purpose in a long call, but a library supporting older Python versions must obey its declared minimum version. The runnable examples here use the third positional argument, so they work on local Python 3.12.13 and the Python 3.14 target.

Without an initializer, a single-item input returns that item unchanged and never calls the reduction function; empty input fails. With an initializer, empty input returns the initializer itself. These branches can affect type, object identity, and mutable-seed sharing and deserve separate tests.

Don't define one mutable initial object as a module constant and mutate it across reductions. Separate calls would then share historical state, a problem of object ownership rather than `reduce()` itself. Create a new seed for every call or have the reduction function return new state to keep the boundary clear.

### Consumption-time errors

Like ordinary transformation errors, a length mismatch from `map(strict=True)` can appear after several items have already been yielded. If the caller writes to an external system while iterating, writes before failure may already have happened. Add validation or a transaction at the business boundary when the operation needs all-or-nothing semantics; strict length checking doesn't roll back earlier actions.

`map()` and `filter()` don't wrap exceptions from user functions. The original exception emerges from the current consumption operation, and the stack includes both consumer and transformation. Put handling at the layer that knows whether to skip, retry, or stop instead of silently filtering bad data under a broad exception.

`reduce()` returns its final value immediately, so exceptions from its user function emerge directly from the `reduce()` call. Even so, that function may have run several times before failure, and any external mutations remain. Keeping the reduction function free of side effects makes this behavior easier to reason about.

<!-- /deep -->

[Checkpoint: python/map-filter-reduce](https://codewiki.com/python/map-filter-reduce/#checkpoint)

## Further reading

- [Python 3.14 built-in function: `map()`](https://docs.python.org/3.14/library/functions.html#map)
- [Python 3.14 built-in function: `filter()`](https://docs.python.org/3.14/library/functions.html#filter)
- [Python 3.14 `functools.reduce()`](https://docs.python.org/3.14/library/functools.html#functools.reduce)
- [Python 3.14 Functional Programming HOWTO](https://docs.python.org/3.14/howto/functional.html)
- [Python 3.14 `itertools.accumulate()`](https://docs.python.org/3.14/library/itertools.html#itertools.accumulate)