--- description: "CodeWiki Data science pitfalls and review checks" globs: [] alwaysApply: false --- # Data science rules This track covers more than one language, so no file globs are inferred. Apply these rules manually when they are relevant. - Sorting labels and values separately breaks record identity. Why: The bars still look orderly, but one category can be displayed with another category's value. Fix: Sort complete records or their indices, and verify the first and last marks with unique categories and a hand-checkable example. Source: [Data visualization](https://codewiki.com/datascience/visualization/) - Autoscaling can make every small multiple fill its panel while equal heights or colors represent different values. Why: Fix: Share numeric domains, ticks, and color normalization across comparable panels; label any separate local view with its range. Source: [Data visualization](https://codewiki.com/datascience/visualization/) - Connecting a line after dropping `None` or `NaN` implies continuous observations across the missing interval; replacing missing values with zero invents a plunge. Why: Fix: Preserve missing states and break segments. Interpolate only when domain rules justify it, and state the method and interval. Source: [Data visualization](https://codewiki.com/datascience/visualization/) - Red and green alone make success and failure indistinguishable for some readers and lose meaning in grayscale output. Why: Fix: Add text labels, shapes, line styles, or patterns, then inspect the artifact at final size, in grayscale, and through its keyboard path. Source: [Data visualization](https://codewiki.com/datascience/visualization/) - Do not assume this is safe: a mean bar without sample size or variation presents a sparse, unstable group as if it were as certain as a large sample. Why: Fix: Label `n`; show raw points, a distribution, or a defined uncertainty interval appropriate to the task; and disclose the aggregation rule. Source: [Data visualization](https://codewiki.com/datascience/visualization/) - Mixing `plt.plot()` with several `ax` objects in one function adds an artist to whichever Axes is current at that moment. Why: It may not be the panel a reader expects from the surrounding code. Source: [Matplotlib](https://codewiki.com/datascience/matplotlib/) - Do not assume this is safe: creating figures in a loop without closing them leaves the pyplot registry and artist trees holding objects. Why: A long-running notebook, worker, or report process accumulates memory, and Matplotlib eventually warns that too many figures are open. Source: [Matplotlib](https://codewiki.com/datascience/matplotlib/) - If each panel normalizes color independently, the same color can represent different values. Why: Generated code often gives each `scatter()` call a separate colorbar, producing a complete-looking figure that cannot be compared horizontally. Source: [Matplotlib](https://codewiki.com/datascience/matplotlib/) - Calling only `set_xticklabels()` attaches text to the current tick positions. Why: If the automatic locator later changes those positions, labels can drift and Matplotlib can emit a `FixedFormatter` warning. Source: [Matplotlib](https://codewiki.com/datascience/matplotlib/) - Truncating a bar chart's baseline to "show the difference" can make a small numerical change look large. Why: Conversely, forcing every chart to start at zero can flatten a line whose purpose is to show variation around a baseline. Source: [Matplotlib](https://codewiki.com/datascience/matplotlib/) - Hard-coding `TkAgg` or `QtAgg` in server code introduces a GUI dependency and a display requirement. Why: Forcing `Agg` in a desktop program that needs window interaction instead produces files without a GUI event loop. Source: [Matplotlib](https://codewiki.com/datascience/matplotlib/) - Do not treat "broadcasting succeeded" as "the axes aligned." Adding shapes `(100, 1)` and `(100,)` produces `(100, 100)`, even if the intent was to add one value per sample. Source: [NumPy](https://codewiki.com/datascience/numpy/) - Do not assume slices are always independent, or assuming every index returns a view. Why: Basic slices usually share memory, advanced indexes usually copy, and mixed indexing is harder to judge by appearance. Source: [NumPy](https://codewiki.com/datascience/numpy/) - Ignoring fixed-width integer overflow or writing a floating result back into an integer array. Why: Array operations follow NumPy's type rules and don't automatically gain the arbitrary precision of Python integers. Source: [NumPy](https://codewiki.com/datascience/numpy/) - Do not treat `np.empty()` as an array filled with empty values or zeros. Why: It can expose arbitrary bit patterns left in that memory block, so an unwritten position has no domain meaning. Source: [NumPy](https://codewiki.com/datascience/numpy/) - Calling global `np.random.seed()` inside a library function. Why: It resets legacy random state shared with other code in the process and makes tests depend on call order. Source: [NumPy](https://codewiki.com/datascience/numpy/) - Comparing floating calculations with `==`, or letting one `NaN` silently propagate through a whole reduction. Why: Binary floating-point rounding and missing-data policy are separate problems. Source: [NumPy](https://codewiki.com/datascience/numpy/) - Assigning after chained selection looks plausible but has no unambiguous single write target. Why: Under pandas 3.0 Copy-on-Write semantics, `df[df["paid"]]["status"] = "ready"` doesn't update the original `DataFrame`. Source: [pandas](https://codewiki.com/datascience/pandas-guide/) - Equal object lengths don't mean operations are positional. Why: When index labels differ or arrive in another order, automatic alignment can create missing values or pair values with the wrong business entities. Source: [pandas](https://codewiki.com/datascience/pandas-guide/) - Plain `bool` and integer dtypes can't represent every missing state, so inference may promote a column to floating point or generic objects. Why: The truth value of `pd.NA` is also ambiguous, and `if value:` raises an exception. Source: [pandas](https://codewiki.com/datascience/pandas-guide/) - Do not assume this is safe: a join without a cardinality check can expand as a Cartesian product when the right table has duplicate keys. Why: pandas also matches null keys on both sides, unlike the null join behavior of common SQL databases. Source: [pandas](https://codewiki.com/datascience/pandas-guide/) - By default, `groupby()` drops rows whose grouping key is missing. Why: The report still runs and looks tidy, but the grouped amount can be smaller than the input amount. Source: [pandas](https://codewiki.com/datascience/pandas-guide/) - Calling `read_parquet()` and then `.lazy()` looks like lazy I/O, but the read has already finished and the scan stage is no longer part of the query plan. Source: [Polars](https://codewiki.com/datascience/polars/) - One expression creates `revenue` inside `with_columns()`, while a sibling expression tries to read `pl.col("revenue")`. Why: Both expressions see the same input schema, where the new alias does not exist yet. Source: [Polars](https://codewiki.com/datascience/polars/) - Calling only `fill_null()` is treated as complete floating-point cleanup. Why: Any `NaN` values remain in the column and may propagate through later calculations. Source: [Polars](https://codewiki.com/datascience/polars/) - A join states only `on` and `how`, with no key cardinality. Why: If duplicate keys appear on the right, a legal many-to-many join multiplies orders and leaves an aggregate that looks plausible but is wrong. Source: [Polars](https://codewiki.com/datascience/polars/) - The current row order from a group or join is treated as an API contract. Why: A change in thread scheduling, input partitioning, or execution strategy can produce a different order in tests and exports. Source: [Polars](https://codewiki.com/datascience/polars/)