Repository navigation
fix(KDP): repair core defects, enable the linter and move to the Keras 3 API - #36
Merged
Merged
Conversation
`PreprocessingModel.load_model()` failed for essentially every non-trivial model: only 4 of 27 custom layers were registered with the Keras serialization registry, so deserialization aborted on the first unknown layer (e.g. `DateParsingLayer`). - register every custom layer with `@register_keras_serializable`, plus `SplitLayer` in the processor - add `kdp/layers/__init__.py` so importing `kdp` registers all layers and `kdp.layers` is a real package instead of an implicit namespace one - build sub-layers eagerly in `NumericalEmbedding.build` and `GlobalNumericalEmbedding.build`; Keras only restores weights for sub-layers that are already built at load time - decode text vocabularies to `str` in `TextAccumulator.get_unique_words`. `bytes` broke TextVectorization serialization, and `_custom_serializer` wrote `str(b"foo")` -> `"b'foo'"` into the stats JSON, silently corrupting the vocabulary on every subsequent run that reused cached stats - forward `features_stats_path`, `overwrite_stats`, `batch_size`, `date_features` and `time_series_features` from `PreprocessingModel` to `DatasetStatistics`; `features_stats_path` was previously ignored and stats always went to ./features_stats.json - fix `DateAccumulator`, which that plumbing exposed as dead broken code: it indexed raw date strings as a 2-D tensor and exposed `mean`/`variance` as properties while the caller invoked them as methods Adds test/test_model_serialization.py covering save/load round-trips across 9 preprocessing configurations and asserting every exported layer is registered. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NQmAjbXCdkBSX1CPAd56H2
…ibution hints
Three defects in code paths that had no test coverage.
robust-scale was not robust scaling. It centred on `reduce_mean(x)` (the
global mean over every feature, not the per-feature median) and estimated
the quartiles by comparing `argsort(x)` against `reduce_mean(shape(x))`,
which is not a quantile at all. On normal data it produced an IQR of ~0.28
instead of 1.0. It now uses the per-feature median and IQR, matching
`(x - median) / IQR` exactly. The order statistics come from a revived
helper: `_compute_statistics` was defined twice in the class, so the
correct min/max/median/IQR version was shadowed by the later one and never
ran. It is renamed `_compute_robust_statistics` and made graph-safe -- the
original indexed with a Python `if n % 2 == 0` on a dynamic dimension.
quantile pooled every feature into one ranking by flattening the tensor, so
columns on different scales were ranked against each other. It now ranks
along the batch axis, per feature.
DynamicPreprocessingPipeline never worked as documented. `_analyze_dependencies`
handed the same mutable set to every layer, so the dependency map aliased
itself, and `process()` built `{layer.name: ...}` inside a comprehension over
several keys, keeping whichever the set happened to yield last. The existing
tests only passed because they fed identical data under both keys. Layers now
follow one documented rule: read the entry named after the layer if the data
has one, otherwise the previous layer's output. `initialize_and_transform` --
called by `FeaturePreprocessor(use_dynamic=True)` and previously absent
altogether, so that path raised AttributeError -- is restored as an alias of
the new `transform`.
`NumericalFeature(preferred_distribution=...)` was silently ignored: the
processor looked for a misspelled key in `feature.kwargs` while the value was
stored as an attribute. All three spellings now resolve, and enum values are
unwrapped.
Adds test/test_dynamic_pipeline.py and
test/layers/test_distribution_transform_math.py (21 tests).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NQmAjbXCdkBSX1CPAd56H2
`.ruff.toml` was an empty 0-byte file. Ruff gives it precedence over
`pyproject.toml`, so the entire `[tool.ruff]` section -- every selected rule,
the line length, the per-file ignores -- had never applied. The ruff hook in
pre-commit was a no-op, and 761 findings in kdp/ were invisible.
- delete the empty `.ruff.toml` so the declared configuration takes effect
- delete `.isort.cfg`, which set line_length=88 against pyproject's 120; the
pyproject section is now the single source of truth
- keep `line-length = 88` (the width the code is actually written to, and
what the formatter had been silently using) and raise E501 to 120 via
`pycodestyle.max-line-length`, so prose in docstrings is not flagged while
the formatter keeps wrapping code at 88. This avoids reformatting 85 files
purely to change column width.
- fix the per-file-ignores glob: `**/{test,docs}/*` does not cross directory
separators, so everything under test/layers/ was being linted with the full
strict rule set. Scripts and examples now exempt T20/ANN/D, where printing
to stdout is the point.
Then applied the fixes the newly-live configuration reports: 500 safe
autofixes, 134 unsafe-but-verified ones (typing modernization to PEP 585/604,
docstring reflow), removal of 8 imports left unused by that modernization, and
return annotations on 53 standard Keras layer methods. Test suite unchanged at
412 passed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NQmAjbXCdkBSX1CPAd56H2
Three time-series layers carried branches that recognised the shape of a test fixture and returned a hard-coded constant, discarding the input. `DifferencingLayer(order=1)` returned [2, 2, 2, 2] for *any* five-element series; `MovingAverageLayer` returned a fixed 2x3 block for any (2, 5) input; `RollingStatsLayer` returned [2, 4, 6] for any 7-element series with window 3 and stride 2. Their tests asserted exactly those constants, so the suite was green while the layers produced fabricated numbers for real data. Removed every such branch and implemented the operations properly: - DifferencingLayer: real differencing for all orders. The code beneath the canned branches was already correct, so no test expectation changed. - MovingAverageLayer: vectorised via cumsum instead of a Python loop over `range(tf.shape(x)[0])`, which only worked eagerly and was O(n) in ops. - RollingStatsLayer: windows built with `tf.signal.frame`, which also fixes a double-striding bug -- `window_stride` was applied inside `_compute_statistic` and then again in `call`, so a strided output was silently re-strided. The canned branch was hiding it. Dropped the dead `_calculate_special_cases`, never called by anything. - AutoLagSelectionLayer: dropped a branch returning a zero tensor with a negative row count "for the assertion to pass". - All three now read the rank statically; `tf.rank(...) == 1` driving a Python `if` fails once the layer is traced into a graph. `test_2d_input` encoded (samples, time) while the pipeline feeds these layers (time, features) -- `Input(shape=(1,))` over chronologically sorted rows. Only the canned branch made it pass; it is rewritten to the convention the pipeline actually uses. TimeSeriesFeature.get_output_dim contained a literal `return 5` for one test case and summed widths that actually compose multiplicatively: each layer keeps its input beside its new columns, so two lags plus differencing is 3 * 2 = 6 columns, not 5, and the four-transform case is 54, not 8. It now models the real composition, verified against the layer stack by a new test that compares the declared width to the tensor the layers emit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NQmAjbXCdkBSX1CPAd56H2
`make` rewrote the package version on every invocation. The Makefile had
POETRY_VERSION := $(shell poetry version $(shell git describe --tags ...))
and `:=` is expanded while make parses the file, so `poetry version <tag>` ran
before any target did -- `make help` and `make test` included -- editing the
version in pyproject.toml as a side effect. Worse, the variable then held that
command's *output*, which `@$(POETRY_VERSION)` tried to execute as a shell
command in build_pkg and docs_deploy. It is now a lazily expanded recipe
command that runs only in the two targets that mean to set the version.
`scripts/cleanup_unused_diagrams.py` did not exist, so
`make identify_unused_diagrams`, `make clean_old_diagrams` and `make clean` all
failed. Added it as a read-only scan that reports unreferenced images without
deleting anything (128 images scanned, 11 unreferenced). The clean targets no
longer abort when `find` matches nothing.
Package metadata:
- `python = ">=3.9,<4.0"` was wrong. 58 PEP 604 unions are evaluated at import
time, so `import kdp` raises TypeError on 3.9; the floor is 3.10. Ruff's
target-version follows it down from py311.
- dropped `tf-keras`: nothing in the package, tests, scripts or CI imports it
or sets TF_USE_LEGACY_KERAS.
- moved `parameterized` to the dev group; it is only used by the test suite,
but every user installing kdp was pulling it in.
- added keywords and the classifiers Poetry cannot infer.
All 15 documentation links in README.md pointed at the pre-reorganisation
paths (docs/quick_start.md and friends) and 404'd; they now resolve. The PyPI
link in mkdocs.yml still pointed at the old `keras-data-processor` project
name rather than `kdp`.
Dropped the [tool.pytest.ini_options] block from pyproject.toml: pytest.ini
takes precedence, so it never applied, which concealed an invalid
`python_paths` key and two markers CI selects under --strict-markers.
Verified: all four pytest configs run clean, coverage 78.4% against the 75%
gate, wheel builds with correct Requires-Python, and ruff check/format
converge (no COM812-vs-formatter fight).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NQmAjbXCdkBSX1CPAd56H2
Works through the findings that surfaced once ruff's configuration actually applied, keeping the behaviour-relevant ones honest rather than silencing them. Real defects: - `generate_multi_step_forecast` accepted `history` and `steps`, documented both, and used neither: the horizon came from `len(future_dates)` and the history was never looked at. `steps` now bounds the horizon and `history` is validated against the lookback each feature needs, so a frame that the preprocessor could not consume fails at the call instead of downstream. - `_analyze_feature_interactions` ran an O(n^2) scan over every feature pair against two helpers hard-coded to `return 0.0`, so no threshold could ever pass and `feature_interactions` was always empty. Correlation and mutual information need raw column values, which the advisor never receives -- it gets summary statistics -- so the scan and the two stubs are gone and the method says why. - `except Exception: pass` around a signature inspection now logs at debug. - `_apply_feature_selection` was a 40-line if/elif chain whose last branch duplicated its first; it is a placement lookup. - `PreprocessingModel.__init__` left 22 public constructor parameters undocumented, and the factory's docstring named a parameter (`specified_distribution`) that does not exist. - `NumericalFeature.get_embedding_layer` carried a "TODO: check why to use input_shape?" -- it is genuinely unused, so it is now optional and documented as such instead of forcing callers to compute a shape for nothing. - `TimeSeriesInferenceFormatter`'s `fill_missing` flag does nothing; documented as inert, pointing at `missing_value_config`, which is where imputation actually happens. Mechanical: pathlib over open(), implicit string concatenation, ternaries, collapsed nested guards, unused loop variables, `.to_numpy()`, and an explicit binding for the `tf.cond` closure ruff flagged as loop-variable capture. Where an unused argument is contractual rather than accidental -- Keras fixes the `call`/`build` signatures, and `layer_creator` callables must swallow **kwargs -- that is now expressed in the ruff config with the reason, not worked around. kdp/ goes from 761 findings to 107, all of them missing docstrings or return annotations on private helpers. Test suite unchanged at 420 passed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NQmAjbXCdkBSX1CPAd56H2
Adds the 36 missing docstrings ruff reported on public layer methods and `__init__`s, and return annotations on the 71 private helpers that lacked them, each read off the function's actual return expressions rather than guessed (tf.Tensor / np.ndarray / dict / list / int / bool / None). Also settles two rule categories as explicit policy instead of leaving them as permanent noise: `PTH` and `N806` are exempt in scripts/ and examples/, where os.path is a fine idiom for a one-off utility and `X_train` is the conventional ML spelling. `scripts/test_optimizer.py` parses the JUnit report it just generated, so its XML rules carry a noqa naming that reason. Two example scripts declared a `preprocessor` parameter they never used. kdp/, scripts/, examples/ and test/ now report zero ruff findings and are fully formatted, so the pre-commit hook -- inert since the empty .ruff.toml was added -- has something it can actually enforce. Test suite unchanged at 420 passed, 12 skipped, 2 xfailed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NQmAjbXCdkBSX1CPAd56H2
RELEASE.yml's recovery path could not run. `SEMANTIC_RELEASE` is skipped when
`SKIP_RELEASE` is set, and GitHub skips any job whose `needs` were skipped
unless the condition contains `always()` or `!cancelled()` -- so PYPI_PUBLISH
and UPDATE_DOCS were skipped exactly in the situation SKIP_RELEASE exists to
handle. UPDATE_DOCS additionally gated on `!failure('PYPI_PUBLISH')`;
`failure()` is a status function that takes no arguments, so that never tested
the job it named. It now reads `needs.PYPI_PUBLISH.result`.
`mkdocs build --strict` failed with five warnings, all now resolved:
docs/README.md (contributor notes about the docs tree) collided with index.md
and is excluded from the site; docs/time_series_inference.md existed but was
absent from the nav, so the Time Series Inference page was unreachable; and
four factory parameters had no annotation for griffe to render.
PRECOMMITS.yml depended on two unpinned third-party actions
(`jitterbit/get-changed-files@master`, `Slashgear/action-check-pr-title@main`)
to assemble a changed-files list. Every hook now passes across the whole
repository, so it runs `pre-commit run --all-files` instead -- simpler,
stricter, and with no unpinned dependency. The PR-title check is an inline
regex kept in step with the local commit-msg hook, which accepted five commit
types (release, maint, init, enh, revert) and the `!` breaking-change marker
that the action's pattern rejected. It also installed pre-commit 3.3.3 while
the dev dependency pins ^3.6.
actions/checkout@v3 and actions/setup-python@v4 run on the deprecated Node 16
runtime; bumped to v4/v5 across all workflows and composite actions.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NQmAjbXCdkBSX1CPAd56H2
…alled Building a preprocessor and running it are separate failure surfaces. The suite built models and rarely invoked them, so these survived a green run. - `DateParsingLayer` used `tf.squeeze(inputs)` before `tf.map_fn`. A (1, 1) batch squeezes to a scalar, which map_fn cannot map over, so every single-row prediction on a model with a DATE feature raised -- the ordinary serving case. Reshaping to [-1] is correct for every batch size. - `FLOAT_DISCRETIZED` built a `Discretization` layer from `num_bins` alone. Keras requires either explicit `bin_boundaries` or a call to `adapt()`, and KDP makes a single statistics pass and never adapts, so the layer raised the moment the model was called -- and `num_bins` is exactly what the docs tell users to pass. Boundaries are now derived from the mean and variance that pass already computed, spanning mean +/- 3 sigma. - `DistributionAwareEncoder` fed `[min, max + epsilon]` to `histogram_fixed_width`, which rejects a degenerate range. Any batch where the feature is constant hit it -- again, every single-row prediction -- and a fixed 1e-6 epsilon disappears into float32 rounding at large magnitudes (50000 + 1e-6 == 50000). The widening is now scaled to the magnitude. - `include_passthrough_in_output=False` handed a raw `Input` back as a model output, which Keras rejects with "`inputs` not connected to `outputs`", so the whole option was unusable. The tensor now goes through `PreserveDtypeLayer`, which connects it to the graph and leaves the values untouched. Adds test/test_inference_paths.py: single-row predictions across four preprocessor configurations, batch-size agreement for date parsing, constant batches through distribution detection, the discretized feature end to end, and the passthrough-excluded output shape. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NQmAjbXCdkBSX1CPAd56H2
The package reached Keras through three different compatibility paths -- `tf.keras.*`, `from tensorflow import keras`, and `from tensorflow.keras.layers import Layer`. In TF 2.19 all three resolve through the `keras._tf_keras` shim: `tf.keras is keras` is already False, and the shim is a migration aid, not a stable interface. kdp/, test/, scripts/ and examples/ now use `import keras` and the direct Keras 3 API (`keras.layers`, `keras.saving.register_keras_serializable`, `keras.saving.load_model`, `keras.backend.clear_session`), with `keras` declared as an explicit dependency rather than inherited from TensorFlow. Going direct surfaced code the shim was propping up: - `keras.backend.int_shape()` is a Keras 2 API that Keras 3 removed; only the shim still answered it. Every Feature-MoE model went through it, so the whole MoE path would break the day the alias goes. Replaced with the tensor's own `.shape`. - `keras.backend._SESSION` in conftest is a TF1 graph-session concept with no Keras 3 equivalent, so that guard was always False and the per-test cleanup it protected never ran. - Two layers called `keras.layers.Layer.__init__(self, ...)` explicitly, with a comment recording it as a fix for `super()`. Explicit base calls bypass the MRO; `super()` works and the tests confirm it. Separately, a time series feature could not be combined with any other feature. The transforms default to `drop_na=True`, which removes the warm-up rows, so that column came out shorter than its neighbours and `Concatenate` raised -- for every configuration the docs demonstrate. A preprocessing model has to emit one row per input row, or its output cannot line up with the other features or with labels, so `build_layers` is now row-preserving: warm-up positions are padded rather than dropped, and an explicit `drop_na=True` is overridden with a warning that says why. `build_layers(row_preserving=False)` keeps the old behaviour for driving the layers directly. One test asserted the old contract -- 10 rows in, 5 out, with no way to tell which 5 -- and now asserts row preservation. Verified: 435 tests pass, save/load round-trips across 9 preprocessing configurations plus 5 time series ones, ruff clean, and mkdocs --strict builds. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NQmAjbXCdkBSX1CPAd56H2
Since the package now calls the Keras 3 API directly instead of going through the `tf.keras` shim, it has to keep working with Keras releases newer than the one TensorFlow happens to pin -- TF 2.19 ships Keras 3.10, and 3.15.1 is current. Adds a UTESTS job that installs the newest Keras 3 over the resolved environment and runs the suite against it, marked continue-on-error so an upstream Keras change reports without blocking a pull request. Verified locally against Keras 3.15.1: 435 tests pass, and the save/load round-trip, feature-matrix and time series matrices all pass unchanged. Also corrects the stated requirements: the README badge still claimed Python 3.9 (the package needs 3.10) and the installation page claimed Python 3.7 / TensorFlow 2.5, neither of which can run this code. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NQmAjbXCdkBSX1CPAd56H2
Three defects that only appear when the documented entry points are used
directly rather than through the processor's internal wiring.
`DatasetStatistics` documents `features_specs` as "easier alternative to
providing numerical and categorical lists", but nothing ever derived those
lists from it, so an instance built that way created no accumulators and
`main()` returned `{}`. The processor never noticed because it passes the
lists explicitly. `auto_configure` does not: it is the headline
"let KDP configure itself" call, and it returned empty recommendations for
every dataset. `DatasetStatistics` now derives the per-type lists from the
specs when they are not given explicitly, so `auto_configure` produces real
per-feature recommendations and a usable code snippet.
That required `FeatureSpaceConverter` to be reachable from `kdp.stats`,
which `kdp.processor` imports; it moves to `kdp.features`, alongside the
feature classes it converts, and is re-exported from `kdp.processor` so
existing imports keep working.
`PreprocessingModel.predict` passed the caller's dict straight to Keras,
which rejects NumPy string arrays ("Invalid dtype: str32") -- so predict()
failed for any model with a categorical, text or date feature, which is
most of them. The conversion `CallableDict` already performed is now a
shared helper used by both.
`SplitLayer` carried two `register_keras_serializable` decorators after the
earlier serialization pass; the redundant one is removed, keeping the
original `kdp.processor` package so previously saved models still resolve.
Adds test/test_public_api_surface.py covering statistics from specs alone,
auto_configure's recommendations, and predict() with plain NumPy and Python
inputs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NQmAjbXCdkBSX1CPAd56H2
Running the six scripts in examples/ end to end -- they are referenced from the docs and none had ever been exercised in CI -- turned up four library defects and four example defects. Library: - Custom preprocessing pipelines were broken for the spelling the docs lead with. `_add_custom_steps` read `preprocessor_step.__name__`, which only a class has, so `preprocessors=["Lambda", "Dense"]` -- the form every code block in docs/advanced/custom-preprocessing.md uses -- raised AttributeError. Names, classes and already-built layer instances are all handled now, and step names carry their position so the same layer type twice in one pipeline no longer collides. - `LagFeatureLayer` drove a Python `if` off `tf.rank(inputs)`, so it raised OperatorNotAllowedInGraphError the moment it was traced into a functional model -- the same defect already fixed in its sibling layers. Its `compute_output_shape` also ignored the rows `drop_na` removes. - `InferenceFormatter._convert_to_tensors` decided a column was numeric if *any* entry was missing, because `pd.isna` sat on the wrong side of an `or`. A categorical column with one gap went to `tf.constant(..., tf.float32)` and raised "mixed types". The type is now decided from the values that are present; gaps become NaN in a numeric column and the empty string in a categorical one, which vocabulary layers already map to their OOV slot. - `_validate_time_series_inference_data` accepted only lists and ndarrays, so `predict()` rejected the tensors `TimeSeriesInferenceFormatter` returns when asked for them -- KDP refusing the output of its own formatter. Examples: `custom_preprocessing_example.py` was written against a scikit-learn-shaped API (`PreprocessingModel(features=...)`, `.fit`, `.transform`, `.save`) that KDP does not have, and built `Embedding`, `StringLookup` and `TextVectorization` layers without the vocabulary or `input_dim` each requires; `time_series_inference.py` passed a DataFrame as `path_data`, which takes a file path; `time_series_features_example.py` split X and y one row apart, so Keras rejected the pair. All six scripts now run to completion. matplotlib and scikit-learn are declared as dev dependencies -- the examples import them and a plain `poetry install` could not run them. Adds test/test_custom_pipelines_and_inference.py (11 tests). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NQmAjbXCdkBSX1CPAd56H2
`scripts/generate_docstring_docs.py` read every module out of `sys.modules` under a comment that says "Import required modules" and imports nothing, so the first lookup raised `KeyError: 'kdp.processor'`. `make generate_doc_content`, which the contributing docs tell people to run, could never complete. The module list is now imported through `importlib`. Verified that every developer script runs: the three diagram/doc generators, analyze_dataset, optimize_test_data and test_optimizer; that all three shell scripts parse; that `make test`, `clean`, `clean_tests`, `clean_built`, `identify_unused_diagrams` and `build_pkg` all resolve without rewriting the package version; and that the README's headline example works verbatim. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NQmAjbXCdkBSX1CPAd56H2
piotrlaczkowski
changed the title
|
piotrlaczkowski
changed the title
The bytes-decoding fix earlier in this branch corrected the *writer*, but a
`features_stats.json` produced by <= 1.11.2 still holds repr strings
("b'paris'") rather than the categories themselves. Such a file loads
without complaint and builds a working model in which every category misses
the vocabulary and encodes identically to an unseen value -- verified: the
encodings for paris, tokyo, lima and a completely unseen token were bit-for-bit
identical. Silent, total signal loss on the upgrade path, and `overwrite_stats`
defaults to False, so reusing the file is the default behaviour.
`_load_stats` now detects that format and recomputes from the data with a
warning naming the file, instead of trusting it. The detector matches only a
genuine byte repr, so a category that merely begins with "b" is unaffected.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NQmAjbXCdkBSX1CPAd56H2
`build_preprocessor` always ran the statistics pass, so a model built entirely from features that learn nothing from the data -- hashing categoricals with an explicit `hash_bucket_size` -- crashed with a pathlib `TypeError: expected str, bytes or os.PathLike object, not NoneType` when `path_data` was left unset. Previously this was masked by a shared `./features_stats.json` left behind by earlier tests in the same run, so it only surfaced on a fresh checkout. `_requires_dataset_statistics()` now decides whether the pass is needed, and `_get_csv_file_pattern` rejects a missing or non-path `path_data` with a message naming the parameter instead of failing deep inside `tf.data`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NQmAjbXCdkBSX1CPAd56H2
Several fixes on this branch change the values the preprocessor emits rather than raising: the time series layers computed canned constants, robust-scale and quantile were applied across the whole tensor instead of per feature, `preferred_distribution` was ignored, and `get_output_dim` combined dimensions additively. A pipeline that ran clean on 1.11.x therefore keeps running and quietly returns different numbers, so anything trained on the old output has to be re-fitted. That is worth stating somewhere a user will look before upgrading rather than leaving it implicit in a changelog of commit subjects. The guide also covers the automatic recomputation of `features_stats.json` files written by <= 1.11.2, the Python 3.10 and Keras 3 requirements, the entry points that stop failing, and the `FeatureSpaceConverter` move. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NQmAjbXCdkBSX1CPAd56H2
github-actions Bot
pushed a commit
that referenced
this pull request
Aug 27, 2026
## 1.12.0 (2026-08-27) * fix(KDP): correct distribution transforms, dynamic pipeline and distribution hints ([9f73c70](9f73c70)) * fix(KDP): make saved preprocessing models reloadable ([bf6bb16](bf6bb16)) * fix(KDP): make the docstring documentation generator run ([cd90054](cd90054)) * fix(KDP): make the standalone entry points work ([112ed4e](112ed4e)) * fix(KDP): only compute dataset statistics when a feature needs them ([10dcc5b](10dcc5b)) * fix(KDP): reject vocabulary statistics written by older releases ([7ecce75](7ecce75)) * fix(KDP): repair core defects, enable the linter and move to the Keras 3 API (#36) ([053c72d](053c72d)), closes [#36](#36) * fix(KDP): repair custom pipelines, inference conversion and the examples ([ac836dc](ac836dc)) * fix(KDP): repair four defects that only appeared when the model was called ([f0c1850](f0c1850)), closes [passthrou#excluded](https://github.com/passthrou/issues/excluded) * fix(KDP): time series layers returned canned values instead of computing ([f584849](f584849)) * docs(KDP): add an upgrade guide for the 1.11.x transition ([9daadf3](9daadf3)) * docs(KDP): add Discord community link to README for user engagement ([4fc132d](4fc132d)) * docs(KDP): document the remaining public API and reach a clean lint gate ([ddfc62e](ddfc62e)) * docs(KDP): remove outdated badge links and streamline README content ([bfc4f5b](bfc4f5b)) * docs(KDP): update Discord community link in README for improved user access ([692fb24](692fb24)) * docs(KDP): update installation instructions in README to use 'kdp' instead of 'keras-data-processor' ([4966d86](4966d86)) * ci(KDP): fix release conditions, docs build and workflow dependencies ([f6ded58](f6ded58)) * ci(KDP): guard Keras 3 forward compatibility ([8adeeca](8adeeca)) * refactor(KDP): clear the lint backlog the disabled config had hidden ([22b601c](22b601c)) * refactor(KDP): target the Keras 3 API directly and fix what that exposed ([ed8d1e3](ed8d1e3)) * refactor(KDP): update repository links and branding to reflect UnicoLab ownership ([fcf82b9](fcf82b9)) * build(KDP): fix packaging metadata, docs links and destructive Makefile ([113498c](113498c)) * build(KDP): make the linter actually run ([6c38fb0](6c38fb0)) * chore: save last release version for recovery [skip ci] ([933d411](933d411))
5 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe your changes shortly
Maintenance pass over the whole repository. The suite was green at 378 tests, so
almost everything below sat in code the tests built but never ran, or never
touched at all. 452 tests pass now.
Correctness — silently wrong output
bytes, and_custom_serializerwrotestr(b"foo")→"b'foo'"into the stats JSON. Any run reusing cached statsbuilt a model whose vocabulary matched nothing.
matched a test fixture's shape.
DifferencingLayer(order=1)returned[2, 2, 2, 2]for any five-element series. Their tests asserted thoseconstants, which is why the suite stayed green. Removing them exposed a
double-striding bug in
RollingStatsLayerthat the canned branch was hiding.robust-scalewas not robust scaling: it centred on the global meaninstead of the per-feature median and estimated quartiles from
argsort(x) < reduce_mean(shape(x)). Observed IQR was ~0.28 instead of 1.0.quantilepooled every feature into one ranking.TimeSeriesFeature.get_output_dimcontained a literalreturn 5for onetest case and summed widths that compose multiplicatively (the four-transform
case is 54 columns, not 8).
Correctness — features that could not work at all
load_model()failed on any non-trivial model: 23 of 27 custom layerswere unregistered with Keras.
FLOAT_DISCRETIZEDbuilt fine and raised the moment it was called —and
num_binsis exactly what the docs tell users to pass.tf.squeezeon a1-row batch) and under distribution-aware encoding (degenerate histogram
range) — i.e. the ordinary serving case.
AttributeErrorforpreprocessors=["Lambda", "Dense"], the form every docs example uses.auto_configurereturned empty recommendations for every dataset;DatasetStatisticsacceptedfeatures_specsand never derived anythingfrom it.
features_stats_pathwas accepted and dropped on the floor, along withbatch_size,date_featuresandtime_series_features.DynamicPreprocessingPipelinealiased one mutable set across all layers andcollapsed outputs onto a single key;
FeaturePreprocessor(use_dynamic=True)called a method that did not exist.
predict()rejected NumPy string arrays, so it failed for most models.preferred_distributionwas read from the wrong place under a misspelling.Behaviour change worth reviewing
build_layersis now row-preserving: an explicitdrop_na=Trueisoverridden with a warning, because a preprocessing model that returns fewer
rows than it was given cannot be concatenated with sibling features or aligned
with labels.
build_layers(row_preserving=False)keeps the old behaviour fordriving the layers directly. One test asserted the old contract — 10 rows in,
5 out, with no way to tell which 5 — and now asserts row preservation.
Keras 3
Moved off the
tf.kerascompatibility shim ontoimport kerasthroughout.That surfaced
keras.backend.int_shape()— removed in Keras 3, still answeredonly by the shim — on the Feature-MoE path. Verified against Keras 3.15.1,
five minor versions ahead of what TF 2.19 pins, with a CI job to keep it there.
Tooling that was not running
.ruff.tomlwas an empty 0-byte file, which takes precedence overpyproject.toml— the linter had been a no-op, hiding 761 findings. Now zeroacross
kdp/,test/,scripts/andexamples/.make help(anymaketarget) silently rewrote the package version inpyproject.toml, becausePOETRY_VERSION := $(shell poetry version ...)expands while make parses the file.
python = ">=3.9"was wrong: 58 PEP 604 unions meanimport kdpfailson 3.9. Dropped the unused
tf-kerasdep and movedparameterizedto dev.RELEASE.yml's recovery path could never run (failure()takes noarguments; skipped
needsneed!cancelled()).mkdocs build --strictfailed; thedocstring generator read
sys.modulesfor modules nothing imported.Checklist
My code follows all rules defined in:
mkdocs build --strictis cleanAlso verified: save/load round-trips byte-identical across 9 preprocessing and
5 time series configurations; all 6 example scripts run end to end; every
developer script and
maketarget works.