diff --git a/.github/workflows/UTESTS.yml b/.github/workflows/UTESTS.yml index 3547bb9..d640517 100644 --- a/.github/workflows/UTESTS.yml +++ b/.github/workflows/UTESTS.yml @@ -58,6 +58,23 @@ jobs: python -m pip install --upgrade pip pip install poetry poetry install --no-interaction + - name: Every test belongs to a group the matrix runs + run: | + # The matrix selects by marker. A test carrying none of these markers + # is collected by nobody and never runs in CI, which is how the whole + # save/load round-trip suite went unexecuted. + orphans=$(poetry run pytest test/ --collect-only -q \ + -m "not unit and not integration and not layers and not processor and not time_series" \ + 2>/dev/null | grep -c "::" || true) + if [ "$orphans" -ne 0 ]; then + echo "::error::$orphans test(s) match no CI marker group:" + poetry run pytest test/ --collect-only -q \ + -m "not unit and not integration and not layers and not processor and not time_series" + exit 1 + fi + echo "All collected tests belong to a marker group." + timeout-minutes: 3 + - name: Run smoke tests run: | poetry run pytest -m "micro" --maxfail=1 --tb=no -q --timeout=30 @@ -160,6 +177,32 @@ jobs: run: poetry run pytest -c pytest-fast.ini --timeout=180 timeout-minutes: 15 + # The docs promise an API. This job runs every self-contained python example + # in docs/ against data synthesised from the block's own feature spec, which + # catches the defects a symbol check cannot: a wrong shape, an option + # combination Keras rejects, or a method whose return value the prose + # describes incorrectly. + doc-examples: + runs-on: ubuntu-latest + needs: [smoke-test] + if: always() && (needs.smoke-test.result == 'success' || github.event_name != 'pull_request') + steps: + - uses: actions/checkout@v4 + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: "3.11" + + - name: Install dependencies + run: | + python -m pip install --upgrade pip + pip install poetry + poetry install --no-interaction + + - name: Run every documented example + run: poetry run python scripts/verify_doc_examples.py + timeout-minutes: 25 + # Coverage job (only on main Python version) coverage: runs-on: ubuntu-latest diff --git a/.gitignore b/.gitignore index 6248155..51a6cac 100644 --- a/.gitignore +++ b/.gitignore @@ -167,3 +167,6 @@ my_tests/* # derivative files *.csv + +# Images written by the example scripts +examples/outputs/ diff --git a/Makefile b/Makefile index c38b7e3..4e888fc 100644 --- a/Makefile +++ b/Makefile @@ -276,3 +276,8 @@ help: printf "\n"; \ }' \ | more $(shell test $(shell uname) = Darwin && echo '--no-init --raw-control-chars') + +.PHONY: verify_doc_examples +## Run every self-contained python example in docs/ against synthesised data +verify_doc_examples: + poetry run python scripts/verify_doc_examples.py diff --git a/README.md b/README.md index 61b90f9..a1bf9ea 100644 --- a/README.md +++ b/README.md @@ -24,11 +24,16 @@ KDP provides a state-of-the-art preprocessing system built on TensorFlow Keras. ## ✨ Key Features - 🚀 **Efficient Single-Pass Processing**: Process all features in one go, dramatically faster than alternatives -- 🧠 **Distribution-Aware Encoding**: Automatically detects and optimally handles different data distributions -- 👁️ **Tabular Attention**: Captures complex feature interactions for better model performance -- 🔍 **Feature Selection**: Automatically identifies and focuses on the most important features -- 🔄 **Feature-wise Mixture of Experts**: Specialized processing for different feature types -- 📦 **Production-Ready**: Deploy your preprocessing along with your model as a single unit +- 🧠 **[Distribution-Aware Encoding](docs/advanced/distribution-aware-encoding.md)**: Detects the shape of each numeric column and transforms it accordingly +- 👁️ **[Tabular Attention](docs/advanced/tabular-attention.md)**: Inter-feature and inter-sample attention, with a multi-resolution mode that treats numeric and categorical columns separately +- 🧮 **[Advanced Numerical Embeddings](docs/advanced/numerical-embeddings.md)**: Learned per-feature embeddings, and a global embedding across all numeric columns +- 🔀 **[Transformer Blocks](docs/advanced/transformer-blocks.md)**: Stackable blocks over the categorical features or over everything +- 🔄 **[Feature-wise Mixture of Experts](docs/advanced/feature-moe.md)**: Specialised expert networks per feature, routed automatically or by hand +- ⏱️ **[Time Series Features](docs/features/time_series_features.md)**: Lags, rolling statistics, differencing, moving averages, wavelets, FFT, tsfresh statistics and calendar features +- 📝 **[Text](docs/features/text-features.md), [Date](docs/features/date-features.md), [Categorical](docs/features/categorical-features.md), [Cross](docs/features/cross-features.md) and [Passthrough](docs/features/passthrough-features.md) features**: Every column type in one model +- 🎛️ **[Auto-Configuration](docs/optimization/auto-configuration.md)**: Inspects your data and recommends the preprocessing to use +- 🔍 **[Feature Selection](docs/optimization/feature-selection.md)**: Gated variable-selection layers on any subset of your features +- 📦 **Production-Ready**: Save and reload the preprocessor with your model as a single Keras artefact ## 🚀 Quick Installation diff --git a/docs/advanced/custom-preprocessing.md b/docs/advanced/custom-preprocessing.md index c3f7b66..7ed6979 100644 --- a/docs/advanced/custom-preprocessing.md +++ b/docs/advanced/custom-preprocessing.md @@ -565,7 +565,6 @@ feature = NumericalFeature( CustomScalingLayer, "Dense" ], - scaling_factor=5.0, # For CustomScalingLayer units=16 # For Dense ) ``` @@ -798,9 +797,12 @@ feature = NumericalFeature( ] ) -# The normalization layer needs to be adapted to the data -model = PreprocessingModel(features={"height": feature}) -model.fit(data) # This initializes the normalization statistics +# The normalization layer is adapted while the preprocessor is built +model = PreprocessingModel( + path_data="data.csv", + features_specs={"height": feature}, +) +model.build_preprocessor() ``` ### 🚀 GPU-Accelerated Custom Preprocessing @@ -825,14 +827,20 @@ import logging logging.basicConfig(level=logging.DEBUG) # Create a model with your custom preprocessing -model = PreprocessingModel(features=features) +model = PreprocessingModel(path_data="data.csv", features_specs=features) # Inspect the model layers -model.build_model() +model.build_preprocessor() model.model.summary() -# Test with small batch -small_batch = data.head(5) -result = model.transform(small_batch) -print(result) +# Test with a small batch. The built model is a Keras model over a dict of +# columns, so feed it one tensor per feature -- `InferenceFormatter` does that +# conversion for you when you have a DataFrame in hand. +from kdp import InferenceFormatter + +batch = InferenceFormatter(model).prepare_inference_data( + data.head(5), + to_tensors=True, +) +print(model.model(batch)) ``` diff --git a/docs/advanced/distribution-aware-encoding.md b/docs/advanced/distribution-aware-encoding.md index df56743..52907fc 100644 --- a/docs/advanced/distribution-aware-encoding.md +++ b/docs/advanced/distribution-aware-encoding.md @@ -108,6 +108,8 @@ preprocessor = PreprocessingModel(
```python +from kdp import FeatureType, PreprocessingModel + from kdp.features import NumericalFeature from kdp.layers.distribution_aware_encoder_layer import DistributionType @@ -129,8 +131,6 @@ preprocessor = PreprocessingModel( features_specs=features_specs, use_distribution_aware=True, distribution_aware_bins=1000, - detect_periodicity=True, # Enable periodic pattern detection - handle_sparsity=True # Enable sparse data handling ) ``` @@ -251,6 +251,10 @@ preprocessor = PreprocessingModel( ## ⚙️ Configuration Options +### On `PreprocessingModel` + +These two are the whole model-level surface: +
@@ -274,6 +278,88 @@ preprocessor = PreprocessingModel( + +
1000 Number of bins for distribution analysis
+
+ +Per feature, set `preferred_distribution` on a `NumericalFeature` to skip +automatic detection and force a specific distribution. + +!!! warning "Automatic detection is a heuristic, and it is not always right" + Measured over six seeded samples, detection identifies `heavy_tailed`, + `log_normal`, `discrete`, `periodic`, `sparse` and `beta` correctly and + consistently. Four shapes are consistently confused with a neighbour: + + | Actual | Detected as | + |---|---| + | `normal` | `multimodal` | + | `uniform` | `multimodal` | + | `multimodal` | `periodic` | + | `exponential` | `log_normal` | + + The encoding still works — a neighbouring distribution's transform is + usually a reasonable choice — but if a column's shape matters to you, + set `preferred_distribution` on the feature rather than relying on + detection. `test/layers/test_distribution_aware_encoder.py` pins this + behaviour, so it cannot change without being noticed. + +### Available transformations + +`transform_type` accepts these; anything else raises. `auto` is the default on +the encoder and picks from the rest using the column's own shape — +whether it is bounded, contains zeros, or contains negative values. + +
+ + + + + + + + + + + + + + + + + + + + + + +
NameRequiresUse for
auto—Let KDP choose from the column's shape.
none—Pass the values through unchanged.
logStrictly positiveLong right tails, such as income.
sqrtNon-negativeMilder right skew; tolerates zeros.
cube-rootAny signSkew in data that also goes negative.
arcsinhAny signLog-like compression that accepts zero and negatives.
box-coxStrictly positivePower transform toward normality.
yeo-johnsonAny signBox-Cox for data including zero and negatives.
logitValues inside (0, 1)Proportions and rates.
min-max—Rescale to a fixed range.
robust-scale—Median and IQR per feature; resists outliers.
quantile—Rank-based, per feature; flattens any shape.
+
+ +!!! tip "`auto` narrows the candidates to what the data allows" + A column with zeros never gets `log`, and one with negative values never + gets `box-cox`, so `auto` cannot produce infinities from an unsuitable + transform. Restrict the search yourself with + `auto_candidates=["log", "sqrt"]`. + +### On the `DistributionAwareEncoder` layer + +The processor builds this layer for you with `detect_periodicity=True`, +`handle_sparsity=True`, `adaptive_binning=True` and `mixture_components=3` +fixed. To change them, use the layer directly through a +[custom preprocessing pipeline](custom-preprocessing.md) — passing them +to `PreprocessingModel` raises `TypeError`. + +
+ + + + + + + + + + @@ -286,6 +372,12 @@ preprocessor = PreprocessingModel( + + + + + + @@ -385,10 +477,7 @@ preprocessor = PreprocessingModel( features_specs=features_specs, use_distribution_aware=True, distribution_aware_bins=1000, - detect_periodicity=True, # For daily/weekly patterns - handle_sparsity=True, # For low-volume periods embedding_dim=32, # Project to fixed dimension - add_distribution_embedding=True # Add distribution information ) ``` @@ -400,6 +489,8 @@ preprocessor = PreprocessingModel(
```python +from kdp import DistributionType, FeatureType, NumericalFeature, PreprocessingModel + features_specs = { "temperature": NumericalFeature( name="temperature", @@ -423,8 +514,6 @@ preprocessor = PreprocessingModel( features_specs=features_specs, use_distribution_aware=True, distribution_aware_bins=500, # Fewer bins for simpler distributions - detect_periodicity=True, # For daily temperature cycles - handle_sparsity=False, # No sparse data expected embedding_dim=16 # Smaller embedding for simpler patterns ) ``` @@ -457,7 +546,7 @@ preprocessor = PreprocessingModel( 👁️ Tabular Attention - + 🎯 Feature Selection diff --git a/docs/advanced/feature-moe.md b/docs/advanced/feature-moe.md index 2e9fda0..bb14e8f 100644 --- a/docs/advanced/feature-moe.md +++ b/docs/advanced/feature-moe.md @@ -25,7 +25,7 @@ features = { "age": FeatureType.FLOAT_NORMALIZED, "income": FeatureType.FLOAT_RESCALED, "occupation": FeatureType.STRING_CATEGORICAL, - "purchase_history": FeatureType.FLOAT_ARRAY, + "purchase_history": FeatureType.FLOAT_NORMALIZED, } # Create preprocessor with Feature MoE @@ -86,7 +86,7 @@ model = result["model"]
- + @@ -96,10 +96,171 @@ model = result["model"] + + + + + + + + + + + + + + + + + + + + + + + +
ParameterTypeDefaultDescription
detect_periodicity boolTrue Special handling for sparse data
adaptive_binningboolNoneLearn bin boundaries from the batch rather than using fixed ones
embedding_dim int
feature_moe_sparsityUse only top k expertsHow many experts each feature may use. Setting it to feature_moe_num_experts routes densely. 2 1-3 (lower = faster, higher = more accurate)
[64, 32] Deeper for complex relationships
feature_moe_assignmentsFeature → expert index map, required by "predefined" routingNoneGroup related features onto the same expert index
feature_moe_dropoutDropout applied inside every expert network0.10.0-0.3 (raise it when experts overfit)
feature_moe_freeze_expertsFreeze expert weights so only the router trainsFalseTrue when reusing pretrained experts
feature_moe_use_residualAdd the original feature back onto its expert outputTrueKeep True unless you want experts to fully replace the input
+## 🎛️ Steering the Experts + +The router is not the only control you have. Four extra parameters decide *how* +experts are assigned, trained and combined. + +### Hand-picked routing + +Set `feature_moe_routing="predefined"` and hand KDP a `feature_moe_assignments` +map to place each feature on a specific expert yourself. Features that share an +expert index are processed by the same specialist network. + +
+ +```python +from kdp import PreprocessingModel, FeatureType + +features = { + "age": FeatureType.FLOAT_NORMALIZED, + "income": FeatureType.FLOAT_RESCALED, + "occupation": FeatureType.STRING_CATEGORICAL, + "education": FeatureType.STRING_CATEGORICAL, +} + +preprocessor = PreprocessingModel( + path_data="data.csv", + features_specs=features, + use_feature_moe=True, + feature_moe_num_experts=2, + feature_moe_routing="predefined", + feature_moe_assignments={ + "age": 0, # expert 0 gets the demographic signals + "education": 0, + "income": 1, # expert 1 gets the financial ones + "occupation": 1, + }, +) + +result = preprocessor.build_preprocessor() +``` + +
+ +The map has to be complete. KDP raises a `ValueError` naming the gaps if a +routed feature has no expert, because the assignment matrix doubles as the +router's weights — an unassigned feature would be multiplied by zero and +vanish from the model. Expert indices are range-checked too, and an index may +be a weight map (`{0: 0.7, 1: 0.3}`) to split one feature across experts. + +In `concat` output mode only numeric and categorical features reach the +mixture, so those are the names the map may contain; the error lists the exact +set if you name something else. + +### Regularising, freezing and residuals + +
+ +```python +from kdp import PreprocessingModel, FeatureType + +features = { + "age": FeatureType.FLOAT_NORMALIZED, + "income": FeatureType.FLOAT_RESCALED, + "occupation": FeatureType.STRING_CATEGORICAL, +} + +preprocessor = PreprocessingModel( + path_data="data.csv", + features_specs=features, + use_feature_moe=True, + feature_moe_num_experts=3, + feature_moe_dropout=0.2, # dropout inside every expert network + feature_moe_freeze_experts=False, # True keeps expert weights fixed + feature_moe_use_residual=True, # add the input back onto the expert output +) + +result = preprocessor.build_preprocessor() +``` + +
+ +
+ + + + + + + + + + + + + + + + + + + + + + + + + +
ParameterWhat it changesReach for it when
feature_moe_dropoutInserts a Dropout layer after every hidden layer of every expert. 0.0 removes the layers entirely.Experts memorise the training set, or you have many experts and little data.
feature_moe_freeze_expertsMarks every expert non-trainable, so gradients only reach the router and the surrounding layers.You loaded pretrained experts, or you want the router to settle before fine-tuning.
feature_moe_use_residualAdds the untouched feature representation onto the expert output. Applied per feature, only where the two widths already match.Almost always — it keeps the original signal reachable. Turn it off when experts should fully replace their input.
+
+ +!!! note "Residuals need matching widths" + The residual is a plain `Add`, so it only fires for features whose + preprocessed width already equals `feature_moe_expert_dim`. Features of a + different width pass through the expert output alone — no error, no + silent reshape. + +### Reading the routing back + +`get_expert_assignments()` on the mixture layer reports, per feature, how much +of it each expert handles. Predefined routing answers from the map you gave it; +learned routing decides from the data, so hand it a batch. + +
+ +```python +moe = preprocessor.model.get_layer("feature_moe_concat") + +# Predefined routing: the map you supplied, normalised to weights. +print(moe.get_expert_assignments()) +# {"age": {0: 1.0}, "income": {1: 1.0}, ...} +``` + +
+ +With learned routing, pass the stacked features the layer sees. Each row keeps +only the experts with a non-zero share, so a run with `feature_moe_sparsity=2` +lists two experts per feature and their weights sum to one. + ## 💡 Pro Tips for Feature MoE
@@ -110,7 +271,7 @@ model = result["model"]

Visualize Expert Assignments

-

Examine which experts handle which features by plotting the assignments as a heatmap to understand your model's internal decisions.

+

Read the routing off the layer with get_expert_assignments() and plot it as a heatmap. See the section below for the call.

diff --git a/docs/advanced/numerical-embeddings.md b/docs/advanced/numerical-embeddings.md index 5ec9263..dc792b3 100644 --- a/docs/advanced/numerical-embeddings.md +++ b/docs/advanced/numerical-embeddings.md @@ -7,6 +7,15 @@
+!!! note "Output width follows `embedding_dim`" + With `use_advanced_numerical_embedding=True`, each numeric feature becomes + an `embedding_dim`-wide block, so the model's numeric output is + `number_of_numeric_features x embedding_dim`. Two features at + `embedding_dim=16` give 32 columns. + + A value set on an individual `NumericalFeature` wins over the model-level + one, so you can give a single column a different width. + ## 📋 Architecture Overview
@@ -92,9 +101,9 @@ features_specs = { preprocessor = PreprocessingModel( path_data="data/my_data.csv", features_specs=features_specs, - use_numerical_embedding=True, # Enable numerical embeddings - numerical_embedding_dim=8, # Size of each feature's embedding - numerical_num_bins=10 # Number of bins for discretization + use_advanced_numerical_embedding=True, # Enable numerical embeddings + embedding_dim=8, # Size of each feature's embedding + num_bins=10 # Number of bins for discretization ) ``` @@ -111,7 +120,7 @@ preprocessor = PreprocessingModel( ```python from kdp import PreprocessingModel from kdp.features import NumericalFeature -from kdp.enums import FeatureType +from kdp import FeatureType # Define numerical features with customized embeddings features_specs = { @@ -139,10 +148,10 @@ features_specs = { preprocessor = PreprocessingModel( path_data="data/my_data.csv", features_specs=features_specs, - use_numerical_embedding=True, - numerical_mlp_hidden_units=16, # Hidden layer size for continuous branch - numerical_dropout_rate=0.1, # Regularization - numerical_use_batch_norm=True # Normalize activations + use_advanced_numerical_embedding=True, + mlp_hidden_units=16, # Hidden layer size for continuous branch + dropout_rate=0.1, # Regularization + use_batch_norm=True # Normalize activations ) ``` @@ -211,14 +220,14 @@ This approach is ideal for: | Parameter | Type | Default | Description | |-----------|------|---------|-------------| -| `use_numerical_embedding` | bool | False | Enable numerical embeddings | -| `numerical_embedding_dim` | int | 8 | Size of each feature's embedding | -| `numerical_mlp_hidden_units` | int | 16 | Hidden layer size for continuous branch | -| `numerical_num_bins` | int | 10 | Number of bins for discretization | +| `use_advanced_numerical_embedding` | bool | False | Enable numerical embeddings | +| `embedding_dim` | int | 8 | Size of each feature's embedding | +| `mlp_hidden_units` | int | 16 | Hidden layer size for continuous branch | +| `num_bins` | int | 10 | Number of bins for discretization | | `numerical_init_min` | float/list | -3.0 | Initial minimum for scaling | | `numerical_init_max` | float/list | 3.0 | Initial maximum for scaling | -| `numerical_dropout_rate` | float | 0.1 | Dropout rate for regularization | -| `numerical_use_batch_norm` | bool | True | Apply batch normalization | +| `dropout_rate` | float | 0.1 | Dropout rate for regularization | +| `use_batch_norm` | bool | True | Apply batch normalization | ### Global Embeddings @@ -259,7 +268,7 @@ This approach is ideal for: ```python from kdp import PreprocessingModel from kdp.features import NumericalFeature -from kdp.enums import FeatureType +from kdp import FeatureType # Define financial features with domain knowledge features_specs = { @@ -305,10 +314,10 @@ features_specs = { preprocessor = PreprocessingModel( path_data="data/financial_data.csv", features_specs=features_specs, - use_numerical_embedding=True, - numerical_mlp_hidden_units=16, - numerical_dropout_rate=0.2, # Higher dropout for financial data - numerical_use_batch_norm=True + use_advanced_numerical_embedding=True, + mlp_hidden_units=16, + dropout_rate=0.2, # Higher dropout for financial data + use_batch_norm=True ) ``` @@ -317,7 +326,7 @@ preprocessor = PreprocessingModel( ```python from kdp import PreprocessingModel from kdp.features import NumericalFeature -from kdp.enums import FeatureType +from kdp import FeatureType # Define patient features features_specs = { diff --git a/docs/advanced/tabular-attention.md b/docs/advanced/tabular-attention.md index e71fabb..8512350 100644 --- a/docs/advanced/tabular-attention.md +++ b/docs/advanced/tabular-attention.md @@ -7,6 +7,17 @@
+!!! note "Output width" + With the default placements the model's output is + `tabular_attention_dim` columns wide — attention projects every + feature into one shared space rather than concatenating them. + `multi_resolution` runs a numeric and a categorical branch and concatenates + the two, giving `2 x tabular_attention_dim`. + + `tabular_attention_embedding_dim` sizes the categorical embedding inside + the multi-resolution branch, so it changes the parameter count but not the + output width. + ## 📋 Overview
@@ -177,7 +188,7 @@ The diagram shows how tabular attention transforms features through a multi-head ```python from kdp import PreprocessingModel, FeatureType -from kdp.enums import TabularAttentionPlacementOptions +from kdp import TabularAttentionPlacementOptions features_specs = { "customer_age": FeatureType.FLOAT_NORMALIZED, @@ -203,7 +214,7 @@ preprocessor = PreprocessingModel( ```python from kdp import PreprocessingModel, FeatureType -from kdp.enums import TabularAttentionPlacementOptions +from kdp import TabularAttentionPlacementOptions features_specs = { # Numerical features @@ -289,7 +300,7 @@ preprocessor = PreprocessingModel( ⚡ Transformer Blocks - + 🎯 Feature Selection diff --git a/docs/advanced/transformer-blocks.md b/docs/advanced/transformer-blocks.md index 1113d21..694bf4d 100644 --- a/docs/advanced/transformer-blocks.md +++ b/docs/advanced/transformer-blocks.md @@ -55,10 +55,9 @@ features_specs = { preprocessor = PreprocessingModel( path_data="data/my_data.csv", features_specs=features_specs, - use_transformer_blocks=True, # Enable transformer blocks - transformer_num_blocks=3, # Number of transformer blocks - transformer_num_heads=4, # Number of attention heads - transformer_dim=64 # Hidden dimension + transfo_nr_blocks=3, # Number of transformer blocks + transfo_nr_heads=4, # Number of attention heads + transfo_ff_units=64, # Feed-forward hidden dimension ) ``` @@ -87,31 +86,25 @@ preprocessor = PreprocessingModel( - use_transformer_blocks - bool - False - Enable transformer blocks + transfo_nr_blocks + int | None + None + Number of transformer blocks. Setting this is what enables them — leave it None and no blocks are added. - transformer_num_blocks - int - 3 - Number of transformer blocks - - - transformer_num_heads + transfo_nr_heads int 4 Number of attention heads - transformer_dim + transfo_ff_units int 64 Hidden dimension - transformer_dropout + transfo_dropout_rate float 0.1 Dropout rate @@ -147,6 +140,8 @@ preprocessor = PreprocessingModel(
```python +from kdp import FeatureType, PreprocessingModel + features_specs = { "age": FeatureType.FLOAT_NORMALIZED, "income": FeatureType.FLOAT_RESCALED, @@ -159,11 +154,10 @@ features_specs = { preprocessor = PreprocessingModel( path_data="data/customer_data.csv", features_specs=features_specs, - use_transformer_blocks=True, - transformer_num_blocks=4, # More blocks for complex customer patterns - transformer_num_heads=8, # More heads for diverse relationships - transformer_dim=128, # Larger dimension for rich representations - transformer_dropout=0.2 # Higher dropout for regularization + transfo_nr_blocks=4, # More blocks for complex customer patterns + transfo_nr_heads=8, # More heads for diverse relationships + transfo_ff_units=128, # Larger dimension for rich representations + transfo_dropout_rate=0.2 # Higher dropout for regularization ) ``` @@ -175,6 +169,8 @@ preprocessor = PreprocessingModel(
```python +from kdp import FeatureType, PreprocessingModel + features_specs = { "user_id": FeatureType.INTEGER_CATEGORICAL, "item_id": FeatureType.INTEGER_CATEGORICAL, @@ -187,11 +183,10 @@ features_specs = { preprocessor = PreprocessingModel( path_data="data/recommendation_data.csv", features_specs=features_specs, - use_transformer_blocks=True, - transformer_num_blocks=3, # Standard configuration - transformer_num_heads=4, # Balanced number of heads - transformer_dim=64, # Moderate dimension - transformer_dropout=0.1 # Standard dropout + transfo_nr_blocks=3, # Standard configuration + transfo_nr_heads=4, # Balanced number of heads + transfo_ff_units=64, # Moderate dimension + transfo_dropout_rate=0.1 # Standard dropout ) ``` diff --git a/docs/contributing/development/auto-documentation.md b/docs/contributing/development/auto-documentation.md index 357761a..877207b 100644 --- a/docs/contributing/development/auto-documentation.md +++ b/docs/contributing/development/auto-documentation.md @@ -65,7 +65,7 @@ To ensure your code is properly included in the automatic documentation: 1. **Use Google-style docstrings** for all classes and methods: -```python +~~~python def my_function(param1, param2): """ One-line description of function. @@ -85,7 +85,7 @@ def my_function(param1, param2): ``` """ # Function implementation -``` +~~~ 2. **Document parameters and return values** to provide clear usage instructions. diff --git a/docs/examples/basic-examples.md b/docs/examples/basic-examples.md index 7212898..c4726c7 100644 --- a/docs/examples/basic-examples.md +++ b/docs/examples/basic-examples.md @@ -5,6 +5,8 @@ Let us go trough some different examples of case scenarios. To get a feeling how ## Example 1: Numerical features ```python +import tensorflow as tf + from kdp.features import NumericalFeature, FeatureType from kdp.processor import PreprocessingModel, OutputModeOptions @@ -70,7 +72,7 @@ ppr = PreprocessingModel( overwrite_stats=True, output_mode=OutputModeOptions.CONCAT, - # Add feature selection to get the most important features + # Gate every feature through a variable-selection layer feature_selection_placement="numeric", # Choose between (all_features|numeric|categorical) feature_selection_units=32, feature_selection_dropout=0.10, @@ -88,6 +90,22 @@ ppr = PreprocessingModel( result = ppr.build_preprocessor() # Transform data using direct model prediction +# A batch is one entry per feature. String features need string values, so +# build each column from the feature's own type. +def _sample(feature): + ft = getattr(feature, "feature_type", feature) + name = ft.name if hasattr(ft, "name") else str(ft) + if name == "STRING_CATEGORICAL": + return tf.constant([["a"]]) + if name == "INTEGER_CATEGORICAL": + return tf.constant([[0]], dtype=tf.int32) + if name == "TEXT": + return tf.constant([["some text"]]) + if name == "DATE": + return tf.constant([["2021-06-15"]]) + return tf.constant([[0.0]]) + +test_batch = {name: _sample(feature) for name, feature in features.items()} transformed_data = ppr.model.predict(test_batch) # Get feature importances @@ -181,6 +199,22 @@ ppr = PreprocessingModel( result = ppr.build_preprocessor() # Transform data using direct model prediction +# A batch is one entry per feature. String features need string values, so +# build each column from the feature's own type. +def _sample(feature): + ft = getattr(feature, "feature_type", feature) + name = ft.name if hasattr(ft, "name") else str(ft) + if name == "STRING_CATEGORICAL": + return tf.constant([["a"]]) + if name == "INTEGER_CATEGORICAL": + return tf.constant([[0]], dtype=tf.int32) + if name == "TEXT": + return tf.constant([["some text"]]) + if name == "DATE": + return tf.constant([["2021-06-15"]]) + return tf.constant([[0.0]]) + +test_batch = {name: _sample(feature) for name, feature in features.items()} transformed_data = ppr.model.predict(test_batch) # Get feature importances @@ -296,6 +330,8 @@ Normally the distribution aware encoder works well in automatic mode, once use_d However we can also manually set the prefered distribution for each numerical feature if we would like to. ```python +import tensorflow as tf + from kdp.features import NumericalFeature, FeatureType from kdp.processor import PreprocessingModel, OutputModeOptions from kdp.layers.distribution_aware_encoder_layer import DistributionAwareEncoder @@ -314,7 +350,7 @@ features = { name="rescaled_float", feature_type=FeatureType.FLOAT_RESCALED, scale=2.0, - prefered_distribution="gamma" + preferred_distribution="gamma" ), # 3. Custom preprocessing pipeline with a custom set normal distribution "custom_float": NumericalFeature( @@ -323,13 +359,13 @@ features = { preprocessors=[ tf.keras.layers.Rescaling, tf.keras.layers.Normalization, - DistributionAwareEncoder, + DistributionAwareEncoder, # from kdp.layers ], bin_boundaries=[0.0, 1.0, 2.0], mean=0.0, variance=1.0, scale=4.0, - prefered_distribution="normal" + preferred_distribution="normal" ), } @@ -341,7 +377,7 @@ ppr = PreprocessingModel( overwrite_stats=True, output_mode=OutputModeOptions.CONCAT, - # Add feature selection to get the most important features + # Gate every feature through a variable-selection layer feature_selection_placement="numeric", # Choose between (all_features|numeric|categorical) # Add tabular attention to check for feature interactions @@ -355,6 +391,22 @@ ppr = PreprocessingModel( result = ppr.build_preprocessor() # Transform data using direct model prediction +# A batch is one entry per feature. String features need string values, so +# build each column from the feature's own type. +def _sample(feature): + ft = getattr(feature, "feature_type", feature) + name = ft.name if hasattr(ft, "name") else str(ft) + if name == "STRING_CATEGORICAL": + return tf.constant([["a"]]) + if name == "INTEGER_CATEGORICAL": + return tf.constant([[0]], dtype=tf.int32) + if name == "TEXT": + return tf.constant([["some text"]]) + if name == "DATE": + return tf.constant([["2021-06-15"]]) + return tf.constant([[0.0]]) + +test_batch = {name: _sample(feature) for name, feature in features.items()} transformed_data = ppr.model.predict(test_batch) # Get feature importances @@ -374,6 +426,9 @@ Numerical embedding is a technique that allows us to embed numerical features in This can be useful for capturing non-linear relationships within/between numerical feature/s. ```python +from kdp.layers import DistributionAwareEncoder +import tensorflow as tf + from kdp.features import NumericalFeature, FeatureType from kdp.processor import PreprocessingModel, OutputModeOptions @@ -397,8 +452,11 @@ features = { preprocessors=[ tf.keras.layers.Rescaling, tf.keras.layers.Normalization, - DistributionAwareEncoder, + DistributionAwareEncoder, # from kdp.layers ], + # Extra keyword arguments are forwarded to the layer that accepts them. + # Rescaling has no default scale, so it has to be supplied. + scale=1.0, ), } @@ -422,6 +480,22 @@ ppr = PreprocessingModel( result = ppr.build_preprocessor() # Transform data using direct model prediction +# A batch is one entry per feature. String features need string values, so +# build each column from the feature's own type. +def _sample(feature): + ft = getattr(feature, "feature_type", feature) + name = ft.name if hasattr(ft, "name") else str(ft) + if name == "STRING_CATEGORICAL": + return tf.constant([["a"]]) + if name == "INTEGER_CATEGORICAL": + return tf.constant([[0]], dtype=tf.int32) + if name == "TEXT": + return tf.constant([["some text"]]) + if name == "DATE": + return tf.constant([["2021-06-15"]]) + return tf.constant([[0.0]]) + +test_batch = {name: _sample(feature) for name, feature in features.items()} transformed_data = ppr.model.predict(test_batch) # Get feature importances diff --git a/docs/examples/categorical-hashing-example.md b/docs/examples/categorical-hashing-example.md index fe3d393..8a3e899 100644 --- a/docs/examples/categorical-hashing-example.md +++ b/docs/examples/categorical-hashing-example.md @@ -179,9 +179,7 @@ features = { feature_type=FeatureType.STRING_CATEGORICAL, category_encoding=CategoryEncodingOptions.HASHING, hash_bucket_size=1024, - hash_with_embedding=True, - multi_hash=True, # Enable multiple hash functions - num_hash_functions=3 # Number of hash functions to use + hash_with_embedding=True ) } ``` diff --git a/docs/examples/complex-examples.md b/docs/examples/complex-examples.md index 8b8c589..f33bf43 100644 --- a/docs/examples/complex-examples.md +++ b/docs/examples/complex-examples.md @@ -24,8 +24,7 @@ features = { "price": NumericalFeature( name="price", feature_type=FeatureType.FLOAT_NORMALIZED, - use_embedding=True, # Create richer representation - embedding_dim=8 # Size of embedding + embedding_size=8 # Size of embedding ), "quantity": NumericalFeature( name="quantity", @@ -37,13 +36,12 @@ features = { "category": CategoricalFeature( name="category", feature_type=FeatureType.STRING_CATEGORICAL, - embedding_dim=32, # Larger embeddings for complex categories - max_vocabulary_size=1000 # Limit vocabulary size + embedding_size=32, # Larger embeddings for complex categories ), "brand": CategoricalFeature( name="brand", feature_type=FeatureType.STRING_CATEGORICAL, - embedding_dim=16 + embedding_size=16 ), # Text features with different token limits @@ -51,7 +49,7 @@ features = { name="description", feature_type=FeatureType.TEXT, max_tokens=100, # Longer limit for descriptions - output_mode="embedding" # Use text embeddings + output_mode="multi_hot" # Bag-of-words over the vocabulary ), "title": TextFeature( name="title", @@ -64,7 +62,6 @@ features = { name="sale_date", feature_type=FeatureType.DATE, add_season=True, # Add seasonal indicators - add_day_of_week=True # Add day of week information ) } ``` @@ -100,7 +97,7 @@ preprocessor = PreprocessingModel( feature_selection_units=32, # Enable caching for better performance - enable_caching=True + use_caching=True ) # Build the preprocessor @@ -136,14 +133,15 @@ This example demonstrates using KDP to preprocess financial time series data for ### Setting Up Features ```python +from kdp import CategoricalFeature, DateFeature, FeatureType, NumericalFeature, PreprocessingModel + # Define financial features features = { # Price data with custom distribution handling "close_price": NumericalFeature( name="close_price", feature_type=FeatureType.FLOAT_RESCALED, - use_embedding=True, - embedding_dim=16, + embedding_size=16, preferred_distribution="heavy_tailed" # Handle market data distributions ), "volume": NumericalFeature( @@ -168,15 +166,13 @@ features = { "market_regime": CategoricalFeature( name="market_regime", feature_type=FeatureType.STRING_CATEGORICAL, - embedding_dim=8 + embedding_size=8 ), # Date information with market-specific features "date": DateFeature( name="date", - feature_type=FeatureType.DATE, - add_day_of_week=True, # Markets behave differently on different days - add_month=True # Capture seasonal effects + feature_type=FeatureType.DATE ) } @@ -186,8 +182,8 @@ financial_preprocessor = PreprocessingModel( features_specs=features, # Enable advanced numerical embeddings for better pattern detection - use_numerical_embedding=True, - numerical_embedding_dim=32, + use_advanced_numerical_embedding=True, + embedding_dim=32, # Enable distribution-aware encoding for market data use_distribution_aware=True, @@ -211,6 +207,8 @@ This example shows how to preprocess user behavior data for churn prediction or ### Setting Up Features ```python +from kdp import CategoricalFeature, DateFeature, FeatureType, NumericalFeature, PreprocessingModel, TextFeature + # Define user behavior features features = { # User demographics @@ -221,13 +219,12 @@ features = { "gender": CategoricalFeature( name="gender", feature_type=FeatureType.STRING_CATEGORICAL, - embedding_dim=4 + embedding_size=4 ), "location": CategoricalFeature( name="location", feature_type=FeatureType.STRING_CATEGORICAL, - embedding_dim=16, - max_vocabulary_size=500 # Limit to top locations + embedding_size=16 ), # Behavioral metrics @@ -250,7 +247,7 @@ features = { "subscription_tier": CategoricalFeature( name="subscription_tier", feature_type=FeatureType.STRING_CATEGORICAL, - embedding_dim=8 + embedding_size=8 ), # Textual data @@ -311,12 +308,13 @@ user_result = user_preprocessor.build_preprocessor() 4. **Save Your Preprocessing Pipeline** ```python - # Save the complete pipeline for reuse - preprocessor.save_model("my_preprocessor.keras") + # Save the complete pipeline for reuse. The argument is a directory: the + # Keras model and the metadata needed to rebuild the pipeline go inside it. + preprocessor.save_model("my_preprocessor") - # Load it when needed + # Load it when needed. load_model returns the model and that metadata. from kdp import PreprocessingModel - loaded_preprocessor = PreprocessingModel.load_model("my_preprocessor.keras") + loaded_model, metadata = PreprocessingModel.load_model("my_preprocessor") ``` ## 🔗 Related Topics diff --git a/docs/features/categorical-features.md b/docs/features/categorical-features.md index 449edf8..ea02a56 100644 --- a/docs/features/categorical-features.md +++ b/docs/features/categorical-features.md @@ -3,57 +3,51 @@

Categorical Features in KDP

-

Learn how to effectively represent categories, leverage embeddings, and handle high-cardinality data.

+

Turn labels and IDs into dense vectors, one-hot columns, or hashed buckets.

## 📋 Overview
-

Categorical features represent data that belongs to a finite set of possible values or categories. KDP provides advanced techniques for handling categorical data, from simple encoding to neural embeddings that capture semantic relationships between categories.

+

KDP learns the vocabulary of a categorical column during the statistics pass, then encodes it one of three ways: a learned embedding, a one-hot vector, or a hash into a fixed number of buckets. Hashing is the only one that needs no vocabulary, so it is also the only one that works without a data pass.

-## 🚀 Types of Categorical Features +## 🚀 The Two Categorical Feature Types
- +
- - - - + + + - - - - + + + - - - - - - - - - - - - - - - - + + +
Feature TypeBest ForExampleWhen to UseFeature typeColumn dtypeUse for
STRING_CATEGORICALText categoriesproduct_type: "shirt", "pants", "shoes"When categories are text stringsFeatureType.STRING_CATEGORICALstringCity names, product codes, channels — anything written as text.
INTEGER_CATEGORICALNumeric categorieseducation_level: 1, 2, 3, 4When categories are already represented as integers
STRING_HASHEDHigh-cardinality setsuser_id: "user_12345", "user_67890"When there are too many unique categories (>10K)
MULTI_CATEGORICALMultiple categories per sampleinterests: ["sports", "music", "travel"]When each sample can belong to multiple categoriesFeatureType.INTEGER_CATEGORICALintegerIDs and codes that are numbers but have no ordering.
+!!! warning "`MULTI_CATEGORICAL` and `STRING_HASHED` do not exist" + Earlier documentation listed these as feature types, along with a + multi-value workflow using `separator` and `multi_hot`. `FeatureType` has + exactly eleven members and neither of these is among them, so + `FeatureType.MULTI_CATEGORICAL` raises `AttributeError`. Hashing is not a + separate feature type either — it is the `category_encoding` option + below. To encode a multi-value column, split it into columns yourself or + use a [custom pipeline](../advanced/custom-preprocessing.md). + ## 📝 Basic Usage
@@ -61,587 +55,261 @@ ```python from kdp import PreprocessingModel, FeatureType -# Simple categorical features -features = { - "product_category": FeatureType.STRING_CATEGORICAL, - "store_id": FeatureType.INTEGER_CATEGORICAL, - "tags": FeatureType.MULTI_CATEGORICAL -} - -preprocessor = PreprocessingModel( - path_data="product_data.csv", - features_specs=features -) -``` - -
- -## 🧠 Advanced Configuration - -
-

For more control over categorical processing, use the detailed configuration:

- -
- -```python -from kdp import PreprocessingModel, FeatureType, CategoricalFeature - -# Detailed configuration -features = { - # Basic configuration - "product_type": FeatureType.STRING_CATEGORICAL, - - # Full configuration with explicit CategoricalFeature - "store_location": CategoricalFeature( - name="store_location", - feature_type=FeatureType.STRING_CATEGORICAL, - embedding_dim=16, # Size of embedding vector - hash_bucket_size=1000, # For hashed features - vocabulary_size=250, # Limit vocabulary size - use_embedding=True, # Use neural embeddings - unknown_token="", # Token for out-of-vocabulary values - oov_buckets=10, # Out-of-vocabulary buckets - multi_hot=False # For single category per sample - ), - - # High-cardinality feature using hashing - "product_id": CategoricalFeature( - name="product_id", - feature_type=FeatureType.STRING_HASHED, - hash_bucket_size=5000 - ), - - # Multi-categorical feature with separator - "product_tags": CategoricalFeature( - name="product_tags", - feature_type=FeatureType.MULTI_CATEGORICAL, - separator=",", # How values are separated in data - multi_hot=True # Enable multi-hot encoding - ) -} - preprocessor = PreprocessingModel( - path_data="product_data.csv", - features_specs=features + path_data="data.csv", + features_specs={ + "city": FeatureType.STRING_CATEGORICAL, + "store_id": FeatureType.INTEGER_CATEGORICAL, + }, ) +preprocessor.build_preprocessor() ``` -
-## ⚙️ Key Configuration Parameters +## ⚙️ Configuration Parameters
- +
- + - + - - - - + + + + + + + + + + - - - + + + - - - + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - + + +
ParameterDescriptionType DefaultNotesDescription
embedding_dimSize of embedding vectors8Higher values capture more complex relationships (8-128)category_encodingstr"EMBEDDING""EMBEDDING", "ONE_HOT_ENCODING" or "HASHING".
embedding_sizeintderived from vocabulary sizeWidth of the learned embedding. Also used when hashing with an embedding on top (default 8 there).
hash_bucket_sizeNumber of hash buckets for hashed features1000Larger values reduce collisions but increase dimensionalityintderived from vocabulary sizeNumber of hash buckets. Hashing only. Setting it explicitly removes the need for a statistics pass.
saltSalt value for hash functionNoneCustom salt to make hash values unique across featuresintNoneSeed for the hash, so two columns hash differently. Hashing only.
hash_with_embeddingApply embedding after hashingFalseCombines hashing with embeddings for large vocabularies
vocabulary_sizeMaximum number of categories to keepNoneNone uses all categories, otherwise keeps top N by frequency
use_embeddingEnable neural embeddings vs. one-hot encodingTrueNeural embeddings improve performance for most models
separatorCharacter that separates values in multi-categorical features","Only used for MULTI_CATEGORICAL features
oov_bucketsNumber of buckets for out-of-vocabulary values1Higher values help handle new categories in productionboolFalsePut a learned embedding on top of the hash instead of a multi-hot vector. Hashing only.
-## 💡 Powerful Features - -
-
-

🧿 Embedding Visualizations

-

KDP's categorical embeddings can be visualized to see relationships between categories:

-
+!!! warning "Options that are silently ignored" + `embedding_dim` (the real name is `embedding_size`), `vocabulary_size`, + `max_vocabulary_size`, `use_embedding`, `unknown_token`, `oov_buckets`, + `multi_hot`, `separator`, `pretrained_embeddings`, `multi_hash` and + `num_hash_functions` are **not read by KDP**. `CategoricalFeature` accepts + any keyword argument, so passing them looks like it works and changes + nothing. Out-of-vocabulary values are always routed to a single reserved + slot; that is not configurable. -```python -# Train the preprocessor -preprocessor.fit() -result = preprocessor.build_preprocessor() - -# Extract embeddings for visualization -embeddings = preprocessor.get_feature_embeddings("product_category") - -# Visualize with t-SNE or UMAP -from sklearn.manifold import TSNE -import matplotlib.pyplot as plt +## 🎛️ The Three Encodings -tsne = TSNE(n_components=2) -embeddings_2d = tsne.fit_transform(embeddings) - -plt.figure(figsize=(10, 8)) -plt.scatter(embeddings_2d[:, 0], embeddings_2d[:, 1]) -plt.title("Category Embedding Visualization") -plt.show() -``` +### Embedding (default) -
-
- -
-

🌍 Handling High-Cardinality

-

KDP provides multiple strategies for dealing with features that have many unique values:

-
+
```python -# Method 1: Limit vocabulary size (keeps most frequent) -user_id_limited = CategoricalFeature( - name="user_id", - feature_type=FeatureType.STRING_CATEGORICAL, - vocabulary_size=10000 # Keep top 10K users -) - -# Method 2: Hash features to buckets (fast, fixed memory) -user_id_hashed = CategoricalFeature( - name="user_id", - feature_type=FeatureType.STRING_CATEGORICAL, - category_encoding=CategoryEncodingOptions.HASHING, - hash_bucket_size=5000 # Hash into 5K buckets -) +from kdp.features import CategoricalFeature, CategoryEncodingOptions, FeatureType -# Method 3: Hash with embeddings (best balance) -user_id_hash_embed = CategoricalFeature( - name="user_id", +CategoricalFeature( + name="city", feature_type=FeatureType.STRING_CATEGORICAL, - category_encoding=CategoryEncodingOptions.HASHING, - hash_bucket_size=2048, - hash_with_embedding=True, - embedding_size=16 + category_encoding=CategoryEncodingOptions.EMBEDDING, + embedding_size=16, ) ``` -
-
+
-
-

🧮 Feature Hashing

-

Feature hashing transforms categorical values into a fixed-size vector representation, ideal for very high-cardinality features. It's now fully integrated with the ModelAdvisor for automatic configuration:

-
+Leave `embedding_size` unset and KDP derives it from the vocabulary size. -```python -# Basic feature hashing -product_id = CategoricalFeature( - name="product_id", - feature_type=FeatureType.STRING_CATEGORICAL, - category_encoding=CategoryEncodingOptions.HASHING, - hash_bucket_size=1024 # Number of hash buckets -) +### One-hot -# Advanced feature hashing with custom salt -# The salt ensures different features use different hash spaces -session_id = CategoricalFeature( - name="session_id", - feature_type=FeatureType.STRING_CATEGORICAL, - category_encoding=CategoryEncodingOptions.HASHING, - hash_bucket_size=2048, - salt=42 # Custom salt value -) +
-# Feature hashing followed by embedding -user_id = CategoricalFeature( - name="user_id", +```python +CategoricalFeature( + name="status", feature_type=FeatureType.STRING_CATEGORICAL, - category_encoding=CategoryEncodingOptions.HASHING, - hash_bucket_size=2048, - hash_with_embedding=True, - embedding_size=16 # Embedding dimension after hashing + category_encoding=CategoryEncodingOptions.ONE_HOT_ENCODING, ) ``` -
-
- -
-

🤖 Auto-configuration with ModelAdvisor

-

KDP's ModelAdvisor now intelligently recommends hashing for high-cardinality features:

-
- -```python -from kdp.model_advisor import recommend_model_configuration -from kdp.stats import DatasetStatistics - -# Analyze dataset statistics -stats_calculator = DatasetStatistics("high_cardinality_data.csv") -stats_calculator.compute_statistics() - -# Get recommendations from ModelAdvisor -recommendations = recommend_model_configuration(stats_calculator.features_stats) - -# The recommendations will include HASHING for high-cardinality features -# Example output: -''' -{ - "features": { - "user_id": { - "feature_type": "CategoricalFeature", - "preprocessing": ["HASHING"], - "config": { - "category_encoding": "HASHING", - "hash_bucket_size": 2048, - "hash_with_embedding": true, - "embedding_size": 16 - }, - "notes": ["High cardinality feature (10K+ values)", "Using hashing for efficiency"] - }, - ... - } -} -''' +
-# Generate code from recommendations -code_snippet = recommendations["code_snippet"] -print(code_snippet) -``` +Output width is the vocabulary size. Good for a handful of categories, wasteful +beyond a few dozen. -
-
+### Hashing -
-

🔍 Choosing Between Encoding Options

-

KDP offers multiple encoding options for categorical features. Here's how to choose:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
EncodingVocabulary SizeMemory UsageNew CategoriesSemantic Information
One-Hot EncodingSmall (< 50)High❌ Requires retraining❌ No relationship capture
EmbeddingsMedium (50-10K)Medium⚠️ Limited by OOV handling✅ Captures relationships
HashingVery Large (10K+)Low (fixed)✅ Handles new values❌ No relationship capture
Hashing with EmbeddingsVery Large (10K+)Low-Medium✅ Handles new values✅ Some relationship capture
- -

The ModelAdvisor analyzes your data and automatically recommends the optimal encoding based on these criteria.

-
+
```python -# End-to-end example with automatic encoding selection -from kdp.features import CategoricalFeature, FeatureType, CategoryEncodingOptions -from kdp.processor import PreprocessingModel -from kdp.stats import DatasetStatistics -from kdp.model_advisor import recommend_model_configuration - -# 1. Analyze dataset -stats = DatasetStatistics("product_dataset.csv") -stats.compute_statistics() - -# 2. Get recommendations -recommendations = recommend_model_configuration(stats.features_stats) - -# 3. Create preprocessing model using recommended config -features = {} -for name, feature_rec in recommendations["features"].items(): - if feature_rec["feature_type"] == "CategoricalFeature": - # Extract configuration for this categorical feature - config = feature_rec["config"] - features[name] = CategoricalFeature( - name=name, - feature_type=getattr(FeatureType, config.get("feature_type", "STRING_CATEGORICAL")), - category_encoding=getattr(CategoryEncodingOptions, config.get("category_encoding", "EMBEDDING")), - hash_bucket_size=config.get("hash_bucket_size"), - hash_with_embedding=config.get("hash_with_embedding", False), - embedding_size=config.get("embedding_size"), - salt=config.get("salt") - ) - -# 4. Create and build the preprocessing model -model = PreprocessingModel( - path_data="product_dataset.csv", - features_specs=features +CategoricalFeature( + name="user_id", + feature_type=FeatureType.STRING_CATEGORICAL, + category_encoding=CategoryEncodingOptions.HASHING, + hash_bucket_size=1024, + salt=42, # optional, decorrelates two hashed columns + hash_with_embedding=True, # embedding instead of multi-hot + embedding_size=16, # width of that embedding ) -preprocessor = model.build_preprocessor() ``` -
-
-## 🔧 Real-World Examples - -
-
-

E-commerce Product Categorization

-
- -```python -# E-commerce features with hierarchical categories -preprocessor = PreprocessingModel( - path_data="products.csv", - features_specs={ - # Main category, subcategory, and detailed category - "main_category": FeatureType.STRING_CATEGORICAL, - "subcategory": FeatureType.STRING_CATEGORICAL, - "detailed_category": FeatureType.STRING_CATEGORICAL, - - # Product attributes as multi-categories - "product_features": CategoricalFeature( - name="product_features", - feature_type=FeatureType.MULTI_CATEGORICAL, - separator="|", - multi_hot=True - ), - - # Brand as a high-cardinality feature - "brand": CategoricalFeature( - name="brand", - feature_type=FeatureType.STRING_CATEGORICAL, - embedding_dim=16, - vocabulary_size=1000 # Top 1000 brands - ) - } -) -``` +Hashing maps values into a fixed bucket count, so it handles unbounded +cardinality and unseen values without growing. Collisions are the trade-off. -
-
+!!! tip "Hashing can skip the statistics pass entirely" + When **every** feature is a hashing categorical with an explicit + `hash_bucket_size`, nothing has to be learned from your data, so + `path_data` is not required and `build_preprocessor()` runs immediately. + Omit `hash_bucket_size` and the bucket count is derived from the + vocabulary, which does require a data pass. -
-

Content Recommendation System

-
+
```python -# Content recommendation with user and item features +from kdp import CategoricalFeature, CategoryEncodingOptions, FeatureType, PreprocessingModel + +# Builds with no dataset at all preprocessor = PreprocessingModel( - path_data="interaction_data.csv", features_specs={ - # User features "user_id": CategoricalFeature( name="user_id", - feature_type=FeatureType.STRING_HASHED, - hash_bucket_size=10000 - ), - "user_interests": CategoricalFeature( - name="user_interests", - feature_type=FeatureType.MULTI_CATEGORICAL, - embedding_dim=32, - separator="," - ), - - # Content features - "content_id": CategoricalFeature( - name="content_id", - feature_type=FeatureType.STRING_HASHED, - hash_bucket_size=5000 - ), - "content_tags": CategoricalFeature( - name="content_tags", - feature_type=FeatureType.MULTI_CATEGORICAL, - embedding_dim=24, - separator="|" + feature_type=FeatureType.STRING_CATEGORICAL, + category_encoding=CategoryEncodingOptions.HASHING, + hash_bucket_size=32, ), - "content_type": FeatureType.STRING_CATEGORICAL - } + }, ) +preprocessor.build_preprocessor() ``` -
-
-## 💎 Pro Tips - -
-
-

🔍 Choose Embedding Dimensions Wisely

-

For simple categories with few values (2-10), use 4-8 dimensions. For complex categories with many values (100+), use 16-64 dimensions. The more complex the relationships between categories, the higher dimensions you need.

-
- -
-

⚡ Pre-train Embeddings

-

KDP allows you to initialize embeddings with pre-trained vectors for faster convergence:

-
+## 📐 Output Widths -```python -# Create initial embeddings dictionary -pretrained = { - "sports": [0.1, 0.2, 0.3, 0.4], - "music": [0.5, 0.6, 0.7, 0.8] -} +
+ + + + + + + + + + + + + + + + + + + + + + + + + +
EncodingOutput width
Embeddingembedding_size (derived if unset)
One-hotvocabulary size
Hashing, multi-hot (default)hash_bucket_size
Hashing with embeddingembedding_size (default 8)
+
-# Use pre-trained embeddings -category_feature = CategoricalFeature( - name="interest", - feature_type=FeatureType.STRING_CATEGORICAL, - embedding_dim=4, - pretrained_embeddings=pretrained -) -``` +## 🔗 Combining With Other Features -
-
+### Crossing two categoricals -
-

🌀 Combine Multiple Encoding Strategies

-

For critical features, consider using both embeddings and one-hot encoding in parallel:

-
+
```python -# Main feature with embedding -features["product_type"] = CategoricalFeature( - name="product_type", - feature_type=FeatureType.STRING_CATEGORICAL, - use_embedding=True -) +from kdp import FeatureType, PreprocessingModel -# Same feature with one-hot encoding -features["product_type_onehot"] = CategoricalFeature( - name="product_type", - feature_type=FeatureType.STRING_CATEGORICAL, - use_embedding=False +preprocessor = PreprocessingModel( + path_data="data.csv", + features_specs={ + "city": FeatureType.STRING_CATEGORICAL, + "channel": FeatureType.STRING_CATEGORICAL, + }, + feature_crosses=[("city", "channel", 10)], ) ``` -
-
+
-
-

🔄 Handling Unknown Categories

-

Configure how KDP handles previously unseen categories in production:

-
+### Feature selection and attention + +
```python -feature = CategoricalFeature( - name="store_type", - feature_type=FeatureType.STRING_CATEGORICAL, - unknown_token="", # Custom token - oov_buckets=5 # Use 5 different embeddings +from kdp import FeatureType, PreprocessingModel + +preprocessor = PreprocessingModel( + path_data="data.csv", + features_specs={"city": FeatureType.STRING_CATEGORICAL}, + feature_selection_placement="categorical", # or "all_features" + tabular_attention=True, + tabular_attention_placement="categorical", + transfo_nr_blocks=2, # transformer over categoricals + transfo_placement="categorical", ) ``` -
-
-## 📊 Understanding Categorical Embeddings - -
-
- graph TD - A[Raw Category Data] -->|Vocabulary Creation| B[Category Vocabulary] - B -->|Lookup| C[Integer Indices] - C -->|Embedding Layer| D[Dense Vectors] +## 💎 Practical Notes - style A fill:#f9f9f9,stroke:#ccc,stroke-width:2px - style B fill:#e1f5fe,stroke:#4fc3f7,stroke-width:2px - style C fill:#e8f5e9,stroke:#66bb6a,stroke-width:2px - style D fill:#f3e5f5,stroke:#ce93d8,stroke-width:2px +
+
+

Reach for hashing on high cardinality

+

User and session IDs blow up a vocabulary. A fixed bucket count keeps the model the same size no matter how many values appear.

-
-

Categorical embeddings transform categorical values into dense vector representations that capture semantic relationships between categories.

+
+

One-hot only for small vocabularies

+

Width equals vocabulary size, so it grows linearly with the number of categories.

+
+
+

Salt when you hash two columns

+

Without different salts, the same value in two columns lands in the same bucket and the model cannot tell them apart.

+
+
+

Integer IDs are not numbers

+

Use INTEGER_CATEGORICAL, not a float type — otherwise the model reads store 7 as greater than store 3.

## 🔗 Related Topics
- - 🔢 - Numerical Features - - - 📝 - Text Features - - - 🧠 - Advanced Embedding Techniques - - - 📊 - Categorical Encoding Examples - -
- ---- - -
- - + 🧮 Categorical Hashing + ➕ Cross Features + 📝 Text Features + 🛠️ Features Overview