nirs4all.controllers.models package

Subpackages

Submodules

Module contents

Model controllers module for nirs4all.

This module contains model controllers for different machine learning frameworks. All model controllers support training, fine-tuning with Optuna, and prediction modes.

Controllers follow the operator-controller pattern where: - Operators (in nirs4all.operators.models) define WHAT models to use - Controllers (here) define HOW to execute them

class nirs4all.controllers.models.AutoGluonModelController[source]

Bases: BaseModelController

Controller for AutoGluon TabularPredictor.

This controller handles AutoGluon models with automatic model selection, ensembling, and integration with the nirs4all pipeline.

AutoGluon automatically: - Trains multiple models (LightGBM, CatBoost, XGBoost, Neural Networks, etc.) - Performs cross-validation - Creates weighted ensembles - Handles hyperparameter tuning internally

Uses lazy loading - AutoGluon is only imported when training starts.

priority

Controller priority (5) - higher than sklearn (6) to prioritize AutoGluon when explicitly requested.

Type:

int

execute(step_info: ParsedStep, dataset: SpectroDataset, context: ExecutionContext, runtime_context: RuntimeContext, source: int = -1, mode: str = 'train', loaded_binaries: List[Tuple[str, bytes]] | None = None, prediction_store: Any | None = None) → Tuple[ExecutionContext, List[ArtifactMeta]][source]

Execute AutoGluon model controller.

Main entry point for AutoGluon model execution in the pipeline.

Parameters:

  • step_info – Parsed step containing model configuration.

  • dataset (SpectroDataset) – Dataset containing features and targets.

  • context (ExecutionContext) – Pipeline execution context.

  • runtime_context (RuntimeContext) – Runtime context.

  • source (int) – Source index. Defaults to -1.

  • mode (str) – Execution mode. Defaults to ‘train’.

  • loaded_binaries – Pre-loaded model binaries for prediction.

  • prediction_store – Store for managing predictions.

Returns:

Updated context

and list of model binaries.

Return type:

Tuple[ExecutionContext, List[ArtifactMeta]]

get_preferred_layout() → str[source]

Return the preferred data layout for AutoGluon.

Returns:

Data layout preference, ‘2d’ for AutoGluon.

Return type:

str

load_model(filepath: str) → Any[source]

Load AutoGluon model from disk.

Parameters:

filepath (str) – Path to the saved model directory.

Returns:

Loaded AutoGluon predictor.

Return type:

TabularPredictor

classmethod matches(step: Any, operator: Any, keyword: str) → bool[source]

Match AutoGluon TabularPredictor configurations.

Parameters:

  • step (Any) – Pipeline step to check.

  • operator (Any) – Optional operator object.

  • keyword (str) – Pipeline keyword (unused).

Returns:

True if the step matches an AutoGluon configuration.

Return type:

bool

priority: int = 5
save_model(model: Any, filepath: str) → None[source]

Save AutoGluon model to disk.

AutoGluon models are saved as directories. This method moves the model’s directory to the specified filepath.

Parameters:

  • model (TabularPredictor) – Trained AutoGluon predictor.

  • filepath (str) – Target path for saving.

class nirs4all.controllers.models.BaseModelController[source]

Bases: OperatorController, ABC

Abstract base controller for machine learning model training and prediction.

This controller provides a unified interface for training, finetuning, and predicting with machine learning models across different frameworks (scikit-learn, TensorFlow, PyTorch). It implements cross-validation, fold averaging, hyperparameter optimization, and comprehensive prediction tracking.

The controller delegates framework-specific operations to subclasses while handling:

  • Cross-validation fold management

  • Model identification and naming

  • Prediction storage and tracking

  • Score calculation and aggregation

  • Fold-averaged predictions (simple and weighted)

optuna_manager

Manager for hyperparameter optimization.

Type:

OptunaManager

identifier_generator

Component for model naming.

Type:

ModelIdentifierGenerator

prediction_transformer

Component for prediction scaling.

Type:

PredictionTransformer

prediction_assembler

Component for assembling prediction records.

Type:

PredictionDataAssembler

score_calculator

Component for calculating evaluation scores.

Type:

ScoreCalculator

index_normalizer

Component for normalizing sample indices.

Type:

IndexNormalizer

prediction_store

External storage for predictions.

Type:

Predictions

verbose

Verbosity level for logging.

Type:

int

execute(step_info: ParsedStep, dataset: SpectroDataset, context: ExecutionContext, runtime_context: RuntimeContext, source: int = -1, mode: str = 'train', loaded_binaries: List[Tuple[str, bytes]] | None = None, prediction_store: Predictions = None) → Tuple[ExecutionContext, List[ArtifactMeta]][source]

Execute model training, finetuning, or prediction.

This is the main entry point for model execution. It handles:

  • Extracting model configuration

  • Restoring task type in predict/explain modes

  • Delegating to finetune() or train() based on configuration

  • Managing prediction storage

Parameters:

  • step_info – Parsed step containing model configuration and operator.

  • dataset – SpectroDataset with features and targets.

  • context – Execution context with step_id, partition info, etc.

  • runtime_context – Runtime context managing execution state.

  • source – Data source index (default: -1).

  • mode – Execution mode (‘train’, ‘finetune’, ‘predict’, ‘explain’).

  • loaded_binaries – Optional list of (name, bytes) tuples for prediction mode.

  • prediction_store – External Predictions storage instance.

Returns:

Tuple of (updated_context, list_of_artifact_metadata).

finetune(dataset: SpectroDataset, model_config: Dict[str, Any], X_train: Any, y_train: Any, X_test: Any, y_test: Any, folds: List | None, finetune_params: Dict[str, Any], predictions: Dict, context: ExecutionContext, runtime_context: RuntimeContext) → Dict[str, Any] | List[Dict[str, Any]][source]

Optimize hyperparameters using Optuna.

Delegates to OptunaManager for Bayesian hyperparameter optimization. Returns optimized parameters that will be used in subsequent training.

Parameters:

  • dataset – SpectroDataset for optimization.

  • model_config – Base model configuration.

  • X_train – Training features.

  • y_train – Training targets.

  • X_test – Test features.

  • y_test – Test targets.

  • folds – List of (train_idx, val_idx) tuples for cross-validation.

  • finetune_params – Optuna configuration with search space and trials.

  • predictions – Prediction storage dictionary.

  • context – Execution context.

  • runtime_context – Runtime context.

Returns:

Dictionary of optimized parameters (single model) or list of dicts (per-fold).

get_effective_layout(step_info: ParsedStep | None = None) → str[source]

Get effective data layout, respecting force_layout if specified.

This method checks if the step configuration has a force_layout override. If not, it falls back to the controller’s preferred layout.

Parameters:

step_info – ParsedStep containing potential force_layout override.

Returns:

Data layout string to use for this step.

get_preferred_layout() → str[source]

Get preferred data layout for the framework.

Returns:

Data layout string (‘2d’ for NumPy arrays, ‘3d’ for TensorFlow, etc.).

Note

Override in subclasses for framework-specific layouts.

get_xy(dataset: SpectroDataset, context: ExecutionContext) → Tuple[Any, Any, Any, Any, Any, Any][source]

Extract train/test splits with scaled and unscaled targets.

For classification tasks, both scaled and unscaled targets are transformed. For regression tasks, scaled targets are used for training while unscaled (numeric) targets are used for evaluation.

In prediction mode, uses all available data (partition=None) instead of splitting.

Also handles sample_partitioner branches, which restrict data to a subset of samples.

Parameters:

  • dataset – SpectroDataset with partitioned data.

  • context – Execution context with partition and preprocessing info.

Returns:

Tuple of (X_train, y_train, X_test, y_test, y_train_unscaled, y_test_unscaled).

launch_training(dataset, model_config, context, runtime_context, prediction_store, X_train, y_train, X_val, y_val, X_test, y_train_unscaled, y_val_unscaled, y_test_unscaled, train_indices=None, val_indices=None, fold_idx=None, best_params=None, loaded_binaries=None, mode='train', test_sample_ids=None)[source]

Execute single model training or prediction.

This refactored method uses modular components to handle: - Model identification and naming - Model loading for predict/explain modes - Training execution - Prediction transformation - Score calculation - Prediction data assembly

Parameters:

  • dataset – SpectroDataset instance

  • model_config – Model configuration dictionary

  • context – Execution context with step_id, y processing, etc.

  • runtime_context – Runtime context.

  • prediction_store – Predictions storage instance

  • X_train – Training data (scaled)

  • y_train – Training data (scaled)

  • X_val – Validation data (scaled)

  • y_val – Validation data (scaled)

  • X_test – Test data (scaled)

  • y_train_unscaled – True values (unscaled)

  • y_val_unscaled – True values (unscaled)

  • y_test_unscaled – True values (unscaled)

  • train_indices – Sample indices for each partition

  • val_indices – Sample indices for each partition

  • fold_idx – Optional fold index for CV

  • best_params – Optional hyperparameters from optimization

  • loaded_binaries – Optional binaries for predict/explain mode

  • mode – Execution mode (‘train’, ‘finetune’, ‘predict’, ‘explain’)

  • test_sample_ids – List of actual sample IDs for test partition (for stacking).

Returns:

Tuple of (trained_model, model_id, val_score, model_name, prediction_data)

load_model(filepath: str) → Any[source]

Optional: Load model from framework-specific format.

Default implementation delegates to artifact_serialization.load(). Subclasses can override to use framework-specific loading.

Parameters:

filepath – Path to load from.

Returns:

Loaded model instance.

priority: int = 15
process_hyperparameters(params: Dict[str, Any]) → Dict[str, Any][source]

Process hyperparameters before use.

Can be overridden by subclasses to structure parameters (e.g. nesting for TensorFlow).

Parameters:

params – Flat dictionary of sampled parameters.

Returns:

Processed dictionary of parameters.

save_model(model: Any, filepath: str) → None[source]

Optional: Save model in framework-specific format.

Default implementation delegates to artifact_serialization.persist(). Subclasses can override to use framework-specific formats: - TensorFlow: .h5 or .keras format - PyTorch: .ckpt or .pt format - sklearn: .joblib format

Parameters:

  • model – Trained model to save.

  • filepath – Path to save (without extension, will be added by implementation).

classmethod supports_prediction_mode() → bool[source]

Check if the controller should execute during prediction mode.

Returns:

True if the controller should execute in prediction mode, False if it should be skipped (e.g., chart controllers)

train(dataset, model_config, context, runtime_context, prediction_store, X_train, y_train, X_test, y_test, y_train_unscaled, y_test_unscaled, folds, best_params=None, loaded_binaries=None, mode='train', train_sample_ids=None, test_sample_ids=None) → List[ArtifactMeta][source]

Orchestrate model training across folds with prediction tracking.

Manages the complete training workflow:

  • Iterates through cross-validation folds

  • Delegates to launch_training() for each fold

  • Creates fold-averaged predictions for regression tasks

  • Persists trained models as artifacts

  • Stores all predictions with weights

Parameters:

  • dataset – SpectroDataset with features and targets.

  • model_config – Model configuration dictionary.

  • context – Execution context with step_id and preprocessing info.

  • runtime_context – Runtime context.

  • prediction_store – External Predictions storage.

  • X_train – Training features (all folds).

  • y_train – Training targets (scaled).

  • X_test – Test features.

  • y_test – Test targets (scaled).

  • y_train_unscaled – Training targets (unscaled for evaluation).

  • y_test_unscaled – Test targets (unscaled for evaluation).

  • folds – List of (train_idx, val_idx) tuples or empty list.

  • best_params – Optional hyperparameters from finetuning.

  • loaded_binaries – Optional model binaries for prediction mode.

  • mode – Execution mode (‘train’, ‘finetune’, ‘predict’, ‘explain’).

  • train_sample_ids – List of actual sample IDs for train partition.

  • test_sample_ids – List of actual sample IDs for test partition.

Returns:

List of ArtifactMeta objects for persisted models.

classmethod use_multi_source() → bool[source]

Check if the operator supports multi-source datasets.

class nirs4all.controllers.models.FoldAlignmentValidator(prediction_store: Predictions, config: ReconstructorConfig | None = None)[source]

Bases: object

Validates fold structure consistency across source models.

Ensures that all source models have compatible fold structures for proper out-of-fold reconstruction.

Checks performed: 1. All models have the same number of folds. 2. Fold indices are sequential (0, 1, 2, …, K-1). 3. No sample appears in multiple validation sets within a model. 4. Sample indices are consistent across folds.

prediction_store

Predictions storage for accessing fold data.

config

Reconstructor configuration.

validate(source_model_names: List[str], context: ExecutionContext, branch_id_override: int | None = -1) → ValidationResult[source]

Validate fold alignment across source models.

Parameters:

  • source_model_names – List of source model names to validate.

  • context – Execution context with branch info.

  • branch_id_override – Optional branch_id override. If -1 (default), use context’s branch_id. If None, don’t filter by branch (for ALL_BRANCHES scope).

Returns:

ValidationResult with any errors or warnings.

class nirs4all.controllers.models.JaxModelController[source]

Bases: BaseModelController

Controller for JAX/Flax models.

Uses lazy loading pattern - JAX is only imported when training or prediction is actually performed.

execute(step_info: ParsedStep, dataset: SpectroDataset, context: ExecutionContext, runtime_context: RuntimeContext, source: int = -1, mode: str = 'train', loaded_binaries: List[Tuple[str, bytes]] | None = None, prediction_store: Predictions = None) → Tuple[ExecutionContext, List[Tuple[str, bytes]]][source]

Execute JAX model controller.

get_preferred_layout() → str[source]

Return the preferred data layout for JAX models.

Flax Dense layers expect (batch, features). Flax Conv layers expect (batch, length, features) i.e. (N, L, C). So ‘3d_transpose’ is suitable for Conv1D.

classmethod matches(step: Any, operator: Any, keyword: str) → bool[source]

Match JAX models and model configurations.

priority: int = 4
process_hyperparameters(params: Dict[str, Any]) → Dict[str, Any][source]

Process hyperparameters for JAX model tuning.

class nirs4all.controllers.models.MetaModelController[source]

Bases: SklearnModelController

Controller for meta-model stacking using pipeline predictions.

This controller handles MetaModel operators, constructing training features from out-of-fold predictions of previous models. It extends SklearnModelController since the meta-learner is always sklearn-compatible.

The key difference from regular model controllers is that get_xy() returns features constructed from predictions rather than the original dataset features.

Key Behavior:

  • Works INDEPENDENTLY of branches (no branch awareness required for basic case)

  • Queries prediction_store for ALL models from previous steps

  • Does NOT modify execution context (unlike MergeController)

  • For branch-aware stacking, uses BranchScope configuration

priority

Controller priority (5) - higher than SklearnModelController (6) to ensure MetaModel operators are handled by this controller.

Type:

int

use_reconstructor

If True, use TrainingSetReconstructor for OOF.

Type:

bool

execute(step_info: ParsedStep, dataset: SpectroDataset, context: ExecutionContext, runtime_context: RuntimeContext, source: int = -1, mode: str = 'train', loaded_binaries: List[Tuple[str, bytes]] | None = None, prediction_store: Any | None = None) → Tuple[ExecutionContext, List[Tuple[str, bytes]]][source]

Execute meta-model controller.

Stores MetaModel operator and prediction_store in context for use by get_xy(). Also stores source models for artifact persistence in Phase 3.

Parameters:

  • step_info – Parsed step with MetaModel operator.

  • dataset – SpectroDataset.

  • context – Execution context.

  • runtime_context – Runtime context.

  • source – Data source index.

  • mode – Execution mode.

  • loaded_binaries – Pre-loaded model binaries.

  • prediction_store – Predictions store.

Returns:

Tuple of (updated_context, list_of_binaries).

get_xy(dataset: SpectroDataset, context: ExecutionContext) → Tuple[Any, Any, Any, Any, Any, Any][source]

Extract train/test splits using meta-features from predictions.

Instead of using the original dataset features, this constructs features from out-of-fold predictions of source models.

For training:

  • X_train: OOF predictions from source models (n_train_samples, n_source_models)

  • y_train: Original target values

For test:

  • X_test: Aggregated source model test predictions

  • y_test: Original target values

Parameters:

  • dataset – SpectroDataset with partitioned data.

  • context – Execution context with partition and branch info.

Returns:

Tuple of (X_train, y_train, X_test, y_test, y_train_unscaled, y_test_unscaled) where X_train and X_test are meta-features from predictions.

classmethod matches(step: Any, operator: Any, keyword: str) → bool[source]

Match MetaModel operators.

Parameters:

  • step – Pipeline step configuration.

  • operator – Instantiated operator object.

  • keyword – Pipeline keyword (unused).

Returns:

True if the operator is a MetaModel instance.

priority: int = 5
use_reconstructor: bool = True
class nirs4all.controllers.models.PyTorchModelController[source]

Bases: BaseModelController

Controller for PyTorch models.

Uses lazy loading pattern - PyTorch is only imported when training or prediction is actually performed.

execute(step_info: ParsedStep, dataset: SpectroDataset, context: ExecutionContext, runtime_context: RuntimeContext, source: int = -1, mode: str = 'train', loaded_binaries: List[Tuple[str, bytes]] | None = None, prediction_store: Predictions = None) → Tuple[ExecutionContext, List[Tuple[str, bytes]]][source]

Execute PyTorch model controller.

get_preferred_layout() → str[source]

Return the preferred data layout for PyTorch models.

PyTorch typically expects (samples, channels, features) for 1D convs. We use ‘3d’ which gives (samples, processings, features) -> (N, C, L).

classmethod matches(step: Any, operator: Any, keyword: str) → bool[source]

Match PyTorch models and model configurations.

priority: int = 4
process_hyperparameters(params: Dict[str, Any]) → Dict[str, Any][source]

Process hyperparameters for PyTorch model tuning.

class nirs4all.controllers.models.ReconstructionResult(X_train_meta: ndarray, X_test_meta: ndarray, y_train: ndarray, y_test: ndarray, feature_names: List[str], source_models: List[str], valid_train_mask: ndarray, valid_test_mask: ndarray, validation_result: ValidationResult, n_folds: int, coverage_ratio: float, meta_feature_info: Any | None = None, classification_info: Any | None = None)[source]

Bases: object

Container for reconstructed training set data.

Holds the meta-feature matrices for training and test sets, along with metadata about the reconstruction process.

X_train_meta

Training meta-features (n_train_samples, n_features).

Type:

numpy.ndarray

X_test_meta

Test meta-features (n_test_samples, n_features).

Type:

numpy.ndarray

y_train

Training targets (n_train_samples,).

Type:

numpy.ndarray

y_test

Test targets (n_test_samples,).

Type:

numpy.ndarray

feature_names

List of feature column names.

Type:

List[str]

source_models

List of source model names used.

Type:

List[str]

valid_train_mask

Boolean mask of valid training samples (after coverage handling).

Type:

numpy.ndarray

valid_test_mask

Boolean mask of valid test samples.

Type:

numpy.ndarray

validation_result

Validation result from fold alignment.

Type:

nirs4all.controllers.models.stacking.reconstructor.ValidationResult

n_folds

Number of folds detected.

Type:

int

coverage_ratio

Ratio of samples with complete predictions.

Type:

float

meta_feature_info

Optional MetaFeatureInfo for feature importance tracking.

Type:

Any | None

classification_info

Optional ClassificationInfo for task type metadata.

Type:

Any | None

Example

>>> result = reconstructor.reconstruct(dataset, context)
>>> X_train = result.X_train_meta[result.valid_train_mask]
>>> y_train = result.y_train[result.valid_train_mask]
>>> # For feature importance tracking
>>> if result.meta_feature_info:
...     model_importance = result.meta_feature_info.aggregate_importance_by_model(
...         feature_importances
...     )
X_test_meta: ndarray
X_train_meta: ndarray
classification_info: Any | None = None
coverage_ratio: float
feature_names: List[str]
meta_feature_info: Any | None = None
n_folds: int
source_models: List[str]
valid_test_mask: ndarray
valid_train_mask: ndarray
validation_result: ValidationResult
y_test: ndarray
y_train: ndarray
class nirs4all.controllers.models.ReconstructorConfig(validate_fold_alignment: bool = True, validate_sample_coverage: bool = True, log_warnings: bool = True, max_missing_fold_ratio: float = 0.0, allow_partial_sources: bool = False, feature_name_pattern: str = '{model_name}_pred', excluded_fold_ids: ~typing.Set[str] = <factory>)[source]

Bases: object

Configuration for TrainingSetReconstructor.

Controls internal behavior of the reconstruction process, separate from the user-facing StackingConfig.

validate_fold_alignment

Whether to validate fold structure consistency.

Type:

bool

validate_sample_coverage

Whether to validate sample coverage.

Type:

bool

log_warnings

Whether to log warnings during reconstruction.

Type:

bool

max_missing_fold_ratio

Maximum ratio of missing folds before error (0.0-1.0).

Type:

float

allow_partial_sources

Allow reconstruction if some sources have partial data.

Type:

bool

feature_name_pattern

Pattern for generating feature column names. Supports: {model_name}, {fold_id}, {classname}, {step_idx}

Type:

str

excluded_fold_ids

Set of fold_id values to exclude (e.g., {‘avg’, ‘w_avg’}).

Type:

Set[str]

Example

>>> config = ReconstructorConfig(
...     validate_fold_alignment=True,
...     log_warnings=True,
...     feature_name_pattern="{model_name}_pred"
... )
__post_init__()[source]

Validate configuration after initialization.

allow_partial_sources: bool = False
excluded_fold_ids: Set[str]
feature_name_pattern: str = '{model_name}_pred'
log_warnings: bool = True
max_missing_fold_ratio: float = 0.0
validate_fold_alignment: bool = True
validate_sample_coverage: bool = True
class nirs4all.controllers.models.SklearnModelController[source]

Bases: BaseModelController

Controller for scikit-learn models.

This controller handles sklearn models with support for training on 2D data, cross-validation, hyperparameter tuning with Optuna, model persistence, and integration with the nirs4all pipeline.

priority

Controller priority (6) - higher than TransformerMixin to prioritize supervised models over transformers.

Type:

int

execute(step_info: ParsedStep, dataset: SpectroDataset, context: ExecutionContext, runtime_context: RuntimeContext, source: int = -1, mode: str = 'train', loaded_binaries: List[Tuple[str, bytes]] | None = None, prediction_store: Any | None = None) → Tuple[ExecutionContext, List[Tuple[str, bytes]]][source]

Execute sklearn model controller with score management.

Main entry point for sklearn model execution in the pipeline. Sets the preferred data layout to ‘2d’ and delegates to parent execute method.

Parameters:

  • step_info – Parsed step containing model configuration and operator.

  • dataset (SpectroDataset) – Dataset containing features and targets.

  • context (ExecutionContext) – Pipeline execution context with state info.

  • runtime_context (RuntimeContext) – Runtime context managing execution state.

  • source (int) – Source index for multi-source pipelines. Defaults to -1.

  • mode (str) – Execution mode (‘train’ or ‘predict’). Defaults to ‘train’.

  • loaded_binaries (Optional[List[Tuple[str, bytes]]]) – Pre-loaded model binaries for prediction mode. Defaults to None.

  • prediction_store (Optional[Any]) – Store for managing predictions. Defaults to None.

Returns:

Updated context and

list of model binaries (name, serialized_model) for persistence.

Return type:

Tuple[ExecutionContext, List[Tuple[str, bytes]]]

Note

  • Automatically sets context[‘layout’] = ‘2d’ for sklearn compatibility

  • Inherits full training, evaluation, and prediction logic from BaseModelController

  • Respects force_layout if specified in step configuration

get_preferred_layout() → str[source]

Return the preferred data layout for sklearn models.

Returns:

Data layout preference, always ‘2d’ for sklearn models which

expect (n_samples, n_features) input format.

Return type:

str

classmethod matches(step: Any, operator: Any, keyword: str) → bool[source]

Match sklearn estimators and model dictionaries with sklearn models.

Prioritizes supervised models (regressors and classifiers) over transformers by checking for predict methods and using sklearn’s is_regressor/is_classifier.

Parameters:

  • step (Any) – Pipeline step to check, can be a dict with ‘model’ key or BaseEstimator instance.

  • operator (Any) – Optional operator object to check if it’s a BaseEstimator.

  • keyword (str) – Pipeline keyword (unused in this implementation).

Returns:

True if the step matches a sklearn estimator (regressor, classifier,

or has predict method), False otherwise.

Return type:

bool

priority: int = 6
class nirs4all.controllers.models.TensorFlowModelController[source]

Bases: BaseModelController

Controller for TensorFlow/Keras models.

This controller manages the complete lifecycle of TensorFlow/Keras models including: - Model instantiation from various configuration formats - Data preparation with proper tensor formatting (2D/3D) - Model compilation with task-appropriate loss functions and metrics - Training with callbacks (early stopping, model checkpointing) - Hyperparameter tuning via Optuna integration - Model evaluation and prediction - Binary serialization for model persistence

The controller automatically detects TensorFlow models and functions decorated with @framework(‘tensorflow’). It uses lazy loading to avoid importing TensorFlow until actually needed.

priority

Controller priority for matching (4).

Type:

int

execute(step_info: ParsedStep, dataset: SpectroDataset, context: ExecutionContext, runtime_context: RuntimeContext, source: int = -1, mode: str = 'train', loaded_binaries: List[Tuple[str, bytes]] | None = None, prediction_store: Predictions = None) → Tuple[ExecutionContext, List[Tuple[str, bytes]]][source]

Execute TensorFlow model training, finetuning, or prediction.

Sets the preferred data layout to ‘3d_transpose’ for TensorFlow Conv1D models, then delegates to the base class execute method.

Parameters:

  • step_info – Parsed step containing model configuration and operator.

  • dataset – SpectroDataset with features, targets, and fold information.

  • context – Execution context with step_id, processing history, partition info.

  • runtime_context – Runtime context managing execution state.

  • source – Data source index (default: -1 for primary source).

  • mode – Execution mode - ‘train’, ‘finetune’, ‘predict’, or ‘explain’.

  • loaded_binaries – Optional list of (name, bytes) tuples for prediction mode, containing serialized model and preprocessing artifacts.

  • prediction_store – External Predictions storage instance for managing prediction results across pipeline steps.

Returns:

  • updated_context: Context dict with added model information

  • artifact_metadata: List of serialized binary artifacts for persistence

Return type:

Tuple of (updated_context, list_of_artifact_metadata) where

Raises:

ImportError – If TensorFlow is not installed.

get_preferred_layout() → str[source]

Return the preferred data layout for TensorFlow models.

TensorFlow Conv1D expects input shape (features, channels) where: - features = number of wavelengths/spectral points (timesteps for convolution) - channels = number of preprocessing methods

The ‘3d_transpose’ layout returns (samples, features, processings) which is correct for Conv1D.

classmethod matches(step: Any, operator: Any, keyword: str) → bool[source]

Determine if this controller should handle the given step.

Matches TensorFlow/Keras models, functions decorated with @framework(‘tensorflow’), and serialized model configurations containing TensorFlow components.

Parameters:

  • step – Pipeline step configuration (dict, model instance, or function).

  • operator – Optional operator instance extracted from step.

  • keyword – Optional keyword identifier for the step.

Returns:

True if this controller should handle the step, False otherwise. Returns False immediately if TensorFlow is not installed.

priority: int = 4
process_hyperparameters(params: Dict[str, Any]) → Dict[str, Any][source]

Process hyperparameters for TensorFlow model tuning.

Supports TensorFlow-specific parameter organization: - Parameters prefixed with ‘compile_’ are grouped under ‘compile’ key

(e.g., ‘compile_learning_rate’ → compile[‘learning_rate’])

  • Parameters prefixed with ‘fit_’ are grouped under ‘fit’ key (e.g., ‘fit_batch_size’ → fit[‘batch_size’])

  • Other parameters are treated as model architecture parameters

Parameters:

params – Dictionary of sampled parameters.

Returns:

Dictionary of processed hyperparameters with proper nesting for TensorFlow compilation and fitting.

class nirs4all.controllers.models.TrainingSetReconstructor(prediction_store: Predictions, source_model_names: List[str], stacking_config: StackingConfig | None = None, reconstructor_config: ReconstructorConfig | None = None, source_model_branch_map: Dict[str, int | None] | None = None)[source]

Bases: object

Reconstructs meta-model training set from out-of-fold predictions.

This is the core class for Phase 2 of the meta-model stacking implementation. It handles the critical task of collecting OOF predictions from source models and constructing feature matrices for the meta-learner.

The fundamental invariant is: No sample sees predictions from a model trained on that sample. This prevents data leakage.

prediction_store

Predictions storage for accessing source predictions.

source_model_names

List of source model names to use.

stacking_config

Configuration for coverage and aggregation strategies.

reconstructor_config

Internal configuration for reconstruction.

fold_validator

Validator for fold alignment.

Example

>>> reconstructor = TrainingSetReconstructor(
...     prediction_store=predictions,
...     source_model_names=["PLS", "RF", "XGB"],
...     stacking_config=StackingConfig(
...         coverage_strategy=CoverageStrategy.DROP_INCOMPLETE,
...         test_aggregation=TestAggregation.MEAN
...     )
... )
>>> result = reconstructor.reconstruct(dataset, context)
>>> print(f"Coverage: {result.coverage_ratio:.1%}")
>>> print(f"Features: {result.feature_names}")
reconstruct(dataset: SpectroDataset, context: ExecutionContext, y_train: ndarray | None = None, y_test: ndarray | None = None, use_proba: bool = False) → ReconstructionResult[source]

Reconstruct meta-model training and test sets from predictions.

Collects out-of-fold predictions for training samples and aggregated predictions for test samples.

Phase 5 Enhancement: Supports classification tasks with probability features for binary and multiclass classification.

Parameters:

  • dataset – SpectroDataset for sample indices.

  • context – Execution context with partition and branch info.

  • y_train – Optional pre-computed training targets.

  • y_test – Optional pre-computed test targets.

  • use_proba – If True, use probability predictions for classification.

Returns:

ReconstructionResult containing meta-feature matrices and metadata.

Raises:

ValueError – If no source models found or critical validation fails.

validate_branch_compatibility(context: ExecutionContext) → ValidationResult[source]

Validate branch compatibility for stacking.

Checks that the current branch context is compatible with stacking based on the configured BranchScope.

Parameters:

context – Execution context with branch info.

Returns:

ValidationResult with any errors or warnings.

class nirs4all.controllers.models.ValidationResult(errors: ~typing.List[~nirs4all.controllers.models.stacking.reconstructor.ValidationError] = <factory>, warnings: ~typing.List[~nirs4all.controllers.models.stacking.reconstructor.ValidationWarning] = <factory>)[source]

Bases: object

Container for validation errors and warnings.

Accumulates validation issues during fold alignment and coverage checks.

errors

List of validation errors (critical issues).

Type:

List[nirs4all.controllers.models.stacking.reconstructor.ValidationError]

warnings

List of validation warnings (non-critical issues).

Type:

List[nirs4all.controllers.models.stacking.reconstructor.ValidationWarning]

is_valid

True if no errors (warnings are allowed).

Example

>>> result = ValidationResult()
>>> result.add_error("FOLD_MISMATCH", "Folds don't align")
>>> result.add_warning("PARTIAL_COVERAGE", "80% coverage")
>>> if not result.is_valid:
...     raise ValueError(result.format_errors())
add_error(code: str, message: str, details: Dict[str, Any] | None = None) → None[source]

Add a validation error.

add_warning(code: str, message: str, details: Dict[str, Any] | None = None) → None[source]

Add a validation warning.

errors: List[ValidationError]
format_errors() → str[source]

Format all errors as a human-readable string.

format_warnings() → str[source]

Format all warnings as a human-readable string.

property is_valid: bool

Check if validation passed (no errors).

merge(other: ValidationResult) → None[source]

Merge another validation result into this one.

warnings: List[ValidationWarning]