nirs4all.pipeline.execution.executor module

Pipeline executor for executing a single pipeline on a single dataset.

class nirs4all.pipeline.execution.executor.PipelineExecutor(step_runner: StepRunner, manifest_manager: ManifestManager | None = None, verbose: int = 0, mode: str = 'train', continue_on_error: bool = False, saver: Any = None, artifact_loader: Any = None, artifact_registry: Any = None)

Bases: object

Executes a single pipeline configuration on a single dataset.

Handles: - Step-by-step execution - Context propagation - Artifact management for one pipeline run - Predictions accumulation for this pipeline

step_runner

Executes individual steps

manifest_manager

Manages pipeline manifests

verbose

Verbosity level

mode

Execution mode (train/predict/explain)

continue_on_error

Whether to continue on step failures

artifact_registry

Registry for v2 artifact management

execute(steps: List[Any], config_name: str, dataset: SpectroDataset, context: ExecutionContext, runtime_context: Any, prediction_store: Predictions | None = None, generator_choices: List[Dict[str, Any]] | None = None) → None

Execute pipeline steps sequentially on dataset.

Parameters:

• steps – List of pipeline steps to execute

• config_name – Pipeline configuration name

• dataset – Dataset to process

• context – Initial execution context

• runtime_context – Runtime infrastructure context

• prediction_store – Prediction store for accumulating results

• generator_choices – List of generator choices that produced this pipeline

Raises:

RuntimeError – If pipeline execution fails

execute_minimal(steps: List[Any], minimal_pipeline: Any, dataset: SpectroDataset, context: ExecutionContext, runtime_context: Any, prediction_store: Predictions | None = None) → None

Execute minimal pipeline for prediction.

This method executes only the steps from a MinimalPipeline, which represents the subset of the full pipeline needed to replay a prediction. It’s the key optimization of Phase 5: instead of replaying the entire original pipeline, we only run the required steps.

The method: 1. Uses the minimal pipeline’s step list (not full pipeline) 2. Injects artifacts via the artifact_provider in runtime_context 3. Runs controllers in predict mode 4. Skips steps not in the minimal pipeline

Parameters:

• steps – List of step configs (from minimal_pipeline.steps[i].step_config)

• minimal_pipeline – MinimalPipeline with artifact mappings

• dataset – Dataset to process

• context – Execution context

• runtime_context – Runtime context with artifact_provider

• prediction_store – Optional prediction store

Note

The artifact_provider in runtime_context should be a MinimalArtifactProvider that provides artifacts by step index from the MinimalPipeline.

initialize_context(dataset: SpectroDataset) → ExecutionContext

Initialize ExecutionContext for pipeline execution.

Parameters:

dataset – Dataset to create context for

Returns:

Initialized ExecutionContext

next_op() → int

Get the next operation ID (for compatibility).