Introducing XDFlow: composable ML pipelines for structured scientific data
We present XDFlow, an open-source (MIT) framework for composable, dimension-aware, leakage-safe ML pipelines on labeled scientific data. Most tensor tooling discards the metadata—labeled dimensions and coordinates—that gives scientific data its meaning, forcing fragile glue code for splitting, alignment, leakage-safe preprocessing, and tuning. Built on xarray and integrating scikit-learn, Optuna, and MLflow, XDFlow composes pipelines from transforms that each declare what they consume and produce, whether they are stateful, and which choices they expose for tuning. This contract lets the framework check step compatibility, drive validation from coordinates already attached to the data, track provenance, and optimize execution—running fold-invariant steps once while refitting learned steps only within training folds. Unlike sklearn.Pipeline, coordinates, grouping, caching, and tuning live inside the pipeline rather than external glue. Originating from electrocorticography-based olfactory decoding, XDFlow generalizes to any domain where axes and coordinates are part of the experiment.
