pip install xdflow
At General Sense, we build olfactory brain-computer interface systems for chemical sensing. These systems are powered by machine learning pipelines that operate on neural recordings rich with structure, context, and metadata.
We treat this metadata as inseparable from the data itself. Coordinates and dimensions are not annotations layered onto an array,but the very things that give it meaning. This philosophy is difficult to realize with most array- and tensor-based tooling, which either rejects labeled inputs outright or strips them to raw arrays, forcing manual tracking of coordinates and metadata. In our view, separating metadata and dimensions from data is like splitting your grandmother's handwritten pozole recipe into two sheets of paper, one with the quantities and another with the ingredients those quantities refer to. Don't do that. You're going to mess it up. Don't mess up pozole.
We use xarray as the foundation for our data representation, since its labeled dimensions and coordinates let us encode this structure directly. But xarray is primarily a data representation layer. It does not by itself provide the ML pipeline semantics needed for modeling, validation, tuning, and leakage-safe execution.
In parallel, the transformations that process these data have distinct behaviors. Some are stateful: they learn from the data and must be refit inside each cross-validation fold. Others are stateless: they apply a fixed transformation to each sample and run once. Like xarray's metadata, these behaviors can be leveraged for more efficient computation, but only if the framework is built to exploit them.
To solve this problem, we built XDFlow, a framework that uses dimensions, coordinates, and transform behavior to make pipelines modular, efficient, and resistant to leakage. Each step operates on labeled scientific data rather than anonymous arrays, and validators and tuners use declared dimensions, coordinates, and transform state to decide how the pipeline should run.
Where rigid pipelines break
Initially, we built a powerful but inflexible task-specific framework. It knew about the stages we commonly used —preprocessing, featurization, classification, validation, tuning — and it also understood the shape of the data flowing between them. It tracked named dimensions and coordinates as the data moved through, checked that each stage received the dimensions it expected, and kept the experimental structure attached to the outputs rather than letting it fall away at the first reshape. On the execution side, it ran stateless preprocessing before cross-validation splits, cached fold-invariant work instead of recomputing it on every fold, refit learned steps only inside training folds, exposed preprocessors, featurizers, and classifiers as hot-swappable choices for a tuner, and enforced expectations about what each stage consumed and produced. The usefulness of the framework stemmed from its encoding and leveraging of both the structure of the data and the behavior of the transformations.
But as our modeling work became more creative, we found ourselves wanting to try architectures that did not fit the fixed stages. Each extension required another hook, another base-class change, or another special case. As soon as we wanted to explore workflows outside that rigid structure, the same assumptions that made it useful started to feel like a cage.
We realized then that the abstraction boundary was in the wrong place. We did not need a slightly more flexible preprocessor-featurizer-classifier framework. We needed a composable execution framework that preserved the same optimization and validation benefits without committing to one fixed architecture. XDFlow was born.
The XDFlow contract
InXDFlow, pipelines are built from composable transforms arranged sequentially orin parallel. A transform can also split by a coordinate and regroup (e.g. a parallel split by subject ID), branch conditionally on coordinate values, or be toggled by the tuner. Each transform declares what it consumes and produces, whether it learns from data, and which choices it exposes for tuning, while coordinates stay attached through the data container. That contract is what lets XDFlow reason about a whole pipeline it didn't have hardcoded: it checks that steps are compatible, decides what can be cached and what must be refit, and keeps predictions, scores, and out-of-fold outputs tied to the coordinates that produced them.
Inside the pipeline, the metadata drives the transform logic. Instead of "z-score along axis 0," a transform name sits dimensions: per_dim=["trial"] means “z-score each trial independently,” without relying on whichever integer axis trial happens to occupy. There is no need to remember that time used to be axis 2 before some earlier step changed the shape. Validation is directed in the same way, so a model can decode stimulus, split by session, group by subject, or stratify by a target coordinate.
This discipline increases robustness. When transforms declare what they expect and produce, the framework catches incompatibilities earlier, preventing many scientific ML bugs that are not syntax errors but rather silent shape, alignment, leakage, and validation errors. During exploration it is common for researchers to print the shape of the data after every transformation as a proxy for checking dimensions; XDFlow makes that brittle ritual moot by integrating dimensions into the object itself, so the pipeline verifies them as it runs.
Additionally, XDFlow tracks data provenance, so data always comes bundled with a record of how it was processed. As a pipeline runs, each transform adds itself and its parameters to a transformation log attached to the data. When the data is inspected downstream, or reloaded after saving, you can see exactly how it was produced. It is a lightweight provenance feature, but it makes replication and bug-finding much easier.
Compute it once
In many scientific ML workflows, the expensive step is not the classifier but the preprocessing. Filtering, dimensionality reduction, artifact rejection, and feature extraction can each dwarf the model in cost. In a cross-validation and tuning loop, that cost is multiplied.
Certain characteristics of the transforms used in a pipeline can be leveraged to make cross validation and tuning more efficient. A stateless step is fold-invariant, so a deterministic spectral transform never needs to be recomputed per-fold. A stateful step like PCA or a classifier must be fit only on the training fold, or validation leaks. Before cross-validation, XDFlow traverses the pipeline to find the boundary where the first stateful transform appears. Everything upstream of that boundary is run once on the full dataset, and everything downstream is cloned and refit inside each training fold. A second level of caching applies during tuning. When a stateful transform's hyperparameters do not change between trials, the result of fitting and applying it is cached and reused across those trials rather than recomputed. The pipeline avoids recomputing anything it can safety reuse.
This composable framework matches the optimization quality of a fixed one without inheriting its rigidity. The fixed framework had its checks and caching wired into known points in a known shape, while XDFlow identifies the same opportunities from the declared structure of the pipeline.
Metadata replaces glue code
Suppose we have data with dimensions trial × channel × time and a stimulus coordinate on each trial. We want to re-reference the channels, turn the time series into band-limited power, normalize pertrial, and decode the stimulus label with a scikit-learn classifier.
By hand, that usually means flattening into samples × features early and tracking stimulus, session, and subject in separate vectors, writing the split logic yourself, remembering which preprocessing steps are safe to run once outside the cross-validation loop, and refitting the learned steps inside each fold without leaking. With XDFlow, thesame workflow looks like this:
from sklearn.linear_model import LogisticRegression
from xdflow.composite import Pipeline
from xdflow.core import DataContainer
from xdflow.cv import KFoldValidator
from xdflow.transforms.cleaning import CARTransform
from xdflow.transforms.spectral import MultiTaperTransform
from xdflow.transforms.normalization import ZScoreTransform
from xdflow.transforms.basic_transforms import FlattenTransform
from xdflow.transforms.sklearn_transform import SKLearnPredictor
container = DataContainer(data) # xarray.DataArray, dims: trial × channel × time
pipeline = Pipeline(
name="decode_stimulus",
steps=[
# re-reference across channels -> trial × channel × time
("reref", CARTransform()),
# multitaper band power -> trial × channel × freq_band
("bandpow", MultiTaperTransform(
fs=500,
freq_ranges={"theta": (4, 8), "beta": (13, 30), "gamma": (40, 80)},
avg_within_freq_bands=True,
avg_over_time_windows=True,
)),
# per-trial normalization -> trial × channel × freq_band
("zscore", ZScoreTransform(per_dim=["trial"])),
# collapse features for sklearn -> trial × feature
("flatten", FlattenTransform(dims=("channel", "freq_band"))),
# decode the stimulus label
("classifier", SKLearnPredictor(
LogisticRegression,
sample_dim="trial",
target_coord="stimulus",
max_iter=500,
)),
],
)
cv = KFoldValidator(
n_splits=5,
shuffle=True,
random_state=0,
stratify_coord="stimulus",
scoring="f1_weighted",
)
cv.set_pipeline(pipeline)
score = cv.cross_validate(container)
print(f"Weighted F1: {score:.3f}")
The pipeline declaration will look familiar to scikit-learn users, but there is a lot more going on under the hood. Re-referencing, band power, and normalization all operate on named dimensions and hand a labeled multidimensional tensor to the next step. MultiTaperTransform turns time into a freq_band dimension, and the steps after it still refer to channel and freq_band by name rather than by integer axis. Only FlattenTransform collapses anything, and it exists solely because scikit-learn (easily used with our SKLearnPredictor wrapper) expects a 2D matrix. ZScoreTransform(per_dim=["trial"]) computes its statistics within each trial, so it stays fold-invariant. The classifier learns from the data, so it is cloned and refit inside each fold, with predictions kept tied to the trials that produced them. The validator stratifies on the stimulus coordinate already attached to the data.
The same contract extends past sequential pipelines. Because a coordinate like subject is part of the data, not a side vector you thread through split logic, a transform can be applied independently per group:
from xdflow.composite import GroupApplyTransform
subject_norm = GroupApplyTransform(
group_coord="subject",
transform_template=ZScoreTransform(per_dim=["trial"]),
)
# drop in as a step: ("subject_norm", subject_norm)
XDFlow discovers the subjects at fit time, fits and applies a separate normalizer to each, and reassembles the trial-aligned output. The same pattern covers sessions, devices, or experimental conditions. In XDFlow, the grouping coordinate is part of the data contract, so the grouping logic can be expressed and handled in a transform rather than as fragile external glue code.
The same philosophy extends to learned feature steps, parallel feature branches, conditional steps, architecture search, Optuna tuning, and MLflow tracking. XDFlow keeps these workflows legible while moving fragile executionlogic like splitting, refitting, caching, alignment, and tuning out of handwritten glue code and into the framework.
Where it fits
XDFlow builds on the parts of the scientificPython stack that already work well. xarray is a natural base for labeled multidimensional arrays. Scikit-learn's estimator interface remains one of the clearest ways to express model fitting. Optuna is useful for hyperparameter and architecture tuning, and MLflow for experiment tracking. XDFlow serves as the analysis-pipeline layer for data whose axes carry meaning while natively integrating each of these tools.
The closest comparison is sklearn.Pipeline. It already handles mechanics like cloning and refitting steps inside cross-validation, but it is built around sample-feature arrays and mostly sequential composition. In practice, using it for structured scientific data means tracking stimulus, session, subject, and other coordinates as parallel vectors outside the pipeline, while writing additional glue for grouped transforms, leakage-safe preprocessing, tuning, caching, and experiment tracking. In XDFlow, all of that lives inside the pipeline contract instead of in glue you write and maintain.
How we use it
XDFlow grew out of our work on electrocorticography-based olfaction models. We use it for decoding odor identity and concentration, for pure odors and mixtures, and for evaluating generalization across sessions and subjects. It underlies all our modeling work, from production pipelines toAI-assisted exploration, and it supports classical machine-learning workflows as well as neural-network ones.
The structural change has been that extension stays local. Instead of modifying a monolithic pipeline when we want to try a new architecture, we write a single transform that fits the XDFlow contract and drop it into a pipeline to be validated and tuned. The core framework stays untouched, and new additions touch only exterior code. Validation is driven by the coordinates already present in the data rather than hand-written per experiment.
The consequence has been fewer bugs, a lower barrier to efficient exploration, and more trust in results we would previously have second-guessed. When a model generalizes across sessions in a surprising way, we can read the array's own record of how it was built and rule out a silent split, alignment, or reshape bug rather than reconstructing the pipeline from memory. The same explicit contracts pay off in AI-assisted exploration, as generated code that operates on the wrong axis or violates a transform's declared interface gets rejected at construction or when the step runs, rather than silently running plausible-looking nonsense.
Why we are sharing it
We built XDFlow for General Sense, but the problem is not specific to us. It appears anywhere scientific data has meaningful dimensions and coordinates. Neuroscience, imaging, sensor arrays, geophysics, climate, and other experimental domains all study richly structured data. Science demands strict validation, benefits from efficient computation, and is undermined by silent errors. What it needs is a way to build reliable, metadata-aware, leakage-safe model pipelines without the burden of heavy and fragile experiment-specific glue code. This is the gap XDFlow is meant to fill.
We are sharing it now because we believe it can help t he broader scientific community.
Try it
XDFlow is open source and MIT-licensed.
pip install xdflow # core
pip install xdflow[all] # with optional extras
- Code: github.com/general-sense/xdflow
- Docs: xdflow.readthedocs.io
- Package: pypi.org/project/xdflow
The API is still evolving, so this is the right time for your feedback. If you work with structured scientific data (neural recordings, biosignals, medical time series, sensor arrays, imaging, climate, geophysics, or any dataset where the axes and coordinates are part of the experiment), try XDFlow and tell us where it breaks, how it can be improved, and how you use it. Issues and pull requests are always welcome.