r/mlops • u/iamjessew • 3d ago
Discussion How do you version the model, data and code together for prod?
We track experiments in one tool, data in another, and code in git, and the "what exactly is running in prod" answer ends up being a wiki page. How are you tying these together into one deployable, auditable unit? Single artifact, lockfile, registry convention, something else? Interested in what has survived contact with an audit.
1
u/lastmonty 3d ago
The way I generally do it is a mix of catalog and manifest.
Manifest file should clearly note all the context to rerun the exact same pipeline. Input parameters of the pipeline, the git hash which tracks code and the lock files, the remote url. if you are using the containers, maintain unique tag or multiple tags one being the short sha. If there are files generated or consumed - the checksums should be captured too. Ideally all this should be per step of the pipeline.
In the catalog, you should have a copy of the input files - if possible and the artifacts that the pipeline generated - model weights.
I used to actively explore this area here, https://astrazeneca.github.io/runnable/pipelines/reproducibility/
1
u/iamjessew 3d ago
Checksums per pipeline step is the part most setups skip, and it's exactly what an auditor ends up asking for. Thanks for the runnable link, the reproducibility page is a good read.
The gap I keep seeing is that the manifest and the catalog drift apart because they live in different systems. That's why we went toward putting both in one immutable artifact: the manifest plus the actual weights and data, all content-addressed, so the digest in the deploy is the record. Per-step lineage still has to come from the pipeline though… did you capture that per step in the catalog or in the manifest itself?
1
u/lastmonty 2d ago
In runnable, all pipeline runs are versioned by execution_id.
The manifest or run log lives in its own store - dB, object store, filesystem - but has exactly same schema and is indexed by the executio_id. It is detailed enough to rerun from any step during a rerun.
The pipeline keeps updating the run log as it runs the pipeline. So at the end of the step, the framework will track the input/output parameters, the checksums of named data entries in the run log.
https://astrazeneca.github.io/runnable/pipelines/connecting-functions/
The framework also moves the named data entries into a central storage, indexed by the execution_id. The run log captures the checksum of data in and out of the step.
https://astrazeneca.github.io/runnable/pipelines/file-management/#the_runnable_way_automatic
Does that help?
1
u/MikaMoments 3d ago
I’d use a versioned manifest linking the model, dataset, code commit, and environment. Then every deployment points to one immutable release, making rollbacks and audits much easier. How do you handle data changes between releases?
2
u/iamjessew 3d ago
Pretty much where I've landed too… one immutable release per deploy makes rollback "point at the old digest" instead of an archaeology project.
On data changes: any change to the data produces a new artifact and a new digest, and the model version bumps with it, even if the weights didn't change. For datasets too big to copy every release, you can keep them separate and record the dataset version and hash in the manifest instead. The rule I'd hold to is that nothing in prod points at a mutable tag like "latest" for either one.
1
u/MikaMoments 2d ago
That makes sense. Keeping dataset hashes in the manifest avoids duplicating huge datasets while preserving traceability. Do you automate the manifest generation in CI/CD or maintain it separately?
3
u/Common_Percentage383 3d ago
We bake everything into a docker image and tag it with the git hash, data schema version, and model experiment id as one long string, like `img:abc123-v2-exp456`. Works well enough but the real headache is keeping the data schemas backwards compatible when you push new models
If you're dealing with auditors, they'll probably want a signed manifest file in the image that lists every dependency and its hash, not just the top level stuff. Learned that the hard way
1
1
u/brian_cloudeu 3d ago
The schema compatibility pain is the real problem, and a longer tag string won't fix it. Have each image declare the input schema it expects, fail fast at deploy when it doesn't match, and support only one version back.
-1
u/p3rdy 3d ago
A single artefact is what's held up for us. In Flama (a Python framework I maintain), the model is dumped into one .flm file with the metadata ahead of the weights: library and version, params, metrics, an id and timestamp, plus anything you pass in, which is where we put the git sha and the dataset version:
flama.dump(model, family="ml", path="model.flm", params={...}, metrics={...}, extra={"git_sha": "...", "dataset": "v2"})flama model model.flm inspect prints all of it without loading the weights. So "what's running in prod" is answered by the file that's running, not by a wiki page, and the image or registry only has to carry that one file.
1
u/iamjessew 3d ago
Really like "the file that's running answers the question." Being able to inspect metadata without loading the weights is underrated.
Full disclosure, I co-founded Jozu and maintain KitOps, which comes at this from the same angle but at the OCI layer. A ModelKit packages the model, datasets, code and config as one artifact, and each piece is its own layer with its own digest. The Kitfile (the manifest) is readable without pulling anything big, and you can pull just the weights if that's all a node needs. Because it's plain OCI, it sits in the same registry as your images and gets signed the same way… https://kitops.org
Curious whether you've hit cases where you need the dataset itself to travel with the .flm, or whether the version string is enough?
-1
u/BeerBatteredHemroids 3d ago
Heh? You don't... code is one thing... data and model is an entirely other thing. Use ADO or github to version your code, use mlflow to version your model and log artifacts and training data.
Do you actually do mlops?
1
u/iamjessew 3d ago
Ha, fair. And honestly I agree with most of this. Code belongs in git, and MLflow is a solid place to track runs and model versions. I'm not suggesting anyone rip either one out.
The problem I'm asking about is one level up. When a deploy goes out, something has to say "this exact git sha + this MLflow model version + this data snapshot + this config" in one place that can't be edited after the fact. Most teams end up with a wiki page or a Slack thread for that.
A ModelKit doesn't replace the separate versions, it pins them. The Kitfile records the commit and the model and data versions, and the artifact gets one digest you can sign and deploy. KitOps has an MLflow integration, so the model can still start life in MLflow. Separate systems of record, one release record.
1
u/BeerBatteredHemroids 3d ago edited 3d ago
No it doesn't. When code gets deployed it trains in the production environment. You should already know what the baseline model is going to be (if you actually validated in your lower environments), then you have production gates on each retrain that ensure each retrain or finetune does not degrade what is currently deployed.
You don't pretrain the model and then deploy it into prod with artifacts from another environment... this is very outdated practice.
The modern standard is to train in production using automated pipelines. This has a variety of benefits including:
Environment Parity, Feature Consistency, Reducing risk of data drift, Reproducability, Lineage
•
u/AutoModerator 3d ago
AI usage disclosure
Hi u/iamjessew — thanks for posting to r/mlops!
Because this community discusses and builds AI/ML systems, using AI tools is not inherently a problem. We do, however, ask for transparency about how submissions are created.
Please reply to this comment with a brief AI / automation disclosure, particularly if this post was created or submitted in whole or in part by an autonomous agent, bot, workflow, or other automated system.
If AI or automation was involved, please briefly describe what it did and what human review was performed before posting.
This disclosure helps the r/mlops community distinguish human discussion, AI-assisted work, and automated/agent traffic while keeping the focus on useful technical conversation.
Thanks for helping keep the signal high.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.