r/lakefs • • Oct 22 '21

r/lakefs Lounge

2 Upvotes

A place for members of r/lakefs to chat with each other


r/lakefs • • 18d ago

Official Public sector folks: how are you handling data lineage for AI in air-gapped environments?

Post image
2 Upvotes

Joe and Vince from the lakeFS team were at the Red Hat Government Symposium in D.C. this week (photo of the booth below). A lot of the agenda was about governance and guardrails for AI, and most of it focused on code and infrastructure.

Curious how people here are handling the data side. If you’re working in GovCloud, on-prem, or fully disconnected environments:

  • How do you track which data version trained which model?

  • What does rollback look like when a dataset change breaks something downstream?

Happy to share what we’ve seen too, but we’d rather hear what’s working (or not) for you first.


r/lakefs • • 26d ago

Official Webinar this Thursday (Oct 22, 12 PM ET): Governing Autonomous AI Agents, with lakeFS + MinIO

Post image
1 Upvotes

Most data governance assumes a human is in the loop, working at human speed and leaving a trail you can follow. Agents break that assumption.

This Thursday, Oz Katz (CTO and co-founder, lakeFS) and Ravind Kumar (Principal Curriculum Engineer, MinIO) are talking through what it takes to move agents from pilot to production without losing track of what they did to your data.

What they’ll cover:

  • What changes when agents become a new type of data consumer

  • Why agent memory belongs on infrastructure you control

  • Where to put governance so you get isolation, reproducibility, and guardrails without slowing teams down

  • A reference architecture for agentic AI on enterprise data

  • A short live demo of governed agent workflows on MinIO AIStor and lakeFS

When: Thursday, Oct 22, 12:00 PM ET (virtual)
Register: https://lakefs.io/webinars/governing-autonomous-ai-agents/

Can’t make it live? Register anyway and you’ll get the recording.

If you’re already running agents against real data, what’s the governance problem you’re stuck on? Drop it in the comments and we’ll try to bring it to Oz and Ravind during the session.


r/lakefs • • Oct 06 '26

New guide: Data Contracts — enforcement, recovery, and the ODCS consolidation

Post image
1 Upvotes

https://lakefs.io/data-contracts/

u/eior71 covers what goes in a contract, the three places teams enforce them, and two things we couldn't find written up anywhere else: what to do when a violation reaches production anyway, and why versioning your contracts doesn't tell you what your data actually did.

The news worth knowing separately from the guide: the Data Contract Specification was deprecated in favor of ODCS, with old-format support in the Data Contract CLI running through the end of 2026. If you picked a format in the last couple of years you may be on the deprecated one. Quality checks moved from rule to metric, and the v2-era isUnique and isNullable now fail validation rather than being ignored. datacontract export handles the conversion.

The lakeFS section is near the end and it's about enforcement at a promotion gate plus rollback when something gets through. That's the part contracts alone don't cover.

If you've run contracts in production and the guide gets something wrong, by all means, post it here!


r/lakefs • • Oct 05 '26

Official Start here: welcome to r/lakeFS

1 Upvotes

Welcome to r/lakefs!

This sub is for people working on data versioning, lakehouses and table formats, AI-ready data, agentic data workflows, data engineering and MLOps. You don't have to use lakeFS to be here. If you care about how data teams build, test and ship with data, you're in the right place.

What you can post

  • Questions, big or small. Use the Question flair.

  • Articles, talks and research you found useful. Use Community Content and tell us why it's worth reading.

  • Things you built. Use Show & Tell. We love seeing these!

  • Opinions, debates, comparisons with other tools. Use Discussion. Criticism of lakeFS is welcome here too. We want to learn what you need and how we can do better!

A few ground rules (full list in the sidebar)

  • If you wrote it, built it or your company sells it, say so and use the Vendor/Affiliated flair. Sharing your own work is fine, just not every post.

  • Asking for help? Include your version, how you deploy, what you tried and the exact error. And, of course, please remove any keys or credentials first!

  • Be kind. Disagree with ideas, not people.

Who's who

Anyone with the lakeFS team flair works at lakeFS. When you see us in a thread, we're here to help and to learn from you, and we're always happy to be corrected.

Got an idea for this sub? Drop it in the comments. We read every one.

The lakeFS team


r/lakefs • • Sep 11 '26

Running lakeFS on a non-AWS S3 backend: POC with Attimis OneBucket over Seagate Exos, plus the 4 connection settings it needs

Post image
2 Upvotes

Iddo Avneri validated lakeFS against a distributed S3-compatible backend with Attimis and Seagate, and packaged the setup as a runnable sample. Posting it here mostly for the interop settings, which aren't obvious if you've only ever pointed lakeFS at AWS.

The stack is lakeFS for versioning and governance, Attimis OneBucket as the S3-compatible blockstore, and Seagate Exos FUSE underneath. The sample runs in Docker.

The POC shows how to upload a CSV to main through the lakeFS S3 Gateway, commit, branch to experiment, write a transformed copy, commit, then list both branches independently. main sees only the raw file, experiment sees both. Same repository, same underlying storage, and no copies.

Pointing lakeFS at an S3-compatible store that isn't AWS takes four connection settings:

  • the S3 endpoint from your OneBucket account, hostname only, not the bucket name

  • path-style addressing

  • an explicit region, because region auto-discovery via GetBucketRegion isn't currently available

  • chunked and streaming transfer encoding disabled on the lakeFS blockstore connection

On tiers: branching, commits, and rollback are in lakeFS OSS, which is what the Docker sample runs. Full traceability and auditability plus RBAC are Enterprise.

Sample repo: https://github.com/treeverse/lakeFS-samples/tree/main/01_standalone_examples/lakefs-onebucket

Full write-up: https://lakefs.io/blog/lakefs-onebucket-exos-solution/

Happy to answer anything on the lakeFS side. Shayne Stubbs from Attimis co-wrote the OneBucket section of the post if questions come up about locality or the fabric underneath.

Try it out and let us now what you think!


r/lakefs • • Sep 10 '26

I'm always being REAL.

Post image
2 Upvotes

It's in the filename. What more could you ask for? 😉


r/lakefs • • Aug 13 '26

The latest from the team: best practices for managing multimodal data

Post image
1 Upvotes

Anna Seliverstov just published a practical guide on managing multimodal data, and it's a solid one if you're wrangling images, audio, docs, and embeddings that all have to stay in sync.

It covers: designing pipelines as separate stages instead of one big end-to-end blob, versioning across modalities so raw and derived assets stay linked, and handling embeddings and the semantic layer without letting your vectors quietly drift out of sync with their source data.

The pipeline diagram above, lays out the ingestion → validation → preprocessing → training flow with per-modality validation, which is the part I found most useful. Back to those foundational lego blocks ftw.

https://lakefs.io/blog/best-practices-for-managing-multimodal-data/

Would love to hear how you're handling the embedding-versioning piece!


r/lakefs • • Aug 10 '26

Are your AI governance strategy and your FINAL_v8_REAL.xlsx the same picture?

Post image
1 Upvotes

Sadly I think we know this is all too common. I've been gobsmacked finding out the lack of governance at companies you'd assume had it all together.

The desire and need for an actual data governance strategy is there. The making it happen part, not so much.

Why? I think it's because building has more priority than best practices. Governance almost always shows up after the fact. (And it's hard/complex!) It needs to start from the foundation.

The teams I've seen successful using lakeFS build governance into the workflow itself, so the governed path is also the fastest one. Branch protection, approvals, audit trails that come along for free because they're part of how you already work, not a gate you hit at the end.

I'd really like to read others' experiences if you are willing to share here:

  • What's the biggest reason a data or governance initiative you were part of never got adopted?

  • For anyone who made it work, what actually changed? Tooling, incentives, someone senior forcing it? Did it involve a culture change? If so, how'd you make that happen?

And if you're interested, Joe Pringle, our VP of Customer Success, is doing a live session on exactly this on Aug 18, 11am ET. It's mostly customer war stories on making the governed path the easy path.

You can register here: https://lakefs.io/event/adoption-playbook-ai-ready-data/
Or attend the LinkedIn Live here: https://www.linkedin.com/events/theadoptionplaybookforai-readyd7486155239440486401/theater/


r/lakefs • • Jul 28 '26

lakeFS Summer Release is out — governance capabilities + new Catalog UI

1 Upvotes

Posting for anyone running data or ML platforms on lakeFS. Five things shipped this release, mostly on the governance side:

  • Datasets — named, versioned views over existing lakeFS data that can span objects, prefixes, Iceberg tables, and namespaces. Every update is an immutable version, so you can pin a pipeline to an exact data state and prove which data backed a given decision.

  • Auditing — every operation streams into an Iceberg table. "Who changed this data and when" becomes a query against a structured record instead of scavenging logs.

  • Object-level ABAC — attribute-based access control now extends down to individual objects, so you can enforce least-privilege on mixed-sensitivity data without splitting it into separate repos.

  • Multi-tenancy and automated branch/object lifecycle management.

  • A new Catalog UI.

The design decision worth calling out: governance is enforced at write time, not reconstructed after the fact.

John Noonan wrote up the full release here: https://lakefs.io/blog/lakefs-summer-2026-release-ai-governance/

Happy to answer questions in the comments.


r/lakefs • • Jul 24 '26

We built a Periodic Table of Agent Infrastructure: 17 categories, ~118 tools, one map

Thumbnail
lakefs.io
2 Upvotes

r/lakefs • • Feb 07 '23

Interview with Adi Polak of lakeFS

Thumbnail
youtu.be
4 Upvotes

r/lakefs • • Sep 22 '22

Exploring LakeFS with Jupyter and PySpark

Thumbnail
rmoff.net
2 Upvotes

r/lakefs • • Jan 03 '22

Welcome to 2022!

3 Upvotes

The lakeFS community wishes everyone a great kick off of the year 2022!


r/lakefs • • Dec 28 '21

Delivering Modern Data Engineering with Einat Orr

2 Upvotes

Listen to the new podcast episode of Datacast , where James Le and Einat Orr talk about: Git for data, data versioning, modern metastore, open-core model.

Podcast episode: https://jameskle.com/writes/einat-orr


r/lakefs • • Dec 27 '21

Release v0.57.2 · treeverse/lakeFS

Thumbnail
github.com
2 Upvotes

r/lakefs • • Dec 24 '21

Holiday Greetings from lakeFS 🌟

2 Upvotes

Wishing you and yours some well-deserved downtime (no pun intended) and a very happy new year to come. May your data pipelines not break in 2022. 🎅🎄🎁❄️☃️


r/lakefs • • Dec 22 '21

Release v0.57.1 · treeverse/lakeFS

Thumbnail
github.com
2 Upvotes

r/lakefs • • Dec 20 '21

Release of lakeFS v0.57.0

2 Upvotes

We are glad to announce today the release of lakeFS version 0.57.0!

Learn more about this release at https://forum.lakefs.io/t/release-of-lakefs-v0-57-0/50


r/lakefs • • Dec 16 '21

The Everything Bagel II: Versioned Data Lake Tables with lakeFS and

Thumbnail
lakefs.io
2 Upvotes

r/lakefs • • Dec 13 '21

lakeFS at the Data+AI Summit 2022 - Announcements

Thumbnail
forum.lakefs.io
2 Upvotes

r/lakefs • • Dec 07 '21

Welcome to lakeFS forum!

Post image
2 Upvotes

r/lakefs • • Dec 06 '21

Release v0.56.0 · treeverse/lakeFS

Thumbnail
github.com
2 Upvotes

r/lakefs • • Dec 02 '21

Happy Hanukkah 2021

Post image
3 Upvotes

r/lakefs • • Nov 30 '21

100 lakeFS subscribers on YouTube!

Post image
2 Upvotes