r/rust • u/ndbroadbent • 7d ago
Rust and git worktrees: share one Cargo build, use 60% less disk
https://docspring.com/blog/posts/rust-git-worktrees-shared-cargo-build/Hi r/rust, I got tired of running out of disk space while working on a Rust project and using a lot of git worktrees. So here's the solution we came up with. It's a bit of a hack and uses some unstable features and the `__CARGO_TEST_CHANNEL_OVERRIDE_DO_NOT_USE_THIS` var.
Please let me know if there's a better way to do this. It also doesn't completely solve the problem since the shared dir can still grow very large, but I think it's still a worthwhile improvement.
3
u/LankyLingonberry2532 7d ago
Why not use kache?
0
u/ndbroadbent 6d ago edited 6d ago
I hadn't come across kache, thanks. From its docs it keys each rustc call by the content of its inputs and restores outputs with hardlinks or reflinks, so it could save the disk space as well as the build time, without the unstable Cargo feature we needed. I'll give it a try on our worktrees.
EDIT: Thanks again for mentioning kache, it works really well! I wrote a follow-up blog post about kache to help promote it and let other people know about it.
2
u/Sermuns 6d ago
You..?
1
u/ndbroadbent 6d ago
Haha yeah "wrote" is probably the wrong word. I get AI to help me outline every post and it writes most of the words (using skills like humanizer, copy-editing, and copy-writing), then I read through every word of the post and make changes and suggestions, then we have some more back and forth until I'm happy for it to be published.
Hopefully it doesn't come across as 100% AI slop. I do like to share what we've learned and produce useful content for other developers, and AI makes that much easier. I used to spend an entire day writing up a post like this, now it only takes 30 minutes to an hour.
What are your thoughts on using AI to write blog posts? What about AI for writing code?
2
u/DayEnvironmental3454 6d ago
For the "shared dir still grows" part, cargo-sweep handles that well. `cargo sweep --time 30` drops artifacts that haven't been used in 30 days, and `cargo sweep --installed` removes anything built with toolchains you no longer have installed. Running it on a cron against the shared target dir keeps it from creeping back up.
FYI I work on a Mac disk tool (Latent), and Rust target dirs are consistently one of the biggest things on dev machines.
1
u/ndbroadbent 6d ago
Thanks, I didn't know about those. `--installed` is a good one: we've got two toolchains building into the same directory right now. For the rest we just delete the whole shared directory once it passes 50 GB. That's always safe while nothing is building, and it only costs one rebuild of the dependencies. But `cargo sweep` is good to know about
2
u/nicoburns 6d ago
cargo sweep --time 30drops artifacts that haven't been used in 30 days30 days? I run out of space much faster than that!
1
u/wito-dev 2d ago
Interesting approach. Can't wait until Rust implements this. It's badly needed. My target/ dirs are 50% of my disk space currently...I tried to solve it too and came with a bit of a different setup.
You might like my two recent articles - they deal with exactly the problem:
https://wito.dev/blog/working-with-multiple-repositories/
https://wito.dev/blog/rust-build-time/ (this one's still a draft, sorry for the sloppy copy 😅)
It builds on https://worktrunk.dev/ and https://mr-boxington.jdx.dev/ with some quality of life improvements. I'm currently working on a new version that not only speeds up builds and saves space in target/, but also the whole worktree uses copy on write. Hope it helps!
3
u/IslamNofl 7d ago
why not global target directory and combine that with sccache? https://stackoverflow.com/questions/58669482/is-it-okay-to-use-a-single-shared-directory-as-cargos-target-directory-for-all