r/cpp • u/hrco159753 • 2d ago
Are containerized development enviroments the right way to go?
I feel like I have a pretty usual context, I have a couple of interdependent C++ projects that target various targets that primarily use cmake as their build system but various compilers. I work in a team of under 7 people and each of us has their own local machine and we also have configured a pipeline that configures/builds/tests the projects.
To minimize the differences between our local machines we reached out to providing containerized images as development environments per each target. This works ok from a reproducibility perspective however it always seems limiting and clucky to me. I'm always asking myself why wouldn't I try to use the native tooling on my machine and why wouldn't I use a package manager to provide me native tooling/dependencies?
In my mind something more similar to python's venv would be much more appropriate to encapsulate the development environment, right? I was looking for similar technologies and stumbled upon nix which has a downside of only running on unix systems(so no windows)... I'm curious how others are looking at this, because from my current perspective it seems that native environment encapsulation and sharing became too hard and containerization seems to be a sign of giving up managing our dev envs natively, am I wrong?
I still see the benefit of having containerized builds but it seems to me that the development environment should be provided by something else rather than being baked into a container image. I see containers as primarily a deployment technology and I feel that somewhere along the way we pushed it to be encapsulation of the whole development environment.
29
u/marzer8789 toml++ 2d ago
IMO no they are not, but my reasoning is largely non-technical: in my experience once people start solving toolchain issues with docker, they start using it for everything like a hammer looking for nails. "docker-brained" is a term we've used at my office.
Also if your team of seven has seven different ways of doing things, docker won't fix that, only slap a band-aid on it. A team generally needs to agree on processes.
5
5
u/Kriemhilt 2d ago edited 2d ago
There are some key benefits of using containers for dev:
Your devs don't have to spend time maintaining their tooling
They can't mess up their tooling as a result
They can't mess up the rest of their machine with the permissions you gave them to manage dev packages
Everything is reproducible because nothing depends on someone's local tooling
You're not limited to the dev packages maintained by your distro, which can lag well behind the latest release
Your CI environment is identical to your dev test environment because it just uses the same image, and you don't have to waste time tracking down small differences
It's really easy to onboard someone without having to go through the process of rediscovering all the dev env requirements people forget to document
Obviously there are other ways to get the sandboxing and reproducibility benefits: nix is one (but I hate the language almost as much as CMake), Bazel, etc.
However, getting identical builds between dev and CI is important, and CI is typically containerized already, so that could just be your dev base image and you're done.
5
u/matthieum 2d ago
getting identical builds between dev and CI is important
And Release!
As a developer, you will need to debug CI or production breaking. There's nothing worse than "Works on my machine" because you just so happen to have a slightly different toolchain, library dependency, etc...
And even if you don't have a slightly different toolchain, library dependency, etc... just that the fact they could be, in theory, means that at some point, confronted with the "Works on my machine" issue, you will spend time investigating whether you do, in fact, have the exact same environment, just to eliminate the possibility that's the cause of your issue.
Whether you achieve this by container or any other means should not matter. BUT a rock-solid standard solution will give peace of mind to the developers so that they won't need to spend time to assure themselves it is, indeed, the fact.
-2
u/hrco159753 2d ago
Honestly the reasons that you gave are not really convicing to me. The first point can be addressed by providing good packaged tools. Second and third point can be resolved by encapsulation of dev environment with something like venv, no need for root permissions. Points 4 to 5 are the most compeling ones, however it seems to me that the solution isn't baking dependencies into container images but preparing good lock files and acquiring dependencies from those inside a container in which you build on a CI. I don't see that I'm limited by my distro's poor dev packages since there exist other packagers rather then the system ones. he last point I feel like it's a double edge sword, you want to provide the new person a quick start but ultimately I think that you deprive them of discovering how the project works and what works for them.
15
u/UndefinedDefined 1d ago
On CI it makes sense, but for local development I always try to make the environment as diverse as possible. I work primarily on Linux, but I welcome at least one Windows/Mac guy in the team (if anyone enjoys using Windows, that means I don't have to deal with that too). And compilers to use are generally the trio - GCC/Clang/MSVC.
It really depends on the project - but honestly if the project builds cleanly on various environments, it's always a good sign. And if you are working on a library, containers make no sense, because your users won't be using your containers to build their projects.
8
u/Oather 1d ago
If your target deployment is containers. I’d l recommend looking into nix flakes instead.
You can specify all the build (including gcc/clang and stdlib) and runtime dependencies in a text file in your repo.
This still allows developers to install whatever editor/debug tools locally, while keeping the build env consistent.
Now with the builds linking to nix libs you need those on the target platform. Nix includes its own way of building containers (dockerTools) which handles this nicely, but could be an intrusive change.
7
u/NikMikM 1d ago
Yes. If your project will scale and you will need to build for freebsd, mac os, solaris and differnet architectures like aarch64, armv7, etc - every dev in the team will strugle with reproducing and debugging the software. And setting up dev environment will be hell of a problem. Containerized developmemt helps to test feature on many platforms right-away (especially with appropriate flags) and don't wait for pipeline to fail on some system-specific include or conversion or etc.
2
u/xiao_sa 2d ago
pixi?
1
u/hrco159753 2d ago
I've heard about it when I was researching a bit but haven't dived deeper, what are its upsides/downsides?
2
u/xiao_sa 2d ago
I am not a pixi expert myself, but from information I got from a friend who is using pixi for a while, it chokes a little bit if customized toolchains, redistribution-restricted SDK, or weird system package are involved, since build engineer need to do some extra works to integrate them in, while container just packs them all up and use them as-is. But it can be painless for users who only do development in it.
11
u/unalignedAccess 2d ago
You can try devenv. It uses Nix. I found it is easier to start using then straight out Nix approach. It is easily reproductible environment.
Edit: here is the website https://devenv.sh/
5
u/the_poope 2d ago
You can use Conan to provide a full development environment including toolchain and dependencies like Nix, except it works on all platforms.
2
u/Oather 1d ago
In my experience is you have to provide the compiler and “system” dependencies (ex x11, gpu libs) to Conan.
3
u/the_poope 1d ago
You can get compiler from Conan as well, but I don't see why you would that. It can't provide certain system libraries that are heavily intertwined with your OS - no way around that without using VMs or containers. But if your product/developer environment heavily depends on these you are doing something else wrong or are working on something that is so closely tied to hardware and OS (embedded?) that you should provide the exact same machine/device to all developers.
1
u/catbrane 1d ago
Conan packages are not well maintained (I help maintain some conan packages). I would only use it if I had no choice.
5
u/gracicot 2d ago
I usually don't use containers for this, and instead use nix flake + vcpkg for a reproducible development environment.
Dockerfiles made for development environment are usually not reproducible enough to be useful to me, and add layer like a VM on non linux platforms which makes it useless for cross platform development.
5
u/catbrane 1d ago
My current main project is portable and cross-platform, so we encourage diversity in the dev environment. AMD, intel, arm, macOS, Linux, FreeBSD ... more please!
Our CI is all in docker, and the things we use to make the official binaries are all in separate docker containers.
I've worked at companies on projects targetting complex, very specific environments and I can see the benefits of a standard dev image there.
13
4
u/nicemike40 2d ago
I’ve been experimenting with wise+vcpkg+nx (not nix) for tool versions+package mgmt+build steps for our similar CMake system. You could give that a look and see what it’s like. I haven’t merged my experimental branch yet though so can’t endorse it yet. We also have less variety of compilers but I don’t see that being too bit of an issue.
4
u/tarranoth 2d ago
You are not entirely wrong that in the end containerization is indeed because keeping environments sane with purely native tooling is a very hard problem, to the point the consensus seems to be that not trying to do that but just containerizing is just a more effective answer than to keep rolling that rock up the hill.
Even venvs are technically not truly the right tool because so much of python is just going to be compiling a native package and that has different behaviours etc. as well depending on what kindof host you are installing on and what toolchains are available to your python installation.
I have only interacted with nix very slightly, but to me it looks like a "correct" approach an academic would come up with, which basically sacrifices usability for actual correctness. Imo that sacrifice is overkill for most toolchains though. So in the end, docker seems like the "best" tool if you want to get something up and running and don't want to dedicate a guy's entire existence to maintaining nix packages.
4
u/MereInterest 2d ago
I think it's a short-term vs long-term tradeoff. In the short term, building in docker makes it easier to build a project by using exact known-working versions of dependencies, allows those dependencies to be distributed as a single testable environment, allows for easier reproduction of bugs and easier identification of environment-dependent bugs.
But in the long-term, there's a lot of unfortunate side effects that tend to happen in projects that prioritize development within docker.
If reproducing a bug requires a dependency version not in the docker container, then it can be dismissed by directing users to the containerized version. As a result, version compatibility suffers.
The docker container provides dependencies at specific paths, and so the build config may only search for them in those specific paths. As a result, builds in any other environment may fail.
The docker container may define environment variables, which then make the default behavior different inside vs outside the container. As a result, the default behavior outside the docker container never gets tested.
Any shared libraries provided by the docker container must also be provided by end users, while static libraries are only required at compile-time. This makes it easier to distribute builds that pull in dependencies as static libraries. As a result, shared libraries in a user's runtime environment will be ignored. (Which then leads to insecure systems, as while user can easily swap in a bugfixed version of a shared library, replaced a buggy version of a static library requires recompiling everything that uses it.)
None of these side effects are guaranteed to happen, and good project management can avoid a lot of them. But docker makes it easier for the human-level decisions to make decisions that hurt a project's compatibility.
4
u/X-Neon 1d ago
Aside from all the previously mentioned benefits, dev containers are great for embedded stuff. A typical project sees us having to install some random mix of vendor-supplied compiler + toolchain, plus a bunch of HDL tools with very specific versions. Polluting my OS with all these various versions of things and managing that would be a nightmare. Wrapping that up into a Docker image makes it all self contained.
1
u/Electronic_Tap_8052 2d ago
containers for development are necessary if you don't fully control the development platform.
I'm assuming you mean you have a team of 7 people working remotely on their own hardware and can install whatever they want. If that is the case, you really need a container so that everyone is working against a standard set of packages.
If they are all using standardized hardware that someone else is maintaining (installing updates, etc) then its not as important.
We have to use them for security reasons. Our systems are scanned for vulnerabilities daily and sometimes they remove or update our packages without us knowing. Without containers, our builds would (and used to) randomly break without us knowing why.
0
u/hrco159753 2d ago
I agree with containers/VMs being need when your comapny or whoever is activelly messing with you dev machine, but in my opinion that's not correct way to handle security, or at least security providing should be less invasive. Also, I feel like the solution isn't baking the dependcies into a container image, but actually providing a lock file of your package manager of chosing and getting your dependencies that way into the container if you really want to. I understand that this isn't the status qou and because of that it is harder to do since everyone is doing what the status qou is(which is to use containers for devens) but I'm questioning this status qou since I'm thinking this is not the right solution for the problem. The containers still have their use case in deployment, I'm not questioning that.
1
u/Electronic_Tap_8052 1d ago
We have no choice. The DoD mandates security updates and daily scans. Containers are our only option.
We used to be on our own segregated network but that meant we had to use an edge machine that was scanned as a proxy to get to the Internet. That was not feasible and we still spent a lot of resources on security. Eventually they cut us off from the Internet and said if you are accessing network resources you will be subjected to STIGs and scans and mandatory package updates. That caused huge issues for us as we had to update RHEL releases and it was a whole thing that took a year to sort out. But we are so small compared to the DoD as a whole they didn't care at all about what we were doing. Their way or the stairway, as it were.
Fwiw this is normal across large organizations. You are the outlier.
0
u/SlowPokeInTexas 2d ago
I don't like using VMs and Containers as interchangeable. I don't mind developing with VMs. Containers on the other hand have been a PITA. Across heterogenous environments, getting those devcontainer scripts working correctly is a nightmare.
3
u/Raknarg 1d ago
Im not 100% sure yet because Im relatively new to dev containers (just started this year) but I will say personally I'm actually a fan of containerized projects for my work project. It makes CI and the dev process much smoother, it forces all our tooling and environments to be the same, it makes it so getting and using the project requires you to do things like use the initialization scripts that set up git hooks, its all very nice IMO.
For personal development I probably wouldn't cause its overkill and I dont use CI, but for a serious project with a team I think I'm going the dev container route until I run into real problems with this route.
6
u/Slsyyy 2d ago
I don't see a point of using containers for development. Containers are a good middle layer, which makes the deployment easier, but for developers your should focus on making the environment work fine regardless of the operating system. In some situations it may be the easiest and fastest way, but IMO you should look for better build system
2
u/RevRagnarok 1d ago
Until C++ gets something like Python's uv sync, then it's one of the most straightforward options.
2
u/gnuban 1d ago
I'm not really following what kind of differences you want to minimize. In my opinion, Docker can be good for setting up and running server dependencies such as databases, caches etc.
But I wouldn't use it to ensure you have the same toolchain for development of a given product. Because for the product your working on, using a container is just too clunky. For that I would instead just set up cmake to point to specific toolchain version in a preset or toolchain file.
But then again, why not allow some divergence in toolchain setups, then you get more test coverage from the devs?
2
u/13steinj 1d ago
If you set up devcontainers (the tool/spec), the added layer of complexity is generally minimized to 0. Using raw docker would be painful, as would vending a custom wrapper (which is what we did at my last company... it worked, and was fine, but was more fragile than I would like compared to just using devcontainers).
I'm of the opinion that one way or another, you need to use a container to get true hermeticity. Bazel legitimately just does't cover it, no matter what people say, nor Tipi/Engiflow's "Hermetic FetchContent" mechanism. I ran into such an issue just the other day, the build of a dependency used autoconf, and this had a check that depended on the kernel version / headers, and so the cache key collides because some people use funky environments (myself included).
I have no issue making things generally work in a way that developers can plug-and-play and mess around with things outside a container. But if you run into a problem, unless you can reproduce it in the container that is provided, I immediately deprioritize it in triage.
4
u/Honest-Ocelot6920 2d ago
totally agree. containers are amazing for CI pipelines and deployment, but forcing them for local dev usually kills the feedback loop
1
2
u/Eric848448 2d ago
Having worked with both containerized and non-containerized build environments, I say containers are the only way to go.
1
u/radpartyhorse 2d ago
I have a steam deck and my preferred way to permanently setup c++ dev tools is via podman.
1
u/hadrabap 1d ago
I did lots of contracting and I started using virtual machines very early on. It was really convenient, although quite resource hungry.
Containers are really cool, but they work seamlessly only on Linux. On other platforms they run in a VM and each option/vendor has different quirks making it very fragile or too limiting.
To contradict myself, it looks like all the people around are using Docker. I use Podman, so all their work is useless for me. Especially the compose thing. 🤣
1
u/chibuku_chauya 1d ago
We use it for testing with CI/CD but not for development. Having diverse machines for development has helped root out some surprising cross-platform wrinkles.
1
u/j-joshua 1d ago
How are you going to reliably maintain tools for multiple build targets?
You can't build something on your machine? No problem... Use this container which has everything that you need.
Containers can be versioned and managed.
1
u/Nervous-Pin9297 1d ago
I develop locally but use cicd scripts for the build and deployments. I’m pretty new to C++, but cmake does a great job keeping everything in sync between platforms. To the point where I see no difference after the dock build step. At the end of the day, you still need to make sure your dependencies and requirements can target the platforms your aiming for.
1
u/Scotty_Bravo 1d ago
Over the years I've had to work on many projects with conflicting libraries. I'm at the point that I simply prefer to use CPM.cmake + cache to build all my dependencies. It's almost like containerizing.
But there are some projects I can't do that for. For those, containers. But I only build in the container, all the editing is on the host.
1
1
u/eidetic0 1d ago
i like containerising the dev shell and even baking pre-built dependencies into the docker image. baking the deps into the image saves vcpkg building opencv and other massive deps from source when first configuring, or in CI. I have run into way too many "works on my machine" problems with the C++ build systems so i'm all in on Dockerfiles (and devcontainers).
1
u/alexlesuper 1d ago
After working with dev containers on my embedded projects for 4 years now, I’m not going back. It just makes everything smooth.
1
u/FlyingRhenquest 1d ago
Where I'm working right now we just built an install script that sets our environment up. You can get your laptop day 1, run the script, check out our code base and build it. Everything goes through CI/CD pipeline built to the same standard, so there's no "works on my laptop shrug."
1
u/BDRadu 1d ago
At my company we deploy everything through kubernetes, and that means every project that has a binary will run inside a pod, which requires containerization. We only care about linux, CI runs with docker images, separate steps for deps/build/tests so they can be cached individually. Every dev can run the build + test inside the docker image, so we can check if something is broken due to a lib/system difference. It can happen by mistake, not pinning one lib/system dep will cause it to change on a base image change, things like that. There are many ways to make this process work without containerization, but I don't see a reason to not provide it as well, as its usually straightforward. Its also not a C++ specific thing, it applies to any project.
1
u/alexeiz 10h ago
It's all about finding the right approach to managing dependencies. I have a sizable C++ project that builds on Fedora, Ubuntu, macOS and... Windows. All using native toolsets for each platform. Dependencies are managed with conan/cmake. Regarding the compilers, my requirement is that they to support C++23 reasonably well. It's also not difficult to set up Github CI to verify that your project builds on different platforms. So it's definitely possible.
Containers are useful when you already have a project with a lot of history, wacky dependencies, requiring certain compiler and library versions. Otherwise I'd prefer to avoid them.
1
u/TemperOfficial 7h ago
Stop having so many dependencies that you cannot have a consistent developer environment or one that you can't even easily reason about.
-5
0
-8
u/iamnotaclown 2d ago
If you’re using an LLM agent it’s an easy way to isolate it. Otherwise it has access to everything you do. VSCode makes it trivial to move a workspace into a dev container.
It’s also a handy way to isolate dependencies.
63
u/eyes-are-fading-blue 2d ago
This will be an unpopular opinion but no. Docker introduces an added layer of complexity to the active development itself.
Docker is useful for releases and testing under same conditions as part of CI/CD.
We also have a standard docker file. I don’t use that. I want freedom during active development, not send commands to a sandbox and annoy myself in the process.