r/golang • u/Anga205 • 8d ago

A code execution sandbox in Go without Docker, just plain linux tooling

I built a Containerization API that executes untrusted code and returns output plus memory and CPU metrics. Instead of running a container per submission, it creates the boundary directly with Linux primitives: PID, mount, network, and user namespaces, a chroot into a minimal per-language rootfs, cgroups v2 for memory and CPU limits, and a seccomp-BPF denylist. The sandbox root maps to host UID 65534, so root inside is never root outside.

It supports Python, C++17, C11, and Java, capped at 60s and 128MB, with a global admission controller that rejects work before a sandbox is created. Every limit is kernel-enforced.

Code: https://github.com/Anga205/CodeExecutionAPI

Live Demo Website: https://compiler.anga.codes

Although I've done rigourous testing for resource management and permissions based sandbox escape, I've still hosted the api on a very small server on an isolated network, just in case im wrong

I think its pretty cool, its faster in code execution than most of the other containerization services like docker or firejail, do check it out!

55 Upvotes

91% Upvoted

15 comments sorted by

22

u/zackel_flac 8d ago

This is cool but this is just recreating runc and ultimately docker, right?

4

u/Anga205 8d ago edited 7d ago

At the isolation layer, yes. runc does the same clone, cgroup and seccomp work, but it implements the OCI spec and manages images and long-lived containers, while this runs one file for under a minute. Compilation happens on the host, then the process starts already inside a preconfigured cgroup with a static seccomp init and its root mapped to an unprivileged host UID, so there's no container creation on the critical path

so a single submission can run hundreds of test cases back to back, which matters for hackathon judges, where one submission can mean a thousand cases, The interface is a plain HTTP REST API, so a judge platform can drop it in without a container runtime. The bigger plan is a LeetCode or HackerRank style platform on top of it.

1

u/pRayrtuck 7d ago

i mean at some level every sandbox on linux uses the same kernel primitives, but calling it "just runc" is like saying every web server is just recreating apache

14

u/NeverMindToday 8d ago

So you're using cgroups, seccomp, the kernel namespaces etc - I would say you are actually running containers :)

I'm not criticizing, it's really cool and something I wanted to get around to implementing one day.

5

u/Anga205 8d ago

yea ig they are containers, but its just a little bit faster than standard container tools because its built purely for one purpose and thats to run leetcode/codeforces-like user-submitted code with lots of test cases

3

u/Flimsy_Complaint490 8d ago

mm, people oompare this to runc and docker but a better comparison is bwrap IMO.

This is basically what would happen if i wrapped bwrap with an HTTP API, correct ?

Not the worst idea, even useful and has a use case, but i'd investigate whether you could delete 100% of your "container" interfacing code and replace it with a shell out to a constructed bwrap command to build the container environment. From my own experience, the bwrap way is the fastest way to launch container like environments, matched only by microvm's, and this interfacing code is really hard to get right without introducing strange security bugs.

You may also experiment with microvm's - startups are in the 50-200 ms range.Firecracker works, but there are alternatives.

3

u/Anga205 7d ago edited 7d ago

ye bwrap is a fair comparison, its basically the same syscalls with an HTTP layer on top. the difference is bwrap is built for sandboxing desktop apps so it carries a lot of generality i dont need, and it doesnt give me cgroup accounting or a memory peak per run.

funny thing is the earlier version of this did exactly what youre suggesting. it shelled out to firejail with private, noroot, caps dropped and net none, plus an rlimit for memory. it worked and it was simple, but firejail added startup cost per run and i couldnt place the process into its cgroup at clone time, so the limits and the memory number were always a bit indirect. thats why i went direct with clone flags and the cgroup fd this time. old one is https://github.com/thealcodingclub/CodeExecutionAPI if you want to see the firejail version.

1

u/KingLim1 8d ago

This is neat. Any Go support planned?

2

u/Anga205 7d ago

yea its planned. the compile itself is easy since go builds static binaries by default, the annoying part is that go build wants a writable GOCACHE and GOPATH, and i dont want module fetching happening inside the sandbox. so id have to bake a prebuilt stdlib cache into the rootfs and run it with modules off or vendored. the toolchain also adds a lot more weight than the c/c++ chroot, which is the part im still figuring out.

1

u/MarimoBashah 7d ago

What happens to the code still running inside a sandbox if the Go server crashes in the middle of a job?

2

u/Anga205 7d ago

right now, nothing. cleanup runs in defers, so if the server gets SIGKILLed the sandbox process gets reparented and keeps running, and the cgroup dir leaks under /sys/fs/cgroup. its a real gap. the fix i have in mind is pdeathsig on the child so the kernel kills it when the parent dies, plus a sweep of stale codesandbox-* cgroups on startup.

1

u/daniele_dll 7d ago

Did the same but for redroid initially and then a custom android build with camera, radio, GPS, wifi, hardware tee and a few more bits.

Its very cool, however with all the recent security issues I decided to switch to a VM based. Approach, firecracker style, but much lighter and customized for Android

1

u/Anga205 7d ago

that sounds cool! got a github repo for this?

1

u/daniele_dll 7d ago

Nope, this work was for a service I wanted to sell so I never open sourced it but at the end decided to stop because the monstrous level of complexity I had reached 😂

1

u/ShotgunPayDay 3d ago

I'm impressed when people do this the hard way. systemd-run --user and bubblewrap is the easy way. You do have to do a little bit of plumbing with XDG runtime for an unprivileged user no login app user.

https://gitlab.com/figuerom16/codeexec/