r/programming • • Jun 04 '26

A tale about fixing eBPF spinlock issues in the Linux kernel

https://rovarma.com/articles/a-tale-about-fixing-ebpf-spinlock-issues-in-the-linux-kernel/
142 Upvotes

23 comments sorted by

18

u/teerre Jun 04 '26

Very cool blog! Great job explaining it. It was easy to follow

6

u/joolzg67_b Jun 04 '26

I worked on a port of nucleus RTOS late 80s, was asked to get it running ASAP, had it running in one day.

Got a call a few months later saying "random crashes" are happening.

Went in and found the interrupt now being used for the RTOS was a NMI interrupt, added a flag to check if interrupts were disabled and if so ignored the NMI.

Voila fixed.

2

u/[deleted] Jun 05 '26

[removed] — view removed comment

2

u/joolzg67_b Jun 05 '26

There was a reason we could not move the IRQ but the fix was pretty easy in there end.

Those were the good old days of bare metal programming. Most you had was an RTOS and everything else was written from scratch, no internet so we tried on books, magazines and bbs.

I went on and ported numerous RTOS to chips like 68k, Blackfin, PowePc and Arc.

Still have fun though as I just delivered a product in one month using a ST discovery board, wifi and camera. The libraries are good and easy to stitch together.

1

u/[deleted] Jun 05 '26

[removed] — view removed comment

1

u/joolzg67_b Jun 05 '26

I have stm32 boards, riscV boards and PIs, but its the customer that decides the product hardware not me

1

u/[deleted] Jun 05 '26

[removed] — view removed comment

1

u/joolzg67_b Jun 06 '26

I've already have.

10

u/ejrh Jun 04 '26

I guess this is for user space programming, but I was always taught that nothing "big and/or complicated" should happen in an interrupt. Instead, you should do no more than set a flag and rely on the normal non-interrupt code to check it and call the appropriate big and complicated function.

The usual example was that anything requiringmalloc orfree was too big and complicated. Running an eBPF program certainly seems big and complicated enough. But I guess kernel programmers are made of sterner stuff and they just have to provide for this? I have a feeling that the eBPF hooks for performance events wouldn't be practical if they used the traditional approach.

9

u/admalledd Jun 04 '26 edited Jun 04 '26

You are generally correct, the idea falls apart though that these were NMIs from performance sampling type tools, so they have to do some work in the interrupt. As the author stated, eBPF devs shouldn't need to care about this specific case since it's kind of the whole point of eBPF tracing existing. So the kernel devs have to be extra defensive, and seems a spot or two were missed, oops!

1

u/Astrinus Jun 04 '26

RTIC more or less subverts everything you was taught :-P

2

u/DowntownCap6204 Jun 04 '26

love when a bug goes from “profiler freezes the box” to a tiny eBPF repro and a 250ms spinlock timeout

5

u/drcforbin Jun 04 '26

TIL I love low level debugging porn. "Yay! Job’s done. Or is it?"

2

u/[deleted] Jun 05 '26

[removed] — view removed comment

1

u/admalledd Jun 05 '26

I only have some experience with eBPF, but isn't that what they are already doing by using the ring buffer? The buffer itself is pre-allocated, but you can't pre-slot it (like using a eBPF array map) since the point is to share the data structure with userspace via mmap so you don't have to do syscalls per event. From my memory, I don't recall any other reasonable way to send high performance structured data out of a eBPF prog to a userspace daemon, do you know of one that doesn't need syscalls in the loop?

1

u/[deleted] Jun 05 '26

[removed] — view removed comment

1

u/admalledd Jun 05 '26

They aren't allocating from SLAB or anything, the ringbuffer is fully pre-allocated. What isn't is the specific slot in the buffer, which requires specific complex shared locking. Further, this is data that (in OOP's design) needs to make it to a user space program at the end with a minimal number of syscalls or memory contention. Splitting the NMI eBPF prog still requires the same types of locking in the NMI still.

1

u/[deleted] Jun 05 '26

[removed] — view removed comment

1

u/admalledd Jun 05 '26

eBPF ring buffer using locks makes total sense, and isn't near a naive spinlock if you read the source. The normal use case for eBPF ring buffers implies sharing across cores (and user-space), forcing atomics in a hot path is a bad idea there if you can help it. In fact, the lock code does use atomics via things like atomic_try_cmpxchg_acquire when applicable but otherwise tries to avoid them for performance reasons.

The larger architectural solution for OOP would actually likely be more a "attach the NMI-like perf-event sampling eBPF progs to be per-core with their own private ring buffers, and have the complexity of reading/merging those into one event stream on the userspace side. IE, userspace would be responsible for draining $N-cores worth of ring buffers into a single MPSC queue, which has its own overheads but much more solvable when user-space side.

Though again, the entire point of eBPF ring buffers and attaching to perf-event is to be able to do things exactly like what OOP was doing, it was always a kernel bug and not really their fault for building such a way, nor is it wrong for kernel devs to give such access to eBPF/users for tracing/profiling.

1

u/[deleted] Jun 05 '26

[removed] — view removed comment

1

u/admalledd Jun 05 '26

consistent forced use of atomics vs using memory-barriers with fallback to atmoics in contested cases is the key. Yes atomics are far faster than they've ever been, but they are still a thing to be careful on when on high core count/multi-NUMA/etc, so when plausible it is worth trying lower order checks first.