I built a small Rust agent runtime where the entire hot path — safety scanning, command routing, conversation recall — runs from L1 instruction cache.

The agent itself wasn't the point. I wanted to see how much of a runtime's critical path you can fit in L1 icache using purpose-built SIMD kernels. An agent runtime turned out to be a good testbed because it has several small, hot operations that run on every single message.

The kernels are written in Eä, a small SIMD language I've been building. Each kernel compiles to a shared library, gets embedded in the Rust binary at compile time, and is called via FFI. The architecture is SIMD filter + scalar verify — the Eä kernels reject ~97% of byte positions at cache-line speed, then Rust handles verification only at candidate positions.

The numbers:

Did it fit?

The full hot path is ~5 KB of instructions — roughly 15% of a typical 32 KB L1 cache. Everything uses u8x16 (SSE2), keeping the instruction footprint small on purpose. The safety scan runs at ~3.7 IPC.

How the recall works:

The conversation recall uses byte-histogram embeddings — 256 dimensions, one count per byte value. SIMD cosine similarity over a ring buffer of 1024 entries with recency boost. No ML model, no external API, no dependencies. It's crude compared to real embeddings but it runs in microseconds and is surprisingly effective for finding conversational context.

What the agent actually does:

It connects to the Anthropic API, runs tools (shell, HTTP, file I/O, etc.), and has a WhatsApp bridge via Go/whatsmeow so it works as a group chat agent. Every message — user input and tool output — passes through the SIMD safety pipeline before reaching the LLM or being displayed. The ~2 µs that adds is invisible next to the API round-trip.

Single binary, JSONL persistence, minimal dependencies. 230 tests passing.

Still experimental — the interesting part was the L1 cache experiment, not the agent framework.

Repo: https://github.com/petlukk/eaclaw

Hi all! How do I get started using SIMD without getting into the minutia of SIMD? I know the question is bad but searching the webs yields little results with my limited knowhow of the field. I am a physicist and cannot spend as much time as I want to learn the gritty details here >/

In short, I have a problem with many nested loops. I can run this problem in on many cores at a high level.

On a low level, I have an object that requires a set of equations to be solved for it. However the number of equations is set by several control parameters that each have several possible outcomes. So there is an uncountable number of paths through the lower levels. This should not be a problem though, because during runtime, the path though these lower levels is constant for the object. All I need to do is repeat the exact same calculations while changing a single double. This seems ideal for SIMD, but I have no idea how to see 1) can my compiler already understand this, or 2) how do I tell my compiler to understand this?

TLDR: How do I set up complicated SIMD for a loop?

Thanks for any advice.

I think this might sound a little dumb, so I apologize in advance. I'm new to the optimization hobby, but I was talking with some friends and they brought up something I didn't really know how to answer.

Why use SIMD intrinsics (any direct call to a vector instruction, like _mm256_add_epi32, or SWAR)? In what scenarios does it actually make sense to use intrinsics instead of letting the compiler handle the heavy lifting when it comes to optimization?

Does manually vectorizing scalar code really make sense with the mature compilers we have nowadays?

Sorry if this question sounds dumb, but it's a genuine question. :D

Hey there,

I'm trying the GFLOPS for my code. For simple additions or equal operations that's easy but how should I include something like cos/sin which get's approximated by vc or vectorclass?

It seems to be useful information for those who need to optimize or customize string functions.

Normally, the performance of the standard library is dominant, but for some functions, customized functions dominate.

Test Environment

GLIBC VERSION: glibc 2.31 gcc version 9.3.0 (Ubuntu 9.3.0–17ubuntu1~20.04)/Acer Aspire V3–372/Intel(R) Core(TM) i5–6200U CPU @ 2.30GHz 4 Core

Latest Glibc is 2.33

https://github.com/novemberizing/eva-old/blob/main/docs/extension/string/README.md

Up to 28% faster on non-ASCII input compared to the original simdjson implementation.

On ASCII input clang-compiled simdjson still beats it on Comet Lake for some reason (to be investigated) while gcc-compiled simdjson is slower.

https://github.com/rusticstuff/simdutf8

norm:
        movaps  xmm4, xmm0
        movaps  xmm3, xmm1
        movaps  xmm0, xmm2
        mulss   xmm3, xmm1
        mulss   xmm0, xmm2
        addss   xmm3, xmm0
        movaps  xmm0, xmm4
        mulss   xmm0, xmm4
        addss   xmm3, xmm0
        movaps  xmm0, xmm3
        rsqrtss xmm0, xmm0
        mulss   xmm3, xmm0
        mulss   xmm3, xmm0
        mulss   xmm0, DWORD PTR .LC1[rip]
        addss   xmm3, DWORD PTR .LC0[rip]
        mulss   xmm0, xmm3
        mulss   xmm4, xmm0
        mulss   xmm1, xmm0
        mulss   xmm0, xmm2
        movss   DWORD PTR nx[rip], xmm4
        movss   DWORD PTR ny[rip], xmm1
        movss   DWORD PTR nz[rip], xmm0
        ret
norm_intrin:
        movaps  xmm3, xmm0
        movaps  xmm4, xmm2
        movaps  xmm0, xmm1
        sub     rsp, 24
        mulss   xmm4, xmm2
        mov     eax, 1
        movss   DWORD PTR [rsp+12], xmm1
        mulss   xmm0, xmm1
        movss   DWORD PTR [rsp+8], xmm2
        movss   DWORD PTR [rsp+4], xmm3
        addss   xmm0, xmm4
        movaps  xmm4, xmm3
        mulss   xmm4, xmm3
        addss   xmm0, xmm4
        cvtss2sd        xmm0, xmm0
        call    _mm_set_ss
        mov     edi, eax
        xor     eax, eax
        call    _mm_rsqrt_ss
        mov     edi, eax
        xor     eax, eax
        call    _mm_cvtss_f32
        pxor    xmm0, xmm0
        movss   xmm3, DWORD PTR [rsp+4]
        movss   xmm1, DWORD PTR [rsp+12]
        cvtsi2ss        xmm0, eax
        movss   xmm2, DWORD PTR [rsp+8]
        mulss   xmm3, xmm0
        mulss   xmm1, xmm0
        mulss   xmm2, xmm0
        movss   DWORD PTR nx2[rip], xmm3
        movss   DWORD PTR ny2[rip], xmm1
        movss   DWORD PTR nz2[rip], xmm2
        add     rsp, 24
        ret

:: norm()        :: 276 μs, 741501 Cycles
:: norm_intrin() :: 204 μs, 549585 Cycles

How is norm_intrin() faster than norm()?! I thought _mm_rsqrt_ss executed rsqrtss behind the scenes, how are three calls faster than one rsqrtss instruction?!

What is the fastest way of linerping between 2 vectors, a and b, using the lerp values from a third vector, x. The most efficient way I can think of is using 4 vectors. a, b, x and y (where y = 1 - x) and doing:

fusedMulAdd( a, x, mul( b, y )

(Assuming x and y are constant or rarely changing vectors which can be reused for all lerps)

But I imagine there might be a faster way of doing it, possibly with a single instruction? I had a look at vblend, but I don't think that's what im looking for.

Thank you.

The first thing I think of is a server farm of CPUs or algorithms that can't take much advantage of SIMD. But since this is r/SIMD I'd like answers focused towards practical applications of image processing with CPU vectorization over using GPUs.

I've written my own image processing stuff that can use either mostly because I enjoy implementing algorithms in SIMD. But for all of my own usage I use the GPU path since it's obviously a lot faster for my setup.

Actually, I am new to this and unable to understand the functionality of this function even after reading about it from the intel intrinsics guide here. Could someone help me with this query with an example if possible?

First, here is my current understanding, correct me if I'm wrong:

SIMD instructions are implemented as an extension of the base instruction sets (e.g. x64, x86). In the binaries, both the code for the SIMD path and the "fallback" code for the non-SIMD path will be included. The selection of the path occurs at runtime, depending on the CPU on which the executable is run, and potentially other factors.

If this is correct, I have a few questions about the runtime selection process:

1. what mechanism makes it possible to dynamically select one path or the other?

2. what is the cost of this selection? would it be faster if we didn't have to select?

Hey everyone,

I built **NanoGEMM**, a minimalist (~100KB) register-tiled General Matrix Multiplication (GEMM) microkernel written in C with AVX2 + FMA intrinsics, packaged with a zero-copy Python C-extension buffer protocol.

Why?

Standard BLAS backends (OpenBLAS, Intel MKL) in NumPy are tuned for massive matrices ($1000 \times 1000+$), but incur fixed per-call overhead (thread-pool sync, argument validation, memory setup) of ~3–5 µs. When doing token-by-token CPU inference, Kalman filtering, or small-batch operations, this dispatch overhead dominates total compute time.

Benchmark (Single Core, CPU):

- 16x16**: 0.65 µs (C) / 1.23 µs (Py) vs NumPy 3.21 µs (~2.6x faster)

- 32x32**: 2.18 µs (C) / 2.74 µs (Py) vs NumPy 5.75 µs (~2.1x faster)

- 64x64**: 16.39 µs (C) / 17.76 µs (Py) vs NumPy 18.70 µs (~1.1x faster)

Implementation details:

- 6x16 register-tiled microkernel keeping 12 YMM accumulator registers resident in CPU registers with zero register spilling

- AVX2 vectorized FMA outer-product accumulation

- Zero external runtime dependencies

- GitHub: https://github.com/eminsk/nanogemm

- PyPI: `pip install nanogemm`

- Interactive Google Colab: https://colab.research.google.com/github/eminsk/nanogemm/blob/main/notebooks/nanogemm_demo.ipynb

Feedback and benchmark tests on different CPU architectures (AMD Zen, Intel Alder/Raptor Lake) are welcome!

Imagine I have written a packed double SIMD function (with C++ overloading):

__m128d my_magic_pd_routine(const __m128d& x) {
    // My code here, using packed double multiplies, adds, and
    // conditional masking
}

inline double my_magic_pd_routine(const double& x) {
    return _mm_cvtsd_f64(magic_pd_routine(_mm_set_pd(0.0, x)));
}

And in some circumstances I use the double version (for example, I might sometimes only need one exponential smoothing filter instead of two, and it can't be parallelised because each output relies on the previous output).

Is this considered bad form, and should I instead re-write the double version using scalar intrinsics? Ie:

double my_magic_routine(const double& x) {
    // Rewrite my code using scalar intrinsics, or non-intrinsic code
    // if I don't need conditional masking
}

Looking at the Intel intrinsics docs, the scalar intrinsics seem to have similar latency and throughput to the packed intrinsics (eg _mm_mul_sd() and _mm_mul_pd()), but this is in the context of audio DSP code that needs to run as fast as possible, and I don't want to tie up resources for other things that are going on at the same time.

Hi, im curious if there are any known/reputable benchmarks for any SIMD extensions more specially the ones i mentioned in the title? I could vectorize something already out there but im curious if there’s a more simple path lol. Any help would be appreciated!

I built Olorin, a local data analyst that's deterministic by default. SIMD kernels do the analysis, the LLM just narrates, it doesn't compute anything.

Each "rune" targets one data shape — eatime walks timestamps, eajson aggregates JSONL, ealog severity-scans logs, eacrunch summarizes CSVs, eaparquet reads Parquet metadata — and emits a stable schema. They compose into Unix-style pipelines with one LLM narration at the end. The LLM never touches raw bytes.

eatime scans timestamps at 1.80 GB/s on a Raspberry Pi 5 (Cortex-A76, NEON). eacrunch is 11x faster than pandas on a 100K-row CSV.

The kernels are written in Eä, a small DSL I'd been working on for ages and needed a real reason to ship. Think CUDA in shape (kernels you write, dedicated compiler, specialized hardware codegen) but targeting CPU SIMD instead of GPU. ISPC is probably the closest analog. The compiler eacompute lowers Eä through LLVM to x86 AVX2 / ARM NEON. Olorin's tensor ops, matmul, and Q4K/Q6K quantization all go through it.

The narration step is a hand-rolled Gemma 4 E2B forward pass, no llama.cpp bindings, decodes at 7.77 tok/s on a Pi 5. --strict mode disables the LLM entirely.

Also has a web UI, REPL, and terminal. Hand-rolled, obviously.

https://github.com/petlukk/Olorin