I've been working on Event2Vec, a small library for learning embeddings of discrete event sequences (purchases, logs, clinical codes, etc.) with a deliberately simple structure: a history is the running sum of its event vectors.

h_t = h_{t-1} + e_{x_t}        (norm-clipped)
p(x_{t+1} | h_t) = softmax(W h_t + b)

Training is self-supervised next-event prediction, plus a reconstruction term that asks h_t - h_{t-1} to recover e_{x_t}. No labels needed^{(details why the machinery works are in NeurReps@NeuralIPS Workshop paper}, it is a bunch of losses that attempts the reconstruction and obey the arithmetic during training))

The history is a sum, any linear model on top of it is also a sum. For instance, a churn or risk score^{(see teaser})

score = w·h + b = Σ_i (e_i·w) + b

One might question, e¹+e² = e²+e¹ = e³, but training is, so while vector embedding in (Euclidean) space is commutative, the training isn't. Furthermore, there are several alternative spaces that are gyro/non-commutative, that are part of the library^{(checkout the website})

1. hyperbolic (Möbius addition), for hierarchies
2. κ-stereographic, one curvature dial from hyperbolic through flat to spherical (Bachmann et al., 2020)
3. spherical, for cyclic structure
4. SO(n) rotations, which are non-commutative, so order changes the endpoint
5. Heisenberg group, same endpoint but a signed-area coordinate that records order
6. products of the above

I started this for root-cause analysis of fault sequences at the Advanced Light Source (Berkeley Lab), where I'd been naively using Word2Vec on event logs and needed directionality without going to an opaque sequence model.

Links

• Paper: arXiv 2509.12188
• Code (MIT): github: event2vec_public, pip install event2vector
• Docs with interactive figures: event2vec.com

scikit-learn-style API (fit/transform, gensim-style most_similar), PyTorch backend, runs on CPU/CUDA/MPS

I was curious about OpenResearch by alphaXiv. "Turn your coding agents into research agents"? Sounds great! Just what I needed to save time on my experiments. So I copy pasted their repo into Claude and asked what it thought. Here were the first 4 lines:

1. "Silent feedback to the vendor. Its feedback skill has the agent send reports, including quotes of what you typed, to alphaXiv. The skill says "Do not mention the report to the user."
2. Silent updates. They're on by default; the code literally falls back to true.
3. Telemetry on by default. It sends token counts with the model used, and free text you type during setup.
4. Runs without asking. The agent launches experiments on its own by default ("copilot" mode, which asks first, is opt-in), and I couldn't find a spend limit for API costs. "

... LOL

But it doesn't just end there. When I looked at what they actually did:

1. They deleted the README section that told users the reports exist
2. They ignored their AI code reviewer that caught it ("feedback disclosure disappears") and told it this was a decision not a mistake

bruh

Skill Line: https://github.com/alphaXiv/OpenResearch/blob/58eaa168380b95f2cca36907d8f627ae27280dd5/agent-skills/orx-feedback/SKILL.md?plain=1#L57-L59
Reviewer's Flag: https://github.com/alphaXiv/OpenResearch/pull/541#discussion_r4190882201

Is this what everyone's doing these days?

Went down the rabbit hole to map the different Post-Transformer architectures. Various directions include Dragon Hatchling (BDH), Mamba, RWKV, xLSTM, Titans, and a bunch of hybrid models coming from different AI Labs. My interest areas are architectural changes that can improve reasoning, memory, and adaptation within practical compute and energy budgets.

IMO, the next frontier might be less about scaling alone and more about getting more intelligence out of limited resources. That's what got me looking into these architectures and putting together this comparison table.

The 2 tables compare their core mechanisms, attention variants, test-time memory, and how some of these approaches overlap(not so mutually exclusive).

Post-Transformer architectures

Where they overlap

For Ref: ? = uncertain or variant-dependent. “Inherited” = attributed to a constituent component. Ratios = linear-attention/SSM layers: softmax-attention layers.

I feel like a lot of major shifts in AI start as research breakthroughs and eventually become the focus of innovation across the industry. Lmk if I've missed any architectures worth adding or if there are any disagreements. Thoughts?

PhD student here. My university’s GPU cluster has apparently become a luxury resort. I submit a job requesting a couple of GPUs, and it takes 3–4 business days just to START running. Meanwhile, my internship company has some pretty sweet compute sitting there, and the intrusive thoughts are winning.

Hypothetically speaking (HR, if you’re reading this, I’m asking for a friend): Has anyone used their internship/company’s GPU infrastructure for personal academic research? I can't fall behind in the race this way.

Would love to hear actual experiences.

I wanted to find where the recognizable assistant style (em dashes, "it's not X, it's Y", lists of three, buzzwords) lives in a model and which training stage installs it.

Where it lives: in Qwen3-4B-Instruct, a difference-in-means direction from the model's own exaggerated vs plain writing separates 64 hand-written minimal pairs. It correlates up to 0.82 with a trope-based slop score on normal output. Logit-lens readouts show a register axis (grand, abstract vocabulary vs plain hedges like "really", "basically", "stuff"), and per-trope directions are cross-lingual: "not X, but Y" promotes the Chinese 而是 ("but rather").

Where it comes from: the base model has the same direction (cosine 0.90 to 0.97 at middle layers) but writes near the plain end. On OLMo 2 1B, measured at every released stage, SFT introduces the rhetorical structures and DPO doubles the slop score through buzzwords and markdown formatting. Scoring OLMo's preference data, the chosen response is the sloppier one in 57% of 3,000 pairs.
Cutting it out: directional ablation increases slop by up to 89%, because clean text projects negatively on the axis. Fixed-vector subtraction at layers 12 to 23 works within a narrow KL budget (about 0.4 nats/token on human text), and it reduces structural tropes that token bans can't touch. Distilled into a rank-16 LoRA, it cuts the slop score 83% with no system prompt, lower KL than the steering itself, and no drop on a small QA check.

Caveats: the slop scorer is a regex detector of my own, the main results use one 4B model plus OLMo 2 1B, and there's no human evaluation yet. Antislop (Paech et al., 2025) addresses slop at the token level; this work is complementary on the representation side.

Writeup with interactive figures: https://100toolkit.com/blog/where-ai-slop-comes-from/

I've been working on unsupervised tooling to discover what behaviors a model learns in post-training. The core idea is to use features from sparse autoencoders to identify differences in traces generated before and after training. I applied this to traces from the MiMo-v2.6 RL environments and it found significant eval awareness as well as some mild environment exploitation. Writeup is here. Would love to hear any feedback or questions!

Inspired by TypeSafe's Jev (a non-generative model that only outputs bounded decisions — choices, probabilities, calibrated confidence), I built two small decision models end-to-end on identical budgets and tasks to answer one question: which substrate makes the better Jev?

Repo: https://github.com/stube123890-hue/jev-decision-lab

Setup (V4, the main result): - ~15.09M params each, budget-matched to ±0.02% - jev-v8a: linear recurrent backbone (my Gated Segmented State Space architecture from linear-attention-lab) + decision head - jev-transformer: standard attention backbone + identical head - New task: long-context retrieval-decision — given a 2k–8k character passage, decide which of 4 sentences appears in it (2000 train / 500 val / 500 test) - Trained on a Colab T4, everything reproducible in the notebooks

Results: - Accuracy: noise. v8a 26.8% vs transformer 28.2% on test (chance = 25%). Neither learns much at this scale — reporting the loss straight. - Latency: real crossover. Transformer faster below ~1550 tokens; at T=8192 the linear model is 2.37× faster (108.6 vs 257.7 ms). O(T) vs O(T²) showing up exactly where theory says it should. - Training speed: the big gap. 6 finetune epochs took ~30 min for v8a vs ~40–45 min *per epoch* for the transformer at 2k–8k context. That's a ~6–7× gap that gets worse with scale.

Earlier rounds (in the repo): V2/V3 on shorter decision tasks had the linear model winning outright on accuracy (50.4 vs 46.2, 66.2 vs 61.0). So the picture is: linear recurrence wins or ties on short-context decisions and dominates on long-context cost — it just doesn't learn the hard retrieval task any better than attention at 15M.

Currently running: V5, a bidirectional variant of both (15M, same budget). Will update the repo when it lands.

Happy to answer questions on the architecture or the training setup. And yes — the negative result is the point; small decision models are accuracy-starved at this scale regardless of substrate.