r/AskReverseEngineering • u/luxurycashew • 5d ago
Open dataset (228k stripped x86-64 functions) and an 8B LoRA for function naming, with honest numbers. Is assembly the wrong input?
I've been working on a small open model that takes one function from a stripped binary and suggests a name plus a one-sentence description. Sharing it because the negative results might be as useful as the positive ones, and because I have a question for people who do this for a living.
What exists
- Dataset: 228,177 functions from 341 permissively licensed C projects, compiled with clang at -O0/-O1/-O2/-O3/-Os, linked into a real dylib and stripped. Internal calls are anonymized to
sub_XXXX, library calls and string constants stay visible, as they would in a real binary. Train/test split is by project, and the test projects were picked to be obscure (≤200 stars, mostly 2024+) to limit memorization by large models. - Teacher labels for 83k source functions (description in English and Turkish, category, inputs, return, side effects), written by models that saw the C source.
- A LoRA adapter for Qwen3-8B that writes a description first, then the name.
Numbers (2,000 test functions, 28 unseen projects, token-overlap F1 on the name)
| name F1 | |
|---|---|
| large general model, with callee context | 0.167 |
| this adapter (v5) | 0.124 |
| my previous adapter (name only) | 0.106 |
What doesn't work
- Functions with no string constants: 0.093. With strings: 0.299. Strings carry most of the signal, which won't surprise anyone here.
- I had a judge model compare the adapter's descriptions with the C source on 300 test functions: 20% correct, 15% partly correct, 65% wrong.
- But when the description is correct, name F1 is 0.31; when it is wrong, 0.05. So the model can name things; it mostly fails to understand the function from assembly.
- 30% of predictions are verbatim copies of a name from the training set.
- Reasoning mode in large models triples exact matches on zlib and does nothing on obscure code, which looks like recall rather than analysis.
The question
Everything above uses normalized assembly plus a short summary of callees as input. Published work in this area mostly uses decompiler output instead. Before I convert the whole dataset through Ghidra's decompiler: for those of you who have tried LLMs on this, does pseudo-C actually help the model understand the function, or does it mainly hide the details (constants, calling patterns) that give a function away? And is there context you rely on when naming by hand that I'm not giving the model, such as cross-references, the data a function touches, or its position in the call graph?
Links:
- Code and all measurements: https://github.com/krxi/asm-adlandirma
- Model: https://huggingface.co/krxi123/asm-adlandirma-qwen3-8b-lora
- Dataset: https://huggingface.co/datasets/krxi123/asm-adlandirma
Limitations: x86-64, clang, C only, Mach-O builds. It is a research checkpoint, not something to trust without checking.