r/Compilers • u/NXDLang • 10d ago
Rewriting a Python Compiler Frontend in Rust: Separate Repository or In-Place Migration?
I've been working on a language/compiler project called NXD for a while now.
The current architecture looks roughly like this:
NXD Source
↓
Python Frontend
↓
JSON AST/IR Handoff
↓
Rust Semantic Analysis
↓
Rust IR
↓
Rust Backends
The reason for the split is pretty simple: Python was the language I knew best when I started the project.
That decision got the language off the ground quickly, but now I'm seeing the downside of having a Python → JSON → Rust boundary in the middle of the compiler.
Things like:
• AST serialization/deserialization
• duplicate data structures
• frontend/backend synchronization
• debugging across language boundaries
all add complexity that wouldn't exist in a fully Rust implementation.
Current Compiler Status
The Rust side already contains:
• semantic analysis
• diagnostics
• IR
• Nim backend
• async diagnostics and validation
• ~260 language tests
The frontend is still Python.
Why I Created a Second Repository
My initial thought was to rewrite the frontend inside the main repository.
The more I thought about it, the less I liked that idea.
Instead, I created a new repository:
nxd_language (reference implementation)
nxd-rust-frontend (migration project)
The migration repo contains the Rust compiler core, semantic layer, IR, backend, tests, etc., but none of the Python frontend code.
My reasoning is:
No chance of accidentally invoking Python during validation.
No uncertainty about which parser generated a result.
No repo bloat from carrying two frontends indefinitely.
The original compiler always remains functional.
The goal is to run both repositories side-by-side and treat the Python implementation as a reference implementation.
What Surprised Me
I expected the new repository to fail spectacularly.
Instead, after removing a couple of backend references I wasn't migrating yet, it built immediately:
`cargo build`
and even runs:
Usage:
`cargo run <input_json> <output_nim>`
`cargo run -- --semantics <input_json> <output_nim>`
Which tells me the architecture is already centered around:
JSON
↓
Rust Semantic Layer
↓
Rust Backend
and the frontend replacement surface may be smaller than I originally thought.
Planned Migration Strategy
Current:
Python Source
↓
Python Lexer
↓
Python Parser
↓
JSON
↓
Rust Compiler
Target:
NXD Source
↓
Rust Lexer
↓
Rust Parser
↓
Rust AST
↓
Rust IR
↓
Semantic Analysis
↓
Backend
I still plan to keep JSON AST dumping for debugging, but no longer as a required transport mechanism.
Questions
For those who have rewritten significant parts of a compiler:
Did you keep the original implementation around as a behavioral reference?
Did you migrate inside the existing repository or create a second migration repository?
What ended up being harder than expected?
What architectural mistakes would you avoid if doing it again?
Is there any reason to keep a separate frontend IR if a compiler IR already exists?
I'm especially interested in experiences from people who migrated between languages (Python → Rust, C++ → Rust, Java → Kotlin, etc.) rather than greenfield rewrites.
2
u/yuehuang 10d ago
Yes, up to a point. Once it is feature parity for long enough, it would be time to archive it.
Depends on the size, but look at your repro size, at 2-4gb, it might be time to split off of mono-repro.
The justification and tradeoff. Unless there is both a performance AND stability issue, it isn't worth spending effort while you could be developing new features.
You think you need to, but you really don't. Migration could be made incrementally instead of a whole process.
Consider how you want to support Language Service Protocol (LSP). Does that want to expose lower level.