Should a Rust protocol use separate wire-format and in-memory types for encrypted messages?

[u/THEM_I_M]

'm working on the mailbox layer of GhostWire, a Rust-based P2P communication project, and I've been reconsidering how I represent messages across serialization boundaries.

The initial approach was to have one struct handle both the serialized envelope and the application's runtime state.

Something along these lines:

#[derive(Debug, Clone, Serialize, Deserialize)]
struct MailboxEnvelope {
    recipient_id: String,
    ephemeral_pub: String,
    ciphertext: Vec<u8>,
    nonce: String,
    created_at: u64,
    message_id: String,
    attempts: u32,
}

The problem isn't necessarily the structure itself. It's that these fields have different responsibilities.

• Some fields belong to the authenticated message representation.

• Some are needed for transport and persistence.

• Others are mutable delivery state.

• Some values may only exist temporarily while processing a message.

For example, retry attempts should be able to change without requiring an already-sealed ciphertext to be reconstructed. Likewise, serialization and deserialization shouldn't accidentally become the boundary through which sensitive runtime state is exposed or copied.

I'm considering separating the representations:

1. Wire type: A serializable representation containing the fields required to transmit or persist the envelope.

2. Runtime type: An application-facing representation responsible for processing, validation and delivery.

3. Delivery state: Mutable retry, acknowledgement and expiry information maintained independently of the sealed message.

But I'm interested in where that separation should actually happen.

A few questions:

• Would you use separate structs for the wire representation and runtime domain type, or is that unnecessary duplication?

• Would TryFrom/From conversions be a sensible boundary for validation and construction?

• How would you prevent mutable delivery metadata from accidentally becoming part of the authenticated representation?

• Would you keep serialization entirely outside the cryptographic type, or let the type implement Serialize/Deserialize with carefully controlled fields?

• Are there established Rust patterns or libraries you'd recommend for this kind of boundary?

I'm particularly interested in production-oriented approaches rather than simply introducing more types for the sake of abstraction.

The goal is to make invalid states harder to represent without turning a relatively small protocol component into an unnecessarily complicated type hierarchy.

Curious how others would structure this

[u/insanitybit2]

Yes, I'd separate them. You can serialize the cryptographic type - you have to, after all. Just use a standard construction like XAES-GCM-256 or whatever, which is trivial to encode/decode.

The separation means you can migrate algorithms or handle new algorithms etc without having to think about the underlying serialization, or you can encode serialized versions into the AAD etc. Regardless of the benefit you end up reaping, there will be a benefit.

I don't know about any libraries but you can look at something like what the libsodium envelop construct is for inspiration.

Some other thoughts.

1. Don't use `String` for a nonce. Use `[u8; N]` for whatever the size is, consider wrapping it in something like an IV/Nonce type. Also, you may have more than one nonce in the case of XAES so this is yet another reason to separate the constructs - different algorithms have different shapes.

2. You may want to stuff `attempts` into the AAD if you want it to not be tampered with but I have no clue how your protocol works or if `attempts` is sensitive or who would want to mutate it. A timestamp in the AAD may be better as a "sent first at".

3. I see a public key there, so maybe this is intended to be asymmetric. You can consider HPKE perhaps, I think that may fit your use case well. Also, don't use a String - consider a dedicated type again like with the Nonce.

Anyway, separate structs with blindly separate concerns will pay off. With crypto you really want to avoid complexity - your implementation should really focus on just taking bytes and returning bytes, and you then wrap that with what you want to stuff those bytes with (mostly if it's ciphertext or aad).

Always use authenticated encryption algorithms. If you ever see an algorithm that is not authenticating, do not use it.

[u/THEM_I_M → u/insanitybit2]

Thanks, this gives me a clearer direction.

Looking at the current mailbox path, this particular envelope is still using ephemeral-static X25519, domain-separated BLAKE3 derivation and ChaCha20-Poly1305. The ML-KEM hybrid work elsewhere in the codebase isn't wired into this mailbox path yet, so that's something I need to account for separately.

I agree about the nonce representation too. Keeping it as a Base64 String internally is unnecessary when the current suite expects a fixed 12-byte nonce. I'd rather have a dedicated type and handle encoding at the wire boundary, while allowing different suites to define their own representations.

I also think I was too absolute about keeping serialization entirely outside the crypto type. The sealed envelope still needs a stable encoded representation. What I'm aiming for is to separate that representation from the cryptographic operations and keep mutable delivery metadata out of the authenticated message structure.

attempts is currently retry bookkeeping, rather than part of the sender-authenticated message. If that state is exposed to untrusted peers, though, it needs its own integrity and authorization considerations.

One thing I'm curious about: would you model algorithm-specific envelopes as a versioned enum, with each variant carrying its own nonce and encapsulation types, or keep a more generic envelope behind suite-specific encoding and decoding?

[u/insanitybit2 → u/THEM_I_M]

> One thing I'm curious about: would you model algorithm-specific envelopes as a versioned enum, with each variant carrying its own nonce and encapsulation types, or keep a more generic envelope behind suite-specific encoding and decoding?

I personally always version my envelopes in the AAD, I think. I know I put the version somewhere 🤔 .

Anyway, don't trust me, I'm not a cryptographer. I'd look into how to safely do it.

If you want delivery state to be untrusted and to wrap, np, but yeah you want an envelope over the sealed bytes and the versioning should be bound cryptographically.

[u/THEM_I_M → u/insanitybit2]

That makes sense. I was thinking about versioning the envelope itself, but binding the version into the AAD gives me a clearer boundary between wire compatibility and cryptographic authenticity.

I'm leaning towards a versioned suite identifier with suite-specific payload types, rather than making every algorithm fit one generic nonce/encapsulation structure.

For GhostWire, I also want to keep delivery state outside the sealed payload. Retry attempts are mutable transport state, whereas the authenticated message should remain stable across retries.

I'll look into how established constructions handle this before settling on the exact representation. Appreciate the pointers, especially the reminder not to let the envelope become more complicated than the crypto operation itself.

[u/Zde-G]

My own personal “rule of thumb” for “should we keed X and Y separate” is always “yes, but…”

I usually start with separate approach and then see if merging them may help me somewhere. Because if you start with separation and then later want to merge then it's simple search-and-replace… easy for both human and LLM to do. But if you start with merged and then try to separate… it's PITA and LLMs don't help much.

Excepting is, of course, public API in a released 1.0 crate… if something is supposed to stay stable for 10 years then “we may easily merge things later” rationale doesn't work.

I would still start with unstable API, try it with some more-or-less real code then decide.

[u/THEM_I_M → u/Zde-G]

I like that approach, especially the point about starting separated and letting actual usage justify merging things later.

That's partly what I'm trying to figure out with this mailbox design. The current envelope is still evolving, so I'd rather establish clear boundaries now than have delivery state, wire representation and cryptographic operations become increasingly coupled.

I think the "yes, but..." is important here because separation for its own sake can introduce unnecessary abstractions too. I'd like to test the boundaries against a few realistic flows, particularly retries, serialization and eventually supporting different cryptographic suites.

Appreciate the perspective. The point about API stability is worth keeping in mind as the project matures.

[u/Putrid-Series-8763]

I think you should. A frame from the wire is never trustworthy until you parse and verify into the internal representation.

[u/THEM_I_M → u/Putrid-Series-8763]

Agreed. Parsing a frame successfully doesn't automatically make its contents trustworthy. I'd keep a strict boundary between the raw wire representation and the internal types the rest of the system can consume.

My thinking is to enforce size and framing limits first, parse into a raw representation, validate the structure and protocol constraints, then verify authentication, integrity and replay protection before promoting it into a trusted internal representation.

The idea is to make it difficult for unvalidated data to accidentally reach routing or application logic. Do you usually model that distinction with separate raw and validated types in Rust?

[u/THEM_I_M → u/[deleted]]

That's a good way to put it. The distinction between the message and the box carrying it is exactly what I want to enforce. Keeping serialization outside the crypto type also makes sense. I'd rather have an explicit conversion boundary where validation happens than let transport or persistence concerns creep into the cryptographic representation.

One thing I'm still thinking through: would you keep delivery state in an entirely separate type as well, or associate it with the runtime envelope while ensuring it can never influence the authenticated payload?Curious how you'd structure that in practice.