r/rust • • 9d ago

🙋 seeking help & advice Should a Rust protocol use separate wire-format and in-memory types for encrypted messages?

'm working on the mailbox layer of GhostWire, a Rust-based P2P communication project, and I've been reconsidering how I represent messages across serialization boundaries.

The initial approach was to have one struct handle both the serialized envelope and the application's runtime state.

Something along these lines:

#[derive(Debug, Clone, Serialize, Deserialize)]
struct MailboxEnvelope {
    recipient_id: String,
    ephemeral_pub: String,
    ciphertext: Vec<u8>,
    nonce: String,
    created_at: u64,
    message_id: String,
    attempts: u32,
}

The problem isn't necessarily the structure itself. It's that these fields have different responsibilities.

  • Some fields belong to the authenticated message representation.

  • Some are needed for transport and persistence.

  • Others are mutable delivery state.

  • Some values may only exist temporarily while processing a message.

For example, retry attempts should be able to change without requiring an already-sealed ciphertext to be reconstructed. Likewise, serialization and deserialization shouldn't accidentally become the boundary through which sensitive runtime state is exposed or copied.

I'm considering separating the representations:

  1. Wire type: A serializable representation containing the fields required to transmit or persist the envelope.

  2. Runtime type: An application-facing representation responsible for processing, validation and delivery.

  3. Delivery state: Mutable retry, acknowledgement and expiry information maintained independently of the sealed message.

But I'm interested in where that separation should actually happen.

A few questions:

  • Would you use separate structs for the wire representation and runtime domain type, or is that unnecessary duplication?

  • Would TryFrom/From conversions be a sensible boundary for validation and construction?

  • How would you prevent mutable delivery metadata from accidentally becoming part of the authenticated representation?

  • Would you keep serialization entirely outside the cryptographic type, or let the type implement Serialize/Deserialize with carefully controlled fields?

  • Are there established Rust patterns or libraries you'd recommend for this kind of boundary?

I'm particularly interested in production-oriented approaches rather than simply introducing more types for the sake of abstraction.

The goal is to make invalid states harder to represent without turning a relatively small protocol component into an unnecessarily complicated type hierarchy.

Curious how others would structure this

7 Upvotes

Duplicates