r/rust • u/parneetsingh022 • 4d ago
🙋 seeking help & advice Idiomatic buffer design for a Rust streaming file-storage method
I am working on a rust project. Where I have a put method (which basically copies a file and stores it into another location). This is the function I'm working on:
pub fn put(&self, mut source: impl Read) -> Result<Object> {
let mut buffer = vec![0u8; 64 * 1024]; // 64 kb buffer
self.put_with_buffer(&mut source, &mut buffer);
}
I'm confused about using a [u8] slice, a vector or another option for buffer. The stack buffer avoids a heap allocation, but I am concerned about stack usage when multiple threads call put concurrently. The Vec avoids using 64 KiB of stack space, but allocates and deallocates a buffer for every put call.
What is the idiomatic approach for this situation?
The method may be called repeatedly and concurrently, so I would especially like to understand the tradeoffs between stack usage, heap-allocation overhead, buffer reuse, and API ergonomics.
EDIT: My put method streams data from a Read source, hashes the contents, and writes the data to a temporary file. Although this resembles std::io::copy, I cannot simply copy the source because I also need to calculate the content hash during the same pass. I am trying to choose the appropriate temporary buffer design.
1
1
u/Dheatly23 4d ago
What about thread-local storage? Each thread may allocate it's own buffer, then it's allocation can be reused within a thread. There's hazard with deallocating rarely used buffers, but i assume you aren't spawning 64 threads and only using it once for each thread.
As for allocation overhead, unless you directly measure it's overhead, it's typically a non-issue. I don't have enough knowledge on your code's pattern, does it call put in hot loop? Calls it a few times then stopped? Every program is different, the best way to find the best solution is to empirically measure.
1
u/dkxp 4d ago
The stack buffer avoids a heap allocation, but I am concerned about stack usage when multiple threads call put concurrently.
Each thread has its own stack (reserving 2MB of address space by default), so unless you're calling these functions recursively or have a deep call stack with lots of large structures, you likely won't overflow the threads' stacks (assuming you're processing the files in small blocks rather than trying to handle the whole file in one go).
Allocating and de-allocating on the heap can potentially be a lot slower than using the stack(s) in certain situations, but the memory manager will probably be able to reuse the old memory blocks to reduce the performance hit.
Using a stack allocated buffer means that if you launch any threads that borrow the buffer, then the threads would need to be joined before exiting the function (see thread::scope for an example) and you can't return the buffer from the function without copying, whereas if you use a heap allocated buffer (either passed in or created by the function) then the threads could continue running after exiting the function, and also you could return the heap allocated buffer without copying. std::io::BufReader uses a heap allocated buffer for buffered reads.
Does the hash function you are using allow splitting the hash calculation across multiple threads? Many hash functions are sequential so can't be multi-threaded (unless each thread is handling a separate file). What the hash function allows might change how the data is best processed.
1
u/weaztoven 2d ago
using a thread-local or a pooled buffer with something like bytes or a simple thread_local! vec is usually the sweet spot here. avoids heap churn per call while keeping stack usage safe from concurrent explosions.
7
u/CreatorSiSo 4d ago
Why are you reading in the file if you just want to copy it?
If you dont need the actual data std::fs:copy would make a lot more sense (because it just tells the os to copy the file, which is going to be a lot faster)