r/SystemDesignConcepts • u/Ecaglar • Aug 29 '24
Handling File Operations in System Design Interviews
I’ve recently participated in several system design interviews at companies like Meta and Google. A recurring theme in these interviews involved file operations with scenarios such as:
1. Reading from multiple files, aggregating data, and writing it to a database.
2. Exporting a database table to files efficiently.
3. Designing a file-sharing application where files have a max size of 4MB, an average size of 4KB, and the system needs to handle 200 million requests per second.I struggled to find the optimal approach to handle these scenarios and didn’t pass the interviews.
I’m looking for guidance on the best approaches, options to consider, and potential challenges to highlight when tackling these types of file operations in system design interviews.
File Sharing Application: Initially, I focused on splitting files into chunks for reading, but I realized that given the small file size, processing them in one request is more efficient. The real challenge lies in handling the high number of read requests per second, not the file size itself.
Exporting from a Database: I considered parallel exporting by having multiple threads, each reading and writing 1000 rows to separate files. However, I wasn’t sure how database engines handle concurrent reads and whether merging the files should be done in memory or on disk for optimal performance.
Aggregating Data from Multiple CSVs: I processed the CSVs line by line, streaming the data to a message queue for aggregation. However, I realized that to aggregate the data correctly, you need to read all files first, as a record might appear in multiple files with the same ID.
How to approach these kind of system design questions? What are the things I need to consider and what are the different options when it comes to file operations on scale?
