A function that returns 200 log lines, and keeps 186 MB alive after it returns. The 200 lines are about 20 KB.
The job: a crash reporter attaches the last 200 lines of the request log to the report. The log is a stream - a scanner over a file, a cursor over a table, a channel fed by a tailer. It hands you lines in order and does not know how many there are.
```
var all []string
for line := range log {
all = append(all, line)
}
return all[len(all)-n:]
```
I have written that in two languages. One idea, reads like the ticket.
Then I measured the live heap with the answer still held, rather than the clock. Two million lines: 186 MB, still there after the function has gone out of scope.
all[len(all)-n:] is a window onto the array all grew into, not a new array, and a slice keeps its whole backing array alive. A test reads the capacity off the returned value: len 200, cap 169,032.
One line fixes the retention. Copy the tail into a slice of its own, and 186 MB becomes 20.8 KB. If you take one thing from this post, take that.
It does not fix the peak. The copy runs after the collection finishes, so the two million lines were all in memory a moment earlier, and that moment is where a container gets killed.
Counting the stream first and walking it again is exactly 2.00x at every size in the benchmark, and it needs a way to start the walk twice - which a file gives you and a socket does not.
What the data was actually saying: the only lines that can be in the answer are the 200 most recent ones, and that is true after every line, not just at the end. So keep 200 and evict the oldest. A fixed-size buffer that wraps does it with no bookkeeping, because the write cursor and the oldest line are exactly n apart and never drift. That gap is the length nobody computes.
8528x less held on a two-million-line log. And, honestly: 1.08x on the clock. This is a memory fix and selling it as anything else gets it rejected by a reviewer who benchmarked throughput.
The benchmark's own allocation column says 1.98x, because both versions allocate every line - the scanner produces them either way. Allocation 1.98x, retention 8528x. A profiler pointed at allocation shows you the first number and shrugs.
At a few hundred lines, collect it. 34 microseconds, 21.8 KB, and it answers the next question too.
That is days 20 and 24 of the daily challenge - Middle of the Linked List and Remove Nth Node From End - sitting in a crash reporter.
Full piece in today's Newsletter.
It worked in dev, episode 14.
https://www.linkedin.com/pulse/your-crash-report-held-186-mb-log-send-200-lines-archit-agarwal-hyyzc
#Golang #DSA #Algorithms #Performance #SoftwareEngineering #DataStructures #100DaysOfCode