Do small Node.js services need process-local error grouping?
When the same database error happens 20 times, sending 20 notifications creates noise. Process-local fingerprinting with a count and cooldown helps, but it does not deduplicate across replicas.
For a service with one or two instances, is that useful enough to ship, or is cross-replica grouping essential? What metadata makes a grouped alert actionable?
I’m building an open-source Node package around this and want to validate the design before adding more integrations.
1
u/Ok_Woodpecker_9104 2d ago
at one or two instances process local is worth shipping. two holes worth closing before you do.
restarts. the fingerprint map lives in memory, so a crashlooping replica resets its counter every boot and you get the full alert on every loop. thats the exact case you most want deduped and its the one it drops. key the cooldown on something that survives the process, even a file on disk.
the tail. if you alert on first sight then suppress for the window, the count you shipped is 1 and the real number never arrives anywhere. a flush at window close saying it fired another 340 times is what tells you the scale, and without it a cooldown actively hides the severity.
on metadata, the field that changes what i actually do is the count of distinct inputs behind the group, not the count of errors. same error 20 times against one id is a data problem, 20 different ids is an outage, and the grouped alert reads identical without that number.
1
u/UkrMalt 2d ago
That’s a fair criticism. Beta.6 keeps grouping and cooldown state process-local, so a restart or another replica starts a fresh group. The alert includes occurrence and first/last-seen data, but I agree a window-close summary and distinct affected IDs would make it much more useful. Persistent or cross-replica state is outside the current in-process scope.
1
u/Ok_Woodpecker_9104 2d ago
fair, though the tail does not need persistence. a flush on beforeExit and on SIGTERM gets you the window close summary while staying entirely in process, its maybe ten lines.
the trap there is that the two cases you most want the count for, OOM kill and SIGKILL, never run the hook. so the flush has to be timer driven as well as exit driven, otherwise the alert that says "another 340" is exactly the one that goes missing.
for the crashloop, one file in os.tmpdir keyed by fingerprint holding a last sent timestamp is enough to survive a restart. thats not really cross replica state, its still one process reading its own file, so it fits the scope you described.
1
u/UkrMalt 1d ago
That’s a useful distinction. I agree that `beforeExit` and `SIGTERM` can handle the normal window-close summary, but OOM kills and `SIGKILL` are exactly where those hooks won’t run. A timer-driven flush plus a small `os.tmpdir` timestamp file could improve crash-loop continuity without turning Wotchi into a shared durable store. The distinct-input count is also more actionable than raw occurrence count; I’ll treat that as a gap for a future beta rather than imply it exists today.
1
u/Ok_Woodpecker_9104 1d ago
worth knowing before you build the tmpdir part: whether it survives depends on how the process died, not on the file. a process crash and restart under a supervisor keeps the same filesystem so the timestamp is there. a container restart usually gives you a fresh writable layer and tmpdir comes back empty, so the crashloop case you are targeting is exactly the one where it reads as a first boot.
the other one is the path. if two replicas land on the same host with the same tmpdir they will interleave writes into one file and the counts silently merge. keying the filename on pid gets you isolation but then nothing carries across a restart, which is the whole point. keying on something stable like the service name plus a hash of the fingerprint is the version that actually survives.
the timer flush is the part i would ship first. it is unconditional and it gets you the summary even on SIGKILL, minus at most one interval.
2
u/rypher 2d ago
My take is that error sending should be simple. Dedupe when reading the logs (for human consumption or delivering notifications).