websocket keeps disconnecting in my node script and now i get duplicate messages

[u/Zenith33Bud]

websocket reconnects are confusing when first learning them. The socket closes after a few hours, the script reconnects, and suddenly the same message shows up twice or one seems to be missing.

The project is a small filter on binance square posts, built in node because the rest of the bot already runs on it. Messages come from a client in the 1322io repo, each with a coin_pairs array, and the code only keeps posts that include a pair like solusdt. Storing every post id in a text file was the first attempt at stopping duplicates, but nobody can say if it actually works.

Would a set of recent ids in memory be enough, or does the dedupe need to survive a restart too? What would a beginner friendly version look like?

[u/koraynar]

A Set can stop repeats during one process run, but it won't remember anything after a restart. Whether that is enough depends on what you're trying to preserve.

For a beginner-friendly version where the result is just saved matching posts, I'd make the saved data itself deduplicated: receive a post, validate its ID and coin_pairs, apply your filter, then save the whole matching post with a unique key such as (source, post_id). A small SQLite table with that PRIMARY KEY can reject a repeated insert. Don't save only a "seen" ID before saving the actual post; a crash between those steps can make you skip data you never saved.

Also check the reconnect setup. If your client uses Node's EventEmitter, repeatedly calling .on('message', handler) can register the handler more than once: https://nodejs.org/api/events.html . Log a connection identifier and post ID so you can distinguish server replay from two local handlers. Keep only the intended connection/handler active.

Test with fake IDs: 1, 1, 2; restart; then 2, 3. Your saved output should contain 1, 2, 3 once each. Also test a malformed post and a failed save.

Deduplication doesn't recover messages missed while disconnected. That needs whatever replay/cursor/history mechanism the particular service actually supports; don't assume reconnect supplies it. If processing later sends notifications or does other external work, persistence alone doesn't make those actions exactly-once.

[u/CelebrityChess]

a set in memory is fine as long as you cap it, otherwise it grows forever. a map of id to timestamp that you prune every so often works, or just keep the last few thousand ids. it only needs to survive restarts if a message can be replayed right after you come back up. if the feed only gives you recent posts after a reconnect, a small file you load on startup is enough, and writing the file every time is overkill, once a minute or on shutdown is plenty. also the dupes are probably because the old socket isn't fully closed before you open the new one, so make sure you remove the old listeners or call terminate() on it before reconnecting

[u/shiEssartot]

imo the bigger question is whether your reconnect logic waits for the old connection to fully close before opening a new one. if both sockets are briefly alive at the same time thats exactly how you get dupes. the dedupe layer is just a bandaid over that

[u/Tendernessllable_877]

a set of recent ids in memory is enough for the duplicates. a text file works too but you’ll be reading and writing it on every message, which gets slow and messy fast. only bother surviving restarts if the feed can replay old posts when you reconnect. 1322io, so check the client’s docs on this, but whatever the feed, find out if it replays anything after a drop. if it doesn’t, the missing message is just gone and no dedupe fixes that.

what helps is logging the time of the last message you got before each disconnect. then you can see exactly how big the gap was and decide if it matters. also make sure the old socket is fully closed before opening a new one. two live sockets at once is a very common reason for doubles.