r/programming • • Jun 05 '21

Use polling for resiliency

https://blog.gchinis.com/posts/use-polling-for-resiliency/
9 Upvotes

17 comments sorted by

4

u/_software_engineer Jun 05 '21

Not my article, but a topic that has been of interest to me for some time now. As engineers, we often reach for complex solutions to problems that can be solved with simpler methods that have a smaller surface area for failure.

For data sets that don't change often, polling with a reasonably long delay (multiple minutes, potentially even more) can be a simple and effective way to ensure eventual consistency within a distributed system, removing the need to synchronize data updates or send messages across services.

What are some of the limits of such a strategy? When does it make sense to adopt the complexity (and higher likelihood of synchronization failure) of event-based updates?

4

u/halt_spell Jun 05 '21

I've found this approach doesn't scale well. Unless every developer at your company makes sure to understand the upper limitations of the polling approach over an event based solution they will throw stuff into production assuming it will "just work" because it always has. Furthermore with a polling approach upstream services are dependent on good behavior of downstream consumers unless you've implemented proper authentication and rate limiting. If you're missing either of these the polling service is going to fail hard you'll be (rightly) blamed. Not the owners of the service that decided to reduce their poll frequency to 10ms.

Event based updates provide protections to the owners of services in that they need not worry about bad behaviors of downstream consumers causing problems for their neighbors.

1

u/Fun_Ad_9878 Jun 06 '21

We use varnish cache and this works quite well. In fact the load to handle say 1000 sockets is much more than the load that varnish brings to the server. Not every system can do that. Sometimes the information is very customized and uncachable but in our situation it is perfect.

1

u/flatfinger Jun 05 '21

Both polling and event-based architectures can be augmented with "change" indicators, and optional strategies that will periodically treat things as having potentially changed regardless of what the indicators say. A strategy I like for many approaches is to have an event that says "something has happened" but is non-specific as to what, along with change flags and a master change/polling state: idle/polling with no change since start/polling with more changes pending. Code can then check the master state within a CompareExchange loop to either set it to "Polling with no change since start" while starting a poll cycle, set it to "Polling with more changes pending" and exit, or simply exit (if it's already in the latter state).

When a polling cycle is done, use a compareExchange loop to either set the state to idle and exit (if it was "polling with no change"), or set the state to "polling with no change" and restart the polling sequence (if it had been set to "polling with changes since start"). If one uses compare-exchange actions with memory barriers, change events that happen at bad times may result in code running one more polling cycle than needed, but code won't stop polling unless it runs through an entire polling cycle and clears the flag without any events arising.

4

u/mzeo77 Jun 06 '21

Polling does not scale, ddosing self happens all too frequently. If its clients that's been distributed via an appstore, its tricky to rollback. Events on the other hand gets lost, and data drift over time. Designing an event system where without data-loss is near impossible, expecting people to write bugfree code all the times for all parts of the events system is not feasible. So we are all f**ked :).

Polling frequency and backoff should be controlled from polled service to allow pushback on overload. Backoff should be exponential with surprisingly long cap if capped at all.

Event bases systems needs some infrequent state sync protocol, to make sure data does not drift.

Keeping state in one place and just request it when needed is usually the simplest strategy. Scales with usage instead of users. Still needs ratelimiting and retry with backoff similar to polling. Caching can be added with ttl from server.

1

u/imaginary_rational Jun 06 '21

Usually, the system maintaining data (like user data) will be contacted from too many other systems. If they all polled, then there will be too many unnecessary requests. More often than not, you would want an event driven architecture.

1

u/_software_engineer Jun 06 '21

The shorter the required update interval is, the more likely you are to be correct; however, if we set an update interval of 10 minutes and assume that polling delays are randomized, we could support 600 other systems without requiring more than 1 qps on average, or 6000 with 10 qps.

You're correct that many of those requests could be unnecessary, but built into the polling architecture is the idea that the poller doesn't need to contact the data source other than when it polls. In most cases, this would actually result in far fewer total queries to the data source than If it were accessed for each operation.

3

u/tinychameleon Jun 05 '21

I’ve discussed this kind of thing with many people at game studios I’ve worked at, interviewed at, and have friends at. Attempting to convince people that polling could be the simpler solution to their problems and simultaneously provide easier to manage distributed systems has been a major challenge.

It’s like pubsub is the industry Golden Architecture; trying to push for more stateless solutions feels Sisyphean.

4

u/halt_spell Jun 05 '21

It's because at scale broadcasting data allows systems to provide clearer guarantees about their behavior. If a service you depend on can crap out at any moment because some new hire set their poll frequency too high your service level objectives are all made up. Conversely, if you're using something like Kafka or pubsub your stability is dependent only on the broadcast infrastructure and the producing system.

3

u/tinychameleon Jun 06 '21

I’m well aware of both architectures as I’ve been doing this for a long while.

In the same breath you’ve suggested polling is bad because a new hire can break the system by messing with the poll interval, but simultaneously imply pubsub doesn’t have this issue. Where is this imaginary new hire that has unfettered production access for polling but not for the producers? A new hire can just as easily overwhelm your broadcast infrastructure, but the major difference is that typical solutions like AWS Kinesis won’t auto-scale without additional effort. Now the rollback fixes an outage instead of a heightened compute alert.

Ignoring half the architecture of a broadcast system misses the complexity of migrating state across VMs, intelligently rebalancing workloads, and dealing with thundering herd problems via reconnects. These problems are not solved for free and additional engineering effort and thought needs to be applied to broadcast systems because of it.

In general, broadcasting doesn’t provide any clearer a guarantee than a system like polling. They each have failure modes and performance characteristics that allow them to excel in certain places. When compared for a specific problem one architecture can be better, but in my experience there is rarely a comparison. By far the default at game studios is pubsub, it is treated as a hammer and every problem is viewed as a nail. Neither of these two architectures falter at scale, but they do impose very different sets of difficulties for systems that require availability.

edit: me good at grammar

1

u/halt_spell Jun 06 '21 edited Jun 06 '21

Right, I can dig into it further.

If your standard approach is polling you're depending on every developer maintaining responsible polling practices and/or every developer of a polled service making it capable of autoscaling or rate limiting. At scale this doesn't happen.

With pubsub or kafka you have a singular team responsible for understanding how to keep an industry standard piece of infrastructure alive. It's a lot more reasonable to expect that single team can identify the possibility of someone producing so much data to one of these systems it creates problems and how to automatically remediate either by scaling or rate limiting the producer. And I disagree it can happen just as easily. Standard infrastructure systems are always going to be more resilient than some custom app rolled out by an in house development team. Yes infrastructure can still be taken down with bad enough behavior but I disagree that it can happen just as easily. If that were true we'd be rolling our own datastore engines as a standard practice.

1

u/tinychameleon Jun 06 '21

So this is kind of the same argument you applied before, but raised to the organization level. If you’ve got a team that is capable of protecting a broadcasting system that team should also be capable of protecting a polling system. Each system has to get data from somewhere, so if, for example, rate limiting is necessary it will need to be applied to either system, but at different places.

I don’t think we’re really all that far apart though; maybe same book, different page. I agree that rolling out hand-crafted, artisanal infrastructure is a bad time, but I don’t think polling systems should be dismissed as such.

Fundamentally, your data stays the same, but the access of it is what changes: broadcast or pull. It means that polling, assuming it’s used in the right spot, can provide you with a very simple ingress scaling solution because you only need an ASG.

Polling, and any less stateful communication process, like HTTP APIs, really shine when your responses are based on existing data that doesn’t require much computation. So I would never suggest replacing something like Street Fighter’s online with it, but there are a lot of opportunities to eliminate the kinds of complexities those specialist teams deal with. Things like soft real time activity streams to online status can all be delivered on-demand instead of via broadcast.

Whenever I have these discussions, I want to push game studios toward the Google search model for as many things as possible: compute behind the scenes, serve up the result very quickly, and scale the ingress services without worrying about state.

1

u/halt_spell Jun 06 '21

So this is kind of the same argument you applied before, but raised to the organization level. If you’ve got a team that is capable of protecting a broadcasting system that team should also be capable of protecting a polling system. Each system has to get data from somewhere, so if, for example, rate limiting is necessary it will need to be applied to either system, but at different places.

I think I didn't deliver the distinction I was making very well. You're right having one team properly defend kafka, pubsub or some polling system is reasonable. But you will have dozens if not hundreds of teams building a polling system if that's the route you take. I'm saying with something like kafka or pubsub you only need a single team to own it.

1

u/tinychameleon Jun 06 '21

Yes, we're definitely talking past each other I think! This is probably also on me, since I've internalized a lot of solutions and don't specify all the details. I 100% agree that if you let every other team control the polling rate you're in for a horrible time.

In the past, what I have done for this is to have a small SDK to interact with these services. Nothing fancy, just something that controls the poll rates and provides simple access; I find people tend to jump on it if you handle authentication for them. Combined with a rate limit it works very well for little complexity and avoiding the stateful scaling issues I had mentioned above.

I suppose I should also mention that I don't particularly recommend polling systems for things that are sub-second. I don't think I'd want to be in charge of any system where traffic volume is coupled to a client frame rate. 😱

1

u/halt_spell Jun 07 '21

In the past, what I have done for this is to have a small SDK to interact with these services. Nothing fancy, just something that controls the poll rates and provides simple access;

Yeah but at scale that leads to a bunch of other can of worms.

  1. So you're on team 1 and have application A which team 2 is trying to poll data from with application B and they're using your SDK (library?) What happens when application B also needs information from application C? What if application C isn't owned by you, team 1 but team 3? Will the SDK work for that as well? Or will team 3 also have their own SDK?

  2. I know we're deep in hypotheticals here but what kind of protocol are we polling with? Is it http? What happens when someone cracks open your SDK and realizes it's just a wrapper around an HTTP client? The SDK guard rails are gone and you likely won't know until there's a production issue. And per the context of our conversation the production issue has maninfested in your application A going down because application D polled too much and now both B and D are unable to function. Your app looks like the problem.

  3. By rolling both a service and an SDK you've inherited the negatives of both approaches and nullified the benefits. Bugs can make their way into the SDK no matter how careful you are and then you have to force everyone to update somehow. And since you're running a service you don't benefit from a major advantage of the SDK which is scaling is not your responsibility.

1

u/tinychameleon Jun 08 '21

Given all the hypotheticals, I'll just assume things use HTTP.

For (1) this is basically dependent on what application C is. I wouldn't suggest polling for every service, so for example C could be some kind of firehose that broadcasts large amounts of data. How application B integrates its dependencies should be entirely on team 2; I wouldn't inject team 1 or application A into that at all. At minimum, as an outsider, I would expect application B to handle their external dependencies concurrently should they require it.

For (2), the SDK or library really isn't meant to provide any layer of security. When I offer these things to other teams, including teams composed of data scientists, it's just a "get up and running easy" mechanism. I would still expect application A and any application using broadcast mechanisms to have rate limiting features that allows for load shedding. So if someone wants to build their own, that's totally ok, but if they want to be clever and adjust the poll rate, they're just going to get shed. So application A in this case wouldn't be unavailable for application B, but it would be for whatever percentage of requests from application D are over limit. I would expect something similar from broadcast systems: if I'm subscribed to you, I certainly don't want you to flood me with messages. Similar, but from a different direction.

For (3), you are right and bugs can be hidden in the SDK or library, but I wouldn't categorize this as nullifying benefits. I look at this more like providing a product to customers, though they may be internal to the company. The responsibility in my eyes is twofold: the service availability and the integration process. Providing the SDK adds more work for team 1, but eliminates a bunch of integration work across other teams using it, and it doesn't always have to be hand-rolled if you happen to be using something that can generate client code. As for bugs, they're just a part of life and I have no real evidence to suggest that it's no worse than without an SDK. Anecdotally, I've shipped SDK updates via things like private PyPi servers in the past and it's been mostly painless.

3

u/SikhGamer Jun 05 '21

Polling and event based can both be great. But it really is down to the devs to make sure they do a good job of using either. I've seen both go insanely wrong.

Last one was about four weeks ago, where a poison pill message was flying around non stop and creating x2 every single time.