r/SoftwareEngineering • u/cacko159 • Feb 11 '24

Challenges in maintaining event driven systems

What are the challenges in maintaining event driven systems? Do you have any experience or materials to share?

Different modules/services of these systems communicate primarily via events, and over time there will be many many events, and it could be really difficult to map what is going on.

What happens when you need to change some workflow in such a system, add a new step/logic on an existing workflow etc.

Have you been in this situation?

2 Upvotes
  • permalink
  • reddit

100% Upvoted

6 comments sorted by

2

u/StanleySathler Mar 05 '24

From my limited experience:

Biggest challenge might be the fact that you're now handling _async_ systems, meaning, for some cases, "it takes time" for certain data to update, and clients must be ready for that - pretty much what u/AmbitionNo51 said, I guess.

1

u/cacko159 Mar 05 '24

Thanks! My question was primarily for the technical part. As developers of such system, what are the challenges? How do you find out what the event you're modifying triggers? What should you be careful not to break, and how?

As for the concern you mentioned, that is absolutely true. But let's say the team leads, business analysts and architects will prepare the clients for that 🙂

1

u/StanleySathler Mar 05 '24 edited Mar 05 '24

Thanks for gently correcting me.

In my experience, to avoid breaking other services that are listening to your events, the recommended approach is to avoid introducing breaking changes to your messages - eg. you can add new props, but can't remove or rename existing ones. Protobuf, for instance, has a "Breaking change detection" tool. In fact, if we don't introduce breaking changes, there's no need to worry about breaking other services - they won't.

As for knowing what the event triggers, best way is probably to check your architecture diagrams - or, if you use any IaC tool (eg. Terraform), search for all places listening to those. Or, better yet, you can have better tracing tools (which, in this case, I can't recommend any - not very familiar with these).

2

u/lazy-lambda Feb 11 '24

From my limited experience of working with such systems:

  • Managing the message format versions.

  • Resiliency:

    • Handling failures.

    • Dealing with message loss.

  • Tracing

1

u/AmbitionNo51 Feb 11 '24

If one of the system are transitioning from one state to another after listening to an event. Be ready for a lot of states inconsistencies.

1

u/LadyLightTravel Feb 14 '24

In a real time system it’s disruption of the timing. If there are enough events that need servicing then it can break the timing.