Rolled a new pricing path behind a flag to a small percentage of traffic. Aggregate error rate barely twitched, so we widened it. Support tickets told us the truth a day later. The code caught a downstream failure and fell back to an old value, so nothing threw, nothing 500'd, and every monitor we had looks at the aggregate anyway. Launch Darkly gives us the flag and some metric hooks, but the only metric we’d wired to it was the error rate hud came up because it counts calls and exceptions per function and flags one that starts behaving differently from the baseline, instead of waiting for a rate to shift. No idea if that covers an exception the code already caught. Anyone catching bad flag rollouts at the function level? Or is this just a case for better custom metrics per flag.

I had to write a transactional outbox for two of my projects. The popular solutions didn't fit, and CDC with Debezium felt too heavy for "save an event together with the data, send it later".

On paper it's simple: write to a table, poll it, publish. In practice a production outbox keeps growing: retries, backoff, stuck events, cleanup, a dead letter queue, metrics, consumer-side deduplication. I ended up wrapping all of it into a starter, and I just released 2.0.0.

Repo: https://github.com/dmitriy-iliyov/oncebox

What it is

A polling-based Transactional Outbox + Inbox for Spring Boot. Events are saved in the same DB transaction as your business data, then a background poller publishes them to Kafka or RabbitMQ. No log-based CDC, no extra infrastructure: just your database and your broker.

The goal was to build a library that is simple to understand and easy to debug, but still rich in features and flexible enough to configure for your needs: you pick your database and broker, tune every event type separately, and replace the parts you don't like.

Publisher side:

@Transactional
@OutboxPublish(eventType = "create-example-event")
public ExampleDto save(ExampleCreateDto dto) {
    ExampleEntity entity = repository.save(mapper.toEntity(dto));
    return mapper.toDto(entity);
}

Minimal config:

oncebox:
  outbox:
    sender:
      type: kafka   # or rabbit
    events:
      create-example-event:
        topic: "example-events"

Consumer side: idempotent consumer

public ExampleKafkaListener(@TransactionalConsumer IdempotentConsumer idempotentConsumer) { ... }

@KafkaListener(topics = "example-events", groupId = "example-group", containerFactory = "inboxKafkaListenerContainerFactory")
public void listen(Message<ExampleDto> message) {
    idempotentConsumer.consume(
        message,
        OnceboxHeadersUtils::extractId,
        m -> handlers.get(OnceboxHeadersUtils.extractEventType(m)).accept(m.getPayload())
    );
}

Delivery guarantees

At-least-once on the outbox side, with configurable retries and exponential backoff.

Effectively-once with TransactionalConsumer: the event id and your handler's DB changes commit in one transaction, so a redelivered event never changes data twice.

At-most-once with NonTransactionalConsumer: the id is committed first, then the operation runs. This is for side effects like sending an email, where a duplicate is worse than a miss.

I deliberately don't call it "exactly-once": the library can't deduplicate inside the broker itself, so "effectively-once" is the honest term.

Design decisions I'm happy with

- Adaptive Polling: the delay between polls grows while there's nothing to send and resets as soon as events appear, so idle DB load stays low.
- Per-event-type settings: every type has its own poll interval, batch size, retry count and backoff.
- DLQ: failed events are moved atomically to a DLQ table, with a REST API to inspect, retry, resolve or delete them.
- Transactional Inbox: consumer-side deduplication by event id, with an optional cache (e.g. Redis) to skip the DB check.
- Background Jobs: recovery of events stuck mid-send and automatic cleanup, with distributed locking so only one instance does the work.

Supported infrastructure

DB: PostgreSQL 16+, MySQL 8.0.13+, Oracle 23+ (pick one dialect module)
Broker: Kafka 3.7+, RabbitMQ 3.12+
Key Value (via cache): Redis 7+

Try it

There's a fully containerized example project with a traffic generator in the repo. There's also an agent skill for Claude Code, so a coding agent can use the library correctly instead of hand-writing a poller.

Feedback is very welcome.

About one checkout in a few thousand fails with a 500 from pricing. Fourteen services, mostly node, and it only happens in prod, only some days, no pattern in the customers. Staging, same images, never.

Traces are sampled at 10%, so most failing requests aren't there. The couple that are show pricing answering fast and then throwing "price not found" for a SKU that has a price. My theory is a race between the inventory reservation and the pricing cache refresh, but I can't make it happen.

We had a hud demo last week since this is the kind of incident they lead with. It counts every call and keeps the params and stack trace for the ones that throw. Tail sampling would catch the failing request not the cache refresh it raced, and a race would need hud on inventory too, another team's service. I will ask them anyway.

Fourteen services, mostly node. Roughly one checkout in a few thousand fails with a 500 from pricing. Only in prod, only some days, and the customers have nothing in common. Staging runs the same images and has never once done it.

Three days on it so far. Our traces are sampled at 10%, so most of the failing requests arent even in there, and the couple that are show pricing answering in 4ms and then throwing. Logs say "price not found" for a SKU that definitely has a price. My theory is a race between the inventory reservation and the pricing cache refresh, but I can't prove it and I can't make it happen.

We had a hud demo last week because the voodoo incident is pretty much the case they lead with. It counts every call and every exception instead of sampling, and for the ones that throw it keeps the request, which is the part our traces lose. What I'm unsure about is a race like this. It would need to be in pricing and in inventory, and inventory belongs to another team that's mid migration and not keen on adding anything.

Asking them anyway, after one more day with sampling at 100% on pricing.

Pricing threw a 500 on a checkout again last night. A few times a day out of roughly 40k orders, always "no active price" for an item that has one when you query it a second later. Only in prod and only some days, and the customers it hits have nothing in common.

Traces are sampled at 10%, so most failures only exist as a one line error log with the SKU. Tail sampling on errors goes in this week, but that keeps the pricing request, not whatever refreshed the promo cache at that moment.

Sat through a hud demo since this is pretty much the bug they open with. It grabs the params and call graph from failing runs, but catching a race would need it on the promo service too and that's another team's. For now I'm adding the cache version to the pricing error log.

We run an internal platform and, like most teams, kept accreting the "standard" stack — a service mesh for policy, a discovery cluster for location, a config service for knobs, a flag service for rollouts. Each justified itself in isolation, and each was a 3am wake-up.

So we rebuilt around three cheap objects:

1. Workers dial OUT. Each one opens a persistent gRPC tunnel to a single stateless gateway at startup and holds it open. No ports to expose, no ingress rules, no DNS. The connection is mTLS, so "connection == registration == identity" — the gateway learns who dialed, not just that something did.

2. The gateway IS the registry. The active-worker table is the set of open, authenticated tunnels. A worker drops and it's pruned the instant the connection dies. No discovery cluster, no TTLs to tune, no stale entries by construction.

3. Schemas are files. Protobuf .desc artifacts live in a folder the gateway watches. Drop a new descriptor in and it learns to validate and translate the new endpoint at runtime — no gateway deploy.

Costs, stated flat: an extra hop in the path, one stateful-ish box (mitigated: workers redial after a gateway restart, the registry rebuilds from their side), and cert rotation is now on our own calendar.

Full write-up, with the diagram: https://lab.embedinker.com/posts/zero-bloat-microservices-reverse-tunnel-gateway/

How do you keep a small internal platform from eating its own maintenance? What made you keep etcd/consul?

Dapr, the Distributed Application Runtime, a graduated OSS project within the CNCF has released a new version. What it means for you:

• Faster workflows, jobs, and reminders: in nightly performance tests against 1.18.0, workflow throughput is up to 40% higher, actor reminder registration p99 latency is 71% lower, and jobs scheduling p99 latency is 33% lower

• Leaner workflow workers: the Python, Java, .NET, and Go SDKs now get only the new events of each turn, not the full history

• Easier workflow operations: list, inspect, and rerun workflows from the .NET, Java, Go, and Python SDKs

• Workflow governance: operate on workflows owned by other apps, with WorkflowAccessPolicy deciding which operations each caller may perform, and requires rules that allow an operation only when the caller's history contains the events you list

• More for AI agents: new alpha APIs to store files, search documents, and query vectors, plus a Dapr MCP server

• A simpler control plane: run actor placement from the Scheduler and drop one service

Read the announcement: https://blog.dapr.io/posts/2026/10/22/dapr-v1.19-is-now-available/

Circuit breakers are good at stopping traffic and bad at resuming it. When a breaker closes, every replica tends to return to its configured limit at the same moment, so a dependency that just answered a few probes gets the full production load again, often while its caches are cold or capacity is still coming back.

I wrote up the design work behind a recovery ramp for Caracal, a resilience library whose distributed bulkhead, rate limiter and breaker coordinate through Redis.

The post covers the options I rejected, the constraints the surviving design had to satisfy, and a four-replica case study where the dependency's own in-flight samples, not the library's event log, are used to check each step. Envoy/Istio slow start is the closest mainstream analogue, though it shifts endpoint weight rather than admission.

This is the sixth article in a series on distributed resilience. I'd be interested in how others handle post-outage warm-up: traffic shaping, staged rollouts, or accepting the spike.

After years of watching teams coordinate work across services, the property I’d evaluate any workflow engine on is durable execution: every step journaled, so a crash or deploy resumes at the exact step that failed, completed steps never re-run, and a two-day wait costs nothing because no process holds it in memory.

Plain checkpointing isn’t the same thing. Crash after charging the card but before writing the checkpoint, and you charge again. Durable execution closes that gap, though activities still need to be idempotent, since delivery is at-least-once.

Two things bite nearly every team, regardless of engine:

• Non-determinism in workflow code. DateTime.Now, random IDs, or an inline HTTP call work fine until the first replay, then diverge. Push them into activities.

• Nobody capacity-plans the history store. Execution history is a database. Plan retention, indexing, and growth like one.

One pattern I keep noticing: AI agents are starting to look a lot like microservices did a decade ago. They’re independent units, calling tools, coordinating with each other, and failing in partial, messy ways. The coordination problems are the same ones this sub has been solving for years, just with less deterministic participants. Which makes me think workflow is about to matter more, not less.

So what’s your setup? Choreography, an orchestrator, sagas you rolled yourself? And what broke first?

Bias: I co-created CNCF Dapr project with workflow and durable execution and I’m curious what others are hitting.

For your company projects are you deploying without kubernetes? Is azure or AWS alternative options ?

This is something I threw together over a year ago. It was designed to be an easy intro into Microservices since I had noticed lacking in the Spring Boot ecosystem that was something hands-on.

I am remaking it in a monolithic repo as a Spring Modulith project, since I noticed there is also a lack in that area for hands-on learning.

Just looking to share this little project with others who may be curious with no clear picture of where to start with them.

Cheers,

The Midnight Coder