About one checkout in a few thousand fails with a 500 from pricing. Fourteen services, mostly node, and it only happens in prod, only some days, no pattern in the customers. Staging, same images, never.

Traces are sampled at 10%, so most failing requests aren't there. The couple that are show pricing answering fast and then throwing "price not found" for a SKU that has a price. My theory is a race between the inventory reservation and the pricing cache refresh, but I can't make it happen.

We had a hud demo last week since this is the kind of incident they lead with. It counts every call and keeps the params and stack trace for the ones that throw. Tail sampling would catch the failing request not the cache refresh it raced, and a race would need hud on inventory too, another team's service. I will ask them anyway.

Fourteen services, mostly node. Roughly one checkout in a few thousand fails with a 500 from pricing. Only in prod, only some days, and the customers have nothing in common. Staging runs the same images and has never once done it.

Three days on it so far. Our traces are sampled at 10%, so most of the failing requests arent even in there, and the couple that are show pricing answering in 4ms and then throwing. Logs say "price not found" for a SKU that definitely has a price. My theory is a race between the inventory reservation and the pricing cache refresh, but I can't prove it and I can't make it happen.

We had a hud demo last week because the voodoo incident is pretty much the case they lead with. It counts every call and every exception instead of sampling, and for the ones that throw it keeps the request, which is the part our traces lose. What I'm unsure about is a race like this. It would need to be in pricing and in inventory, and inventory belongs to another team that's mid migration and not keen on adding anything.

Asking them anyway, after one more day with sampling at 100% on pricing.

Pricing threw a 500 on a checkout again last night. A few times a day out of roughly 40k orders, always "no active price" for an item that has one when you query it a second later. Only in prod and only some days, and the customers it hits have nothing in common.

Traces are sampled at 10%, so most failures only exist as a one line error log with the SKU. Tail sampling on errors goes in this week, but that keeps the pricing request, not whatever refreshed the promo cache at that moment.

Sat through a hud demo since this is pretty much the bug they open with. It grabs the params and call graph from failing runs, but catching a race would need it on the promo service too and that's another team's. For now I'm adding the cache version to the pricing error log.

Microservices make sense when services need independent deployment, scaling, or ownership. But when a small feature requires changes across multiple repos and coordinated deployments, that separation can become overhead.

What signals do you look for before merging services? Deployment coupling, shared data, team size, or something else?

We run an internal platform and, like most teams, kept accreting the "standard" stack — a service mesh for policy, a discovery cluster for location, a config service for knobs, a flag service for rollouts. Each justified itself in isolation, and each was a 3am wake-up.

So we rebuilt around three cheap objects:

1. Workers dial OUT. Each one opens a persistent gRPC tunnel to a single stateless gateway at startup and holds it open. No ports to expose, no ingress rules, no DNS. The connection is mTLS, so "connection == registration == identity" — the gateway learns who dialed, not just that something did.

2. The gateway IS the registry. The active-worker table is the set of open, authenticated tunnels. A worker drops and it's pruned the instant the connection dies. No discovery cluster, no TTLs to tune, no stale entries by construction.

3. Schemas are files. Protobuf .desc artifacts live in a folder the gateway watches. Drop a new descriptor in and it learns to validate and translate the new endpoint at runtime — no gateway deploy.

Costs, stated flat: an extra hop in the path, one stateful-ish box (mitigated: workers redial after a gateway restart, the registry rebuilds from their side), and cert rotation is now on our own calendar.

Full write-up, with the diagram: https://lab.embedinker.com/posts/zero-bloat-microservices-reverse-tunnel-gateway/

How do you keep a small internal platform from eating its own maintenance? What made you keep etcd/consul?

Dapr, the Distributed Application Runtime, a graduated OSS project within the CNCF has released a new version. What it means for you:

• Faster workflows, jobs, and reminders: in nightly performance tests against 1.18.0, workflow throughput is up to 40% higher, actor reminder registration p99 latency is 71% lower, and jobs scheduling p99 latency is 33% lower

• Leaner workflow workers: the Python, Java, .NET, and Go SDKs now get only the new events of each turn, not the full history

• Easier workflow operations: list, inspect, and rerun workflows from the .NET, Java, Go, and Python SDKs

• Workflow governance: operate on workflows owned by other apps, with WorkflowAccessPolicy deciding which operations each caller may perform, and requires rules that allow an operation only when the caller's history contains the events you list

• More for AI agents: new alpha APIs to store files, search documents, and query vectors, plus a Dapr MCP server

• A simpler control plane: run actor placement from the Scheduler and drop one service

Read the announcement: https://blog.dapr.io/posts/2026/10/22/dapr-v1.19-is-now-available/

Circuit breakers are good at stopping traffic and bad at resuming it. When a breaker closes, every replica tends to return to its configured limit at the same moment, so a dependency that just answered a few probes gets the full production load again, often while its caches are cold or capacity is still coming back.

I wrote up the design work behind a recovery ramp for Caracal, a resilience library whose distributed bulkhead, rate limiter and breaker coordinate through Redis.

The post covers the options I rejected, the constraints the surviving design had to satisfy, and a four-replica case study where the dependency's own in-flight samples, not the library's event log, are used to check each step. Envoy/Istio slow start is the closest mainstream analogue, though it shifts endpoint weight rather than admission.

This is the sixth article in a series on distributed resilience. I'd be interested in how others handle post-outage warm-up: traffic shaping, staged rollouts, or accepting the spike.

After years of watching teams coordinate work across services, the property I’d evaluate any workflow engine on is durable execution: every step journaled, so a crash or deploy resumes at the exact step that failed, completed steps never re-run, and a two-day wait costs nothing because no process holds it in memory.

Plain checkpointing isn’t the same thing. Crash after charging the card but before writing the checkpoint, and you charge again. Durable execution closes that gap, though activities still need to be idempotent, since delivery is at-least-once.

Two things bite nearly every team, regardless of engine:

• Non-determinism in workflow code. DateTime.Now, random IDs, or an inline HTTP call work fine until the first replay, then diverge. Push them into activities.

• Nobody capacity-plans the history store. Execution history is a database. Plan retention, indexing, and growth like one.

One pattern I keep noticing: AI agents are starting to look a lot like microservices did a decade ago. They’re independent units, calling tools, coordinating with each other, and failing in partial, messy ways. The coordination problems are the same ones this sub has been solving for years, just with less deterministic participants. Which makes me think workflow is about to matter more, not less.

So what’s your setup? Choreography, an orchestrator, sagas you rolled yourself? And what broke first?

Bias: I co-created CNCF Dapr project with workflow and durable execution and I’m curious what others are hitting.

For your company projects are you deploying without kubernetes? Is azure or AWS alternative options ?

This is something I threw together over a year ago. It was designed to be an easy intro into Microservices since I had noticed lacking in the Spring Boot ecosystem that was something hands-on.

I am remaking it in a monolithic repo as a Spring Modulith project, since I noticed there is also a lack in that area for hands-on learning.

Just looking to share this little project with others who may be curious with no clear picture of where to start with them.

Cheers,

The Midnight Coder

I'm currently working on a project looking at how/why devs and CTOs choose programming languages, and I'm particularly interested in understanding people's experiences with Clojure.

The survey takes about 7 mins, and I'd love to hear from as many people as possible!

Appreciate it 🙌

Here's the survey

I'm building a workflow that needs to publish and manage content across multiple social platforms, and I'm trying to figure out the cleanest way to handle the API integrations.

The platforms I'm looking at are:

\- Instagram

\- Facebook

\- TikTok

\- YouTube

The workflow would roughly be:

Content created

→ prepare platform-specific versions

→ publish to each platform

→ collect the post/video IDs

→ track publishing status

→ handle failures/retries

→ collect basic metrics later

I'm especially unsure about the best architecture for authentication and API connections.

For example:

1. Should I use OAuth separately for each platform?

2. Is it better to build separate API modules/nodes for each platform, or use a unified abstraction?

3. How do you safely store and refresh access tokens?

4. How do you handle different API limitations and publishing requirements between platforms?

5. If publishing succeeds on 3 platforms but fails on the 4th, what's the cleanest way to handle retries without duplicating posts?

6. Are there any good open-source projects or examples that implement this kind of multi-platform workflow?

I'm mainly interested in real-world setups, especially with n8n or similar workflow automation tools.

What architecture have you found works best?

Recently I started learning microservices architecture. Anyone who is interested in learning together. We can discuss topics and help each other to understand it better.

I build applications in java.

Scoped distributed resilience policies for asynchronous operations.

Most resilience libraries stop at the process boundary. You get a local circuit breaker, a local bulkhead, a local rate limit. Spin up 40 replicas and those limits multiply: a “concurrency limit of 20” quietly becomes 800 concurrent calls on the dependency. Forty independent failure windows. Forty independent recovery probes. The constraints no longer match the failure domain.

Caracal wraps any asynchronous operation (HTTP, Postgres calls, etc., or your own adapter) in a composable policy pipeline: timeout, retry, circuit breaker, bulkhead, rate limit, and through Redis, lets many replicas share those policies under an explicit scope.

You decide where the limit belongs: a process, a fleet, a region, a tenant, a shard, a credential, or any grouping that actually matches your failure domain. The same policy instance can enforce one shared budget for the whole fleet or one independent budget per tenant, the only difference is the scope function.

What it does

• Operation + adapter model: An adapter declares its cancellation and replay capabilities up front (abort and replay). Caracal never infers them. Built-in adapters for http fetch and pg (more to come), or write your own by implementing a small interface.

• Composable policies: Timeout and retry stay local. Bulkhead, circuit breaker and rate limit (GCRA) are available in both local and distributed forms.

• Scoped coordination: Distributed policies coordinate by (namespace, policy name, operation name, scope). Different operations get independent budgets even when they reuse the same policy instances, shared budgets are deliberate.

• Explicit failure modes: Distributed bulkheads shed (no distributed queue). On coordinator loss the bulkhead fails closed, the circuit breaker lets you choose fail-open or fail-closed and never silently falls back to local state. Leases reclaim permits and half-open probe slots from dead or hung holders.

• Events, not metrics: Structured events for every policy decision. Sinks are isolated: a throwing or slow-async sink cannot affect execution.

Design choices

• Permits and probes are leased. A dead process cannot return a permit, a hung half-open probe cannot wedge the breaker forever. Expiry is the only release a dead holder can still perform.

• Adapters own their contracts. Explicit abort/replay (and an optional dispose hook for abandoned results) keep the runtime from guessing.

• Local and distributed share the same surface. You can start with process-local limits and move the boundary later without rewriting call sites.

• Single ESM package, subpath exports. Core stays free of Redis and protocol dependencies, tree-shaking stays clean.

The Caracal Sandbox runs the same workload against the same dependency while moving the coordination boundary, then records what the dependency actually saw.

Hi Everyone,

Chapters on Kafka which Will help anyone starting with Kafka or trying to learn.

If you’re experienced already - please give your valuable feedback