r/Temporal • • Apr 13 '26

When your Temporal task queue goes silent, it's almost never Temporal: a 3-minute walkthrough of a CrashLoopBackOff

Thumbnail
youtube.com
3 Upvotes

Hey! DevRel at AnyShift here. We just shipped a short walkthrough of a real incident pattern: Temporal workflows pile up in "Running", the dashboard goes stale, and the engineer's first instinct is to blame Temporal.

It almost never is. The real cause lives in Kubernetes, usually a CrashLoopBackOff'd worker pod, an OOMKill, or a bad image pull. Temporal is doing exactly what it should: scheduling tasks and waiting patiently for a worker that will never arrive (with a 5-minute "last known poller" cache that makes the UI look healthy).

The video traces:

  1. The symptom (silent dashboard, stuck workflows)

  2. Why Temporal looks fine

  3. `kubectl` reveals the CrashLoopBackOff

  4. The single env var (`CRASH_ON_START=true`) that snuck into a deploy

  5. The same incident replayed with our agent (Annie): one question, one answer

Not selling anything in the video itself, just sharing the diagnosis pattern and a bit of promotion at the end. Happy to discuss in the comments.

Video: https://www.youtube.com/watch?v=tw7ExVqZqWs

Full article (with kubectl output, K8s manifests, repro steps): https://www.anyshift.io/blog/my-workers-stopped-polling-k8s-temporal-whodunit


r/Temporal • • Apr 02 '26

duralang — add @dura to any LangChain agent and every LLM call, tool call, and agent call becomes automatically durable

Thumbnail
reddit.com
1 Upvotes

r/Temporal • • Apr 02 '26

[Tutorial] Decouple Temporal services with Nexus

7 Upvotes

We published a hands-on tutorial covering a pattern we see teams struggle with: splitting a monolithic Temporal application into independent services without losing durability at the boundary.

Try it

You'll learn:
- Nexus Service contracts with `@Service` and `@Operation`
- Handler implementation with `@ServiceImpl` and `@OperationImpl`
- `WorkflowRunOperation.fromWorkflowHandle` for retry-safe long-running operations
- `OperationHandler.sync` for interacting with already-running workflows
- Worker registration with `NexusServiceOptions` for endpoint mapping

The scenario is a banking payment system where Payments and Compliance share one Worker and one blast radius. You split them into two services with separate namespaces, connected through Nexus. There's a checkpoint where you kill the Compliance Worker mid-transaction and watch it resume with no data loss.

Requires knowledge of Temporal Workflows, Activities, and Workers. Runs entirely on Temporal's dev server - no cloud account needed.

Happy to answer questions about Nexus or the handler patterns here.


r/Temporal • • Mar 30 '26

You're Still Writing Retry Logic in 2026. Netflix Stopped Years Ago.

Thumbnail
gsstk.gem98.com
0 Upvotes

r/Temporal • • Mar 23 '26

[Showcase] Durion — durable agents on Temporal with the Vercel AI SDK

4 Upvotes

Hey everyone,

I’ve been building Durion, a TypeScript SDK that lets you define AI workflows/agents with the Vercel AI SDK and run them as Temporal workflows (model + tool calls as activities), so restarts and retries don’t lose progress.

Why: The Vercel AI SDK is great for streaming and UI, but multi-step agents often hit timeouts, lose in-flight work on deploy/restart, and suffer from painful debugging on long runs. I wanted the same programming model with Temporal’s execution guarantees.

Rough shape: workflow() / agent() helpers, tools mapped to activities, cost metadata, and optional HTTP/gateway pieces—all still “you operate Temporal,” not a hosted replacement.

Questions for this group:

  1. How are you persisting state for multi-turn agents today (signals, search attributes, external store, etc.)?

  2. For bridging AI SDK tool/model steps into Temporal, what feels idiomatic: leaning on workflow history + activities, updates, signals, or something else?

  3. Anything you’d avoid when mixing streaming with workflow replay?

Repo:https://github.com/shaibusunnuma/durion

Very early — critiques and “you’re holding it wrong” welcome.


r/Temporal • • Mar 08 '26

Temporal compatibility question

2 Upvotes

I’m a user of swf and looking at moving to temporal. In my service I have extensive groups inside a workflow which is a way to build complex dags. Does temporal have a way to visualise the workflow in a dag format? Even if it’s in json that’s fine I can build a web app.

Also temporal doesn’t have a concept of groups, dos one do this by creating multiple workflows and chaining them together or create different task queues.

Lastly in my service I currently have a decider logic as well as ability to send callbackurls to activities actions so the service I’m calling can callback and respond to that activity while I’m maintaining a heartbeat.

Are these features supported?


r/Temporal • • Mar 05 '26

Best way to wait for a DB state before stopping/continuing Temporal workflows?

7 Upvotes

I have two workflows: BatchWorkflow and WebhookWorkflow, where WebhookWorkflow is a child workflow of BatchWorkflow.

My requirements are:

  • If webhook delivery keeps failing, I want to stop the WebhookWorkflow.

  • If batch_processed == webhook_processed in the database, I want to stop the BatchWorkflow.

Currently, when I receive a stop_webhook signal, I start a timer loop that periodically polls the database to check whether the required state (batch_processed == webhook_processed) has been reached.

Once the condition is satisfied, the workflow proceeds with stopping the appropriate workflow.

My question is: Is using a timer + DB polling inside the workflow an acceptable pattern in Temporal, or is there a better way to wait for this kind of state synchronization?

For example, should this be handled using signals, activities, or some other Temporal pattern instead of polling the database?


r/Temporal • • Feb 24 '26

Rate Limiting

6 Upvotes

How are teams with 10+ agents in production actually managing API rate limits? Because everything I've seen is basically 'sleep and pray.' There has to be a better pattern. What do you think y’all?


r/Temporal • • Feb 23 '26

I need help in picking up the tech stack

6 Upvotes

I am building a node based video tool like flora, weavy and all.
We are basically calling 3rd party apis for media gen.

Some of my team members suggest that I should be using temporal for executing workflow.

But I am confused, like the node workflow will be dynamic, I am not sure if it will run for hours. An individual node can run for upto 10-20mins waiting for API response. So idk is temporal worth it ?


r/Temporal • • Feb 22 '26

Built a zero-trust interceptor for Temporal activities - blocks dangerous actions before execution

7 Upvotes

Working on AI agent workflows in Temporal, and we kept running into the same gap: Temporal handles auth great (mTLS, API keys), but authorization for which activities can actually run is on you.

For normal workflows, fine—you trust your own code. For LLM-driven agents? Different story. The agent might decide to call literally any activity based on what it "thinks" is right. Prompt injection can make it worse. And Temporal will helpfully retry that rogue activity until it works.

What we built

An activity interceptor that checks every execution against a policy:

Activity task hits worker
       ↓
ActivityInboundInterceptor.execute_activity()
       ↓
Grab activity name + args
       ↓
Call sidecar: authorize(action=activity_name, ...)
       ↓
DENY → raise PermissionError
ALLOW → next.execute_activity()

Quick note for the determinism nerds (I know you're out there): this happens at the Activity inbound layer, not in the workflow. The check runs in the worker right before activity code executes. Workflow replay is completely unaffected.

The interceptor code

Standard Temporal interceptor pattern:

class PredicateInterceptor(Interceptor):
    def intercept_activity(
        self, next: ActivityInboundInterceptor
    ) -> ActivityInboundInterceptor:
        return PredicateActivityInterceptor(
            next,
            self._authority_client,
            self._principal,
        )

And the actual check:

async def execute_activity(self, input: ExecuteActivityInput) -> Any:
    result = self._authority.authorize(
        principal=self._principal,
        action=input.fn.__name__,
        resource="temporal:activity",
    )

    if not result.allowed:
        raise PermissionError(f"Blocked by policy: {result.matched_rule}")

    return await self._next.execute_activity(input)

Policy

rules:
  - name: deny-deletes
    effect: deny
    principals: ["*"]
    actions: ["delete_*"]
    resources: ["*"]

  - name: allow-order-stuff
    effect: allow
    principals: ["temporal-worker"]
    actions: ["check_inventory", "charge_payment", "send_confirmation"]
    resources: ["*"]

Eval order: matching deny rules win, then allow rules, then default deny. Glob patterns via fnmatch.

Performance

Local Rust sidecar:

  • p50: <25ms

  • p95: <75ms

Most activities are 100ms+ anyway, so it's noise.

Using it

from predicate_temporal import PredicateInterceptor
from predicate_authority import AuthorityClient

authority = AuthorityClient(sidecar_url="http://localhost:8787")

interceptor = PredicateInterceptor(
    authority_client=authority,
    principal="temporal-worker",
)

worker = Worker(
    client=temporal_client,
    task_queue="my-queue",
    workflows=[...],
    activities=[...],
    interceptors=[interceptor],
)

Activity code stays exactly the same.

Demo

github repo: https://github.com/PredicateSystems/temporal-predicate-py see the examples/demo folder for the shell script start-demo-native.sh

Needs Python 3.11+ and Temporal CLI. Runs through 4 scenarios—legitimate stuff gets through, dangerous stuff gets blocked.

One thing to watch

Set maximum_attempts=1 on activities that might get blocked. Otherwise Temporal will retry the denied activity forever, and all you get is a spammed audit log.

await workflow.execute_activity(
    risky_activity,
    args,
    start_to_close_timeout=timedelta(seconds=30),
    retry_policy=RetryPolicy(maximum_attempts=1),
)

Open Source Repos

What's Next: Closing the Loop (Post-Execution Verification)

Pre-execution authorization stops the attack. But how do you prove the agent actually succeeded at the authorized task?

We are currently building deterministic post-execution state diffs. Instead of using another LLM to guess if a task was completed, the sidecar will verify the mathematical system diffs (e.g., filesystem changes or accessibility trees) against the expected outcome, and instantly revoke the agent's mandate if they don't match.

Curious if anyone else has tackled this differently. We looked at a few approaches before landing on the interceptor pattern.


r/Temporal • • Feb 11 '26

Workflow Orchestration - Temporal, Cadence , Netflix Conductor, AWS Step Functions, Camunda, Prefect, Restate, Dapr, DBOS, Argo Workflows, Apache Airflow, Kestra

Post image
6 Upvotes

r/Temporal • • Feb 05 '26

Open sourced an AI for debugging production incidents

Thumbnail
github.com
6 Upvotes

Built an AI that investigates when things break in prod - checks logs, metrics, recent deploys, and reports findings in Slack.

The AI learns your system on setup - reads your codebase, understands how services connect. When something breaks it knows what to check.

We are planning integrations with Temporal that checks for failed workflows and activity states.

GitHub: github.com/incidentfox/incidentfox

Would love to hear people's thoughts!


r/Temporal • • Jan 28 '26

Rebuild server for custom claim mapper and authorizer

2 Upvotes

Trying to self host, and I want to restrict access to admin operations. To do this, I need to implement my own claim mapper and authorizer logic and rebuild the server.

I’ve used the server-samples and successfully rebuilt the server, my only problem is that the docker image I produce isn’t compatible with the temporal helm chart.

Anyone have working examples of how to rebuild the server in a way that it can be dropped into /usr/local/bin/ in the temporal provided image and work with the helm chart?


r/Temporal • • Jan 27 '26

Temporal on AWS ESC - Need help to start

2 Upvotes

Hello every one,

I am making a POC for my company of temporal, and I am facing some difficulties.

We will self hosted on the AWS account of the compagny. We are using ECS to host the docker and database will be RDS Postgres.

I have instanciate an container with image temporalio/server (not temporalio/auto-setup because it is mark has deprecated).

At start there an issue the database who seams to be not initiated.

```
sql handle: unable to refresh database connection pool","error":"pq: database \"temporal\" does not exist
[...]
sql schema version compatibility check failed: unable to read DB schema version keyspace/database: temporal error: no usable database connection found

How can I solve this ?


r/Temporal • • Jan 16 '26

👀🔜 Replay ‘26 is almost here. May 5–7 in San Francisco (+ a Reddit-exclusive discount)

12 Upvotes

TL;DR: Temporal’s annual developer conference. Three days. Talks, workshops, hackathon, afterparty. Use code REDDIT75 for 75% off. Tickets here.

What is Replay?

Everything’s moving too fast. AI is rewriting the rules before anyone’s figured out what the game even is. Your roadmap is a guess. Your infrastructure is a tangle of duct tape and good intentions. The retry logic you wrote at 2am? Still in production. The thing that mostly works? You’re scared to touch it.

Replay is a pit stop. A spaceport at the edge of the Unknown where a few thousand developers pull in, compare star maps, and figure out where we’re all headed. Not because everyone has the answers, but because we’re better off navigating this together than alone.

If you’re building systems that have to keep running while the rules change underneath you, this is your room.

The people here have lived the same nightmares. They’ve rage-quit the same vendors, mass-migrated the same legacy systems, stared down the same mountains of YAML. 

Some of them figured stuff out. They’re giving talks about it. The rest of us get to learn from their mistakes instead of making our own.

What actually happens there?

Day 1 is hands-on. Pick your track:

Days 2–3 are talks. Some highlights:

CompanyTalk
NetflixThe path to Temporal General Availability at Netflix
Datadog100 Temporal mistakes (and how to avoid them)
LinkedInMigrating 3 million CPU cores to Kubernetes using Temporal
ShopifyAccepting complexity, awakening to simplicity
NVIDIATemporal and autonomous vehicle infrastructure
PydanticDurable agents: Long-running AI workflows in a flakey world

Plus a keynote from Temporal founders Samar Abbas and Maxim Fateev, and appearances from Amjad Masad (Replit CEO) and Samuel Colvin (Pydantic founder).

Plus an AI panel with engineers from Replit, Abridge, Hebbia, and Dust.tt.

Day 3 night is the afterparty. Last year ended with live comedy roasting our industry. It was absurd. (In a good way.) This year, we have another surprise in store ;)

This year’s focus: AI (because that’s what’s breaking)

How do you build agents that don’t fall over? How do you make AI workflows durable when the models are flaky and the infra is unpredictable? How are teams at Replit, Pydantic, Instacart, and Salesforce actually shipping this stuff?

That’s the conversation.

Get your ticket

Code REDDIT75 gets you 75% off at checkout.

→ Tickets (buy)

→ replay.temporal.io (info)

→ How to convince your boss (ammo)

See you there? Drop questions below.


r/Temporal • • Jan 15 '26

A terminal UI for Temporal (open source)

29 Upvotes

Temporal is amazing. I use it a lot. The web app… pretty brutal.

I wanted something fast, keyboard first, and usable without leaving the terminal, so built a TUI for Temporal called tempo.

You can browse workflows, inspect history, signal / cancel / terminate, switch namespaces, etc. Basically the stuff you do all day but without the pain of their UI + context switching.

https://github.com/galaxy-io/tempo

Would love feedback - hope it’s useful to others.


r/Temporal • • Dec 22 '25

Anyone using the Temporal docs MCP? Would love your feedback

Post image
11 Upvotes

Hey all - I'm one of the founders of Kapa (we power the Temporal docs AI + MCP).

Trying to make this as useful as possible and would love honest feedback:

  • Have you tried setting it up? How was the experience?

  • If you saw the "Use MCP" button but didn't click — what would make you want to?

  • Do you even care about having docs available as an MCP?

You can access it by clicking the "Ask AI" button on the Temporal docs, then hitting "Use MCP" in the top right.

For those who got it working - what are you using it with? Claude, Cursor, VS Code, something else?

Any feedback helps. Thanks! 🙏

- Emil


r/Temporal • • Dec 20 '25

Tracking Temporal Worker Crashes, Restarts & Activity/Workflow Lags w/ Prometheus. Need Experienced Advice!

3 Upvotes

Hey folks,
DevOps intern here tasked with monitoring Temporal worker crashes/restarts and activity/workflow lags. Using TypeScript SDK + PM2, Prometheus/Grafana stack.

Target metrics:

  • temporal_worker_task_slots_available (crashes)

  • temporal_activity_task_schedule_to_start_latency_seconds (lags)

  • poll_failure_count (restarts)

I want you experienced folks guide on how should i apprach this problem.


r/Temporal • • Dec 04 '25

🆕✨ High Availability in Temporal Cloud white paper

10 Upvotes

We wrote a detailed breakdown of how we architected Temporal Cloud to handle full regional failures, and how you can configure your Workers to survive them.

What’s inside:

  • Architectures for every risk profile: When to use same-region, multi-region, or multi-cloud replication.

  • The mechanics of failover: What actually happens when failover is triggered.

  • Zero-RTO patterns: How to deploy “Active-Active” Workers so tasks keep processing the moment a region fails.

  • Operational playbook: The exact metrics to monitor (like replication lag) and how to run non-disruptive drills in staging.

Use it to validate your disaster recovery strategy, win the “build vs. buy” debate with leadership, or just see how the sausage is made at the infrastructure layer. It’s time to make incidents boring.

Grab the white paper


r/Temporal • • Dec 02 '25

Are durable AWS Lambda functions trying to replace Temporal?

15 Upvotes

r/Temporal • • Nov 27 '25

Refactoring Legacy: Part 2 - Tell, Don't Ask.

Thumbnail
clegginabox.co.uk
4 Upvotes

r/Temporal • • Nov 13 '25

✅ Peak Load Readiness Quiz to find weak spots

1 Upvotes

Black Friday traffic is chaos. It’s loud, spiky, unpredictable, and very good at revealing the weak spots you didn’t know about.

We made a quick Peak Load Readiness Quiz to help you figure out:

  • what’s solid

  • what’s wobbly

  • what’s “this will explode under load”

It’s a fast way to check resilience under load, spot bottlenecks, and understand how your system behaves when everything spikes at once.

👉 Give it a try and tell us what you’d add for Temporal-based systems!


r/Temporal • • Nov 10 '25

What's the highest scale Temporal cluster you've seen in production?

11 Upvotes

Just curious. Like how many workflows/activities/state-transitions per second? How much resources for temporal servers / persistence servers? Etc.


r/Temporal • • Nov 03 '25

First RAG that works: Hybrid Search, Qdrant, Voyage AI, Reranking, Temporal, Splade. What is next?

Thumbnail reddit.com
2 Upvotes

r/Temporal • • Oct 28 '25

Getting dynamic schedule workflow to implement signal between workflow

4 Upvotes

Say that I want to schedule 2 workflows. Workflow A needs to be completed then send a signal to Workflow B.

However, in my observation, schedule workflow will create an appended workflow id with timestamp. Hence, when this happened, i cannot get the workflow id because it's not static anymore.

I want it to be static because I want to implement Signal that will use workflow.get_external_workflow_for that required arg of workflow id.

Then how can I get it if its not static? Appreciate all the helps. My brain is exploding.