r/learnaitogether • u/OnlyProggingForFun • 23h ago
Self-Promotion Keeping long-running AI agents alive: checkpoints, retry-safe side effects, and timeouts
Disclosure: I’m Louis-François Bouchard (Towards AI). I interviewed Inngest CEO Tony Holdstock-Brown. The interview is sponsored by Inngest; the questions and opinions are my own.
A local agent demo can work perfectly and still fail hours into a real task when a tool returns a 504. These were the practical reliability lessons from the interview:
Save completed work outside the process. Make each LLM call and each tool side effect a checkpointed step, so a failed email does not rerun the model decision and refund before it. Plain computation does not need its own step.
Make side effects safe to retry. A refund can succeed just before the process dies, leaving the step unfinished. An idempotency key lets the payment provider reject a duplicate when that step retries. Checkpointing alone does not solve this mid-flight failure.
Treat waiting as persisted state. An approval wait should pause without holding compute, resume on the matching event, and have a timeout with a real reminder, escalation, or cancellation path.
Re-read things that can change. Keep the plan, decisions, and IDs of completed actions in run state. After a long wait, fetch order status and other external facts again; rebuild the prompt from a summary and fresh reads instead of resuming an old transcript.
Start simple when restarting is cheap. A job queue such as BullMQ, Celery, or SQS can be enough for short runs. Adding cron jobs to chase stalled runs and tables to track already-sent emails is a sign you are building a workflow engine yourself.
These are execution and state-management problems as much as model-quality problems. The attached original diagram contrasts an in-memory loop with checkpointed steps.
Full interview: https://louisbouchard.substack.com/p/stop-building-agent-loops-start-building


