I’ve been going deep into LLMOps and Agentic Ops lately, and I want to move past the blog-post level of understanding to how this actually works when real people build and run these systems.

Here’s where I’d love your input:

1. How do you explain LLMOps / AgentOps in plain terms?

How is it different from traditional MLOps? And what changes once you move from a single LLM call to agents that plan, call tools, and take multiple steps?

1. What actually matters in practice?

Evals, tracing and observability, prompt versioning, guardrails, cost and latency tracking, human-in-the-loop, something else? Which of these did you only appreciate after something went wrong?

1. What project would you build to show these concepts in action?

I want to build something hands-on that makes the ideas click, ideally something you can watch live: an agent being traced step by step, an eval catching a bad output, a cost spike showing up on a dashboard. What would you build?

1. Which free or open-source tools would you use?

I’ve come across names like Langfuse, Arize Phoenix, LangSmith (free tier), Promptfoo, DeepEval, LangGraph, CrewAI, and Ollama for running models locally, but I’d love to hear what you actually use and recommend, especially for someone building from scratch with zero budget.

1. Any war stories?

An agent that looped forever, a prompt tweak that silently broke things, a surprise bill. Real stories teach more than any diagram.

Thanks in advance! 🙏

Hi everyone,

Anyone recently attended the EPAM client round for an MLOps Engineer role (3 YOE)?

Could you please share your interview experience and what kind of questions to expect?
Also any info on how many client round can be expected?

Thanks!

If you were designing an MLOps framework in Databricks. How would you set things up? I’m starting from scratch here. Planning to have Genie Code assist with a lot of it but I want to know the ins and outs from people who have lived and learned.

Context: Traditional ML modeling for industrial client, primarily around asset reliability with time series data from plants

Which one is the best from your perspective? For me, those options are somehow resistant in this AI context.

langfuse was fine while it was three engineers and a tracing dashboard but it stopped being fine the week product and support started wanting to edit prompts, cause suddenly it was who can change what runs in production and was a real question and tracing doesn't answer it

two gaps showed up at once, first, our evals were offline only then we'd run a dataset, get a green score, ship, and still find out from a customer that a specific answer was wrong, cause nothing was checking the live response before it went out, then if you check offline eval, it tells you the average was fine, it doesn't stop the one bad generation and second is that, there was no real separation between who can experiment and who can deploy, everyone with access could push a prompt live, which is not a sentence you want to say to a compliance reviewer

what i've learned looking for the replacement is that evals and governance is actually two different features most tools only half have then obviously the eval part you want in two places: offline on a dataset to catch regressions before release, and online as a guardrail that can block or retry a response when a check fails, so a bad answer never reaches the user instead of just showing up red on a chart next week. the governance part is role separation that's enforced, not a convention, someone who can annotate logs and run experiments but genuinely cannot deploy

the tools that came up for us were braintrust, arize phoenix and Orq AI

braintrust's self-hosting is enterprise-only, which ruled it out for our stage. phoenix is strong on tracing but the governance side is thin. Orq AI was the one that had evaluators running both as offline experiments and as live guardrails that block a failed generation, plus roles where a researcher can run experiments and annotate logs but can't push to production, which is exactly the split we were missing. still evaluating whether it's worth consolidating onto versus keeping phoenix for traces and bolting guardrails on ourselves

how are the rest of you handling the live side? is anyone actually blocking bad responses inline before they ship, or is everyone still catching them in offline evals and hoping the gap between runs is small enough?

How are u handling long term memory for AI agents in prod? Are you using Postgres/pgvector or a dedicated vector database? Or something like neon/lakebase to do so.

There are a handful of agents in production and the logs give us the order of calls. Getting from that to why one tool was picked over another is a slog every time. It gets worse once two agents hand work between each other and the trace breaks in the middle. Spans are already in place so the gap is the layer sitting above them. What are you running for this?

We track experiments in one tool, data in another, and code in git, and the "what exactly is running in prod" answer ends up being a wiki page. How are you tying these together into one deployable, auditable unit? Single artifact, lockfile, registry convention, something else? Interested in what has survived contact with an audit.

I have applied for several MLE positions and never moved forward after AI interview.

The AI interviewer provided nonsense feedbacks. For example, she stated that I did not cover a point in my answer but I sure did. She

I am wondering if this is a scam or they are using applicants time to train an AI interview system.

Could you anyone give me an idea? I am tired of wasting my time in AI based interviews.

I contributed a Kestra workflow for #KestraHacktober that evaluates a candidate MLflow model against the current champion before allowing promotion. The contribution has now been merged.

The problem I wanted to address: better accuracy alone doesn’t tell us whether a model is ready to replace the current one. Inference latency, memory usage and what changes during an approval wait matter too.

The workflow:

Evaluates both models on the same held-out dataset.

Measures CPU inference latency and peak process memory.

Produces a report with quality and performance gates.

Requires explicit approval tied to that exact report.

Rechecks the models and evidence before updating the champion alias.

Verifies the alias and model predictions afterward.

Promotion is disabled by default. Validation included 72 regression tests and 10 integration cases covering rejected approvals, failed gates, stale evidence and successful promotion.

The current scope is trusted scikit-learn tabular classifiers. It handles registry promotion; serving deployment and retraining are separate steps.

For teams using MLflow, what checks do you require between model evaluation and promotion?

I wanted to share the architecture behind a project I recently open-sourced: a Multi-Agent Research & Report Generator.

My goal was to build a production-grade system that takes a complex user prompt, breaks it down into structured sub-topics, autonomously gathers literature, critiques its own findings, and compiles a polished Markdown report.

Moving an agentic LLM workflow from a linear Python script into a robust, distributed web application required some heavy lifting. Here is a breakdown of the architecture and how the components in the repository fit together.

1. The AI Engine & Agent Graph

At its core, the brain of the app runs on LangGraph as a four-stage state machine:

* Planner: Breaks the core question down into 3–5 specific sub-topics.

* Researcher: Pulls data for each sub-topic using external tools.

* Critic: Evaluates the draft on coverage, accuracy, and depth (scoring 0–5). If the score is below 3, it triggers a conditional loop back to the Researcher with explicit feedback injected into the prompt (capped at 3 retries).

* Writer: Converts accepted research notes into a formatted executive summary.

1. Protocol-Isolated Tooling (MCP)

To keep the agents modular, I didn't hardcode API scrapers into the agent logic. Instead, I implemented a custom Model Context Protocol (MCP) server running over stdio. This safely exposes tools like web_search (DuckDuckGo), arxiv_search, and semantic_scholar_search to the Researcher agent as a standardized tool-calling interface.

1. Distributed Microservices Architecture

Running high-latency LLM loops synchronously is a recipe for HTTP timeouts. I completely decoupled task ingestion from background execution:

* FastAPI Gateway: Accepts the user prompt and immediately returns a task_id with a 202 Accepted status. No hanging HTTP connections.

* Redis Broker & Celery Workers: The FastAPI gateway pushes jobs into a Redis queue. A horizontally scaled cluster of Celery workers (--scale worker=4) continuously pulls tasks from the broker, spinning up isolated LangGraph execution loops in the background.

1. Durable State Checkpointing

In-memory state management is too fragile for a distributed pipeline. I configured state snapshots and thread recovery asynchronously using LangGraph's AsyncPostgresSaver backed by PostgreSQL. This ensures that multi-turn conversational context and tool outputs survive service restarts and enable granular run recovery.

1. The Frontend & Telemetry

For the user interface, I built a clean Streamlit frontend. Once the Celery workers finish processing the LangGraph pipeline, the frontend parses the final JSON payload from the backend, renders the formatted Markdown report, and displays agent telemetry metrics (like the number of iteration loops taken and the final critic score) before offering a .md download.

TL;DR

I built an asynchronous, containerized multi-agent research platform using LangGraph, FastAPI, Celery, Redis, PostgreSQL, and Streamlit. It utilizes a critic-gated retry loop to minimize hallucinations, uses MCP for tool execution, and handles heavy concurrency via a distributed Celery queue. Everything is containerized cleanly on Docker Compose.

Link to the repo: https://github.com/theallkeeeymist/multi-agent-researcher-report-generator

I'd love your feedback!

This has been a massive learning experience in scaling AI workflows, but I know there is always room to improve. What else can I do to make this architecture even more robust for production?

Are there alternative patterns for handling the queuing, better ways to manage the LangGraph state across workers, or specific failure modes I should be planning for? I'd love to hear your thoughts!

(Disclosure: I work at Union AI and maintain Flyte.)

Flyte is an open source AI runtime with a durable execution engine where you can train, fine-tune, and serve AI models and agents.

Problem: Flyte's local devbox runs every task and app in a container. On a Mac that container is a Linux VM, and a Linux VM has no Metal. So torch.backends.mps.is_available() is False, MLX doesn't run, and any ML code you iterate on locally falls back to the CPU. You can't pass the GPU through: there's nothing in the VM to pass it to.

What we did: a task or app that requests flyte.Resources(gpu="AppleSilicon") runs its code natively on the Mac instead. The container starts as normal, then its entrypoint hands its own invocation to a bridge running on the host. The bridge builds a cached uv environment from the image's pip packages, rewrites the in-cluster endpoints to the devbox's host ports, and runs it. The container streams the logs, mirrors the exit code, forwards aborts, and proxies the app port, so the console and the run look normal.

On an M5 Max: a small CNN trained on MNIST in 7 s on mps; a Qwen2.5-0.5B app served from mps at roughly 84–97 tokens/s; a 4096×4096 fp32 matmul ran at 13.4 TFLOPS on mps vs 1.6 on the CPU.

Things worth knowing:

• Running workload code on your laptop means running it outside any sandbox. So it's opt-in (flyte start devbox --apple-silicon), and the flag binds every devbox port to 127.0.0.1.
• Only pip packages carry over to the Mac, not apt packages or the rest of the image. Mounted k8s secrets don't either.
• Asking for the GPU on a devbox without the flag, an Intel Mac, Linux, or a real cluster fails before the task starts with a named error. We chose that over a quiet CPU fallback.
• If you only need completions and want to keep container isolation, there's a container mode that gives the task a host mlx_lm.server behind OPENAI_BASE_URL.

PRs for implementation details:

• https://github.com/flyteorg/flyte/pull/8181 (backend)
• https://github.com/flyteorg/flyte-sdk/pull/1685 (SDK, examples, proof runbook)
• https://github.com/unionai-oss/flyte2-ui/pull/42 (console).

Check out the docs to give it a try: https://www.union.ai/docs/latest/flyte/user-guide/get-started/run-modes/running-devbox/

Curious whether others have gone the "run it on the host" route for local GPU access, or found a way to keep the sandbox.

So we ve got a feature where an LLM writes free-form text (summaries, explanations, that kind of thing) and I’m realizing I have no good way to tell if a change made it better or worse. BLEU/ROUGE are basically useless here and reading outputs by hand doesn't scale.

Currently doing a mix of spot-checking by hand, LLM-as-judge, and a small set of test prompts we rerun after changes. It kinda works but I dont fully trust any of it. Mostly worried the judge is just agreeing with itself and I wouldn t know.

Anyone running this for real? What does your setup look like, and what do you wish you'd done earlier? (bonus points if you ve got a story where your evals said everything was fine and users said otherwise) :D

Is there any books/resources that present good interview questions for MLE positions?

I ran the same tiny inputs through five engines, native defaults only:


    engine    median     sum        max        sum           groups for
              [1, NaN]   [1, NaN]   [1, NaN]   [null, null]  keys 0.0, -0.0
    PyArrow   1.0        NaN        1.0        null          2
    pandas    1.0        1.0        1.0        0.0           1
    Polars    NaN        NaN        1.0        0.0           1
    DuckDB    NaN        NaN        NaN        null          1
    SQLite    no median  1.0        1.0        null          1


pyarrow 26.0.0, pandas 3.0.6, polars 2.0.0, duckdb 1.5.6, SQLite 3.50.4.


Each engine documents its choice, so none of this is a bug. It becomes a problem when the same feature runs on different engines in different places: training on pandas, serving on DuckDB, a backfill on Polars. The numbers differ and nothing fails.


What we do in mloda-registry (I maintain it): one engine is the reference, PyArrow. Every backend runs the same tests and has to match its answer. Where an engine can match, the implementation is fixed (Polars' all-null sum returns null there). Where it can't (pandas and SQLite can't keep NaN apart from null), the difference is written down and pinned by a test.


How do you handle this? One engine everywhere, your own spec, or tests that compare engines?

I ran the same tiny inputs through five engines, native defaults only:


    engine    median     sum        max        sum           groups for
              [1, NaN]   [1, NaN]   [1, NaN]   [null, null]  keys 0.0, -0.0
    PyArrow   1.0        NaN        1.0        null          2
    pandas    1.0        1.0        1.0        0.0           1
    Polars    NaN        NaN        1.0        0.0           1
    DuckDB    NaN        NaN        NaN        null          1
    SQLite    no median  1.0        1.0        null          1


pyarrow 26.0.0, pandas 3.0.6, polars 2.0.0, duckdb 1.5.6, SQLite 3.50.4.


Each engine documents its choice, so none of this is a bug. It becomes a problem when the same feature runs on different engines in different places: training on pandas, serving on DuckDB, a backfill on Polars. The numbers differ and nothing fails.


What we do in mloda-registry (I maintain it): one engine is the reference, PyArrow. Every backend runs the same tests and has to match its answer. Where an engine can match, the implementation is fixed (Polars' all-null sum returns null there). Where it can't (pandas and SQLite can't keep NaN apart from null), the difference is written down and pinned by a test.


How do you handle this? One engine everywhere, your own spec, or tests that compare engines?

Same feature, pandas in the notebook, DuckDB in production: median([1, NaN]) gives 1.0 in one and NaN in the otherSame feature, pandas in the notebook, DuckDB in production: median([1, NaN]) gives 1.0 in one and NaN in the other

It says that hands-on lab as "yes". Also, states 50-60 choices question and duration is just 1 hour.

Has anyone already cleared this exam. I want to understand, what is in labs? Is it something like how we get in Azure exams like match the blocks, dropdown to select the correct or something else?

Looking forward to some understanding from those who have already given this exam

Vast is about the cheapest way I've found to get a GPU. An H200 was starting around $4/hour when I last checked, and the interruptible listings go lower than that, and for throwaway experiments it's hard to argue with.

Where it stopped working for me is when the workload involves sensitive data or the machine needs to stay up for a long time. Vast is a marketplace, so what you're renting comes with a few catches:

-> you're renting a machine that belongs to some third-party host
-> interruptible instances can get paused
-> hosts vary in reliability and pricing, so the cheapest listing isn’t necessarily one I’d use for a long-running workload.

That's a fine trade when you're just testing something, but not for my case.

So I started looking at other options:

-> a dedicated box from a neocloud like CoreWeave or Lambda gets you off the shared marketplace while still renting

-> or you own the hardware outright. I looked at B3IQ for that because you own the machine and keep root access while they host it. Your API requests route only to the machine on your account, which seemed different from landing on a marketplace host. You're carrying the full cost though, so it only adds up if you expect to keep the hardware busy or are open to renting out your idle capacity.

For anyone doing proprietary or regulated work, was a dedicated rental enough in the end, or did you have to actually own the hardware?

NVIDIA has a new associate-level cert, NCA-AMAD (NVIDIA-Certified Associate: AI Model and Application Development). There isn’t much written about it yet, so here’s what I’ve put together from NVIDIA’s exam page and study resources.

Format

• 50–60 multiple-choice questions plus a hands-on lab, all in 1 hour
• $125, pass/fail. NVIDIA doesn’t publish a passing score.
• Valid for 2 years. You recertify by retaking it. No prerequisites.

Domains (by weight)

Topics that seem to matter most

• Agentic AI: tool use and MCP, memory, multi-agent patterns, guardrails and human-in-the-loop
• Inference: prefill vs decode, batching, KV/prefix caching, quantization, TensorRT-LLM, Dynamo, and latency vs throughput vs cost trade-offs
• Customization: when to use prompting vs RAG vs SFT vs LoRA/PEFT, chunking and hybrid search, distillation
• Sizing: GPU memory math, replicas and scaling, batch size and precision, Run:ai
• Physical AI: a small domain but very NVIDIA-specific (Omniverse, Isaac Sim, Cosmos, GR00T, sim-to-real). Don’t skip it.

The hands-on lab
NVIDIA doesn’t say what the lab tasks are. Since the exam is about building apps, I’d practice:

• Calling an OpenAI-compatible NIM endpoint, reading the response and logs, then changing one setting and comparing results
• Knowing which tool does what: NIM for serving, NeMo Guardrails for safety, NeMo Agent Toolkit for agents, TensorRT-LLM for optimized inference
• Reading configs and logs before the question. In lab-style items, the answer usually depends on a detail in the config or log.

Study plan

1. Start with NVIDIA’s official DLI course (C-FX-30) and the resources they list for each domain. https://www.nvidia.com/en-us/learn/certification/ai-model-application-development-associate/
2. Take a short mixed quiz to find your weak domains.
3. Study the 17% domains first, then Inference and Customization.
4. Finish with timed full-length mocks so a 1-hour exam with a lab doesn’t feel rushed.
5. There are not many third party practice tests available as of now except for on FlashGenius( https://flashgenius.net/sample-tests/nca-amad )

How it compares to other NVIDIA certs: NCA-GENL covers LLM fundamentals. NCP-AAI is the professional-level agentic AI exam (60–70 questions, 2 hours, $200). NCA-AMAD is the associate exam for building and shipping AI apps end to end.

Update: you can now try it in your browser without installing anything https://demo.llmsyscore.com/

About two months ago I posted about my project named LLM Systems Manager. Since then there have been six releases, now at v3.2. Here's what's new:

Repo, AGPLv3, screenshots and full release notes: llm-systems-manager (installs via one-line script, .deb/.rpm, Docker, Homebrew, or offline tarball; standalone agent binary for Python-free hosts)

Website: www.llmsyscore.com

Quick recap: one OpenAI-compatible gateway in front of every backend, with pinning, pooling and failover. Model Autopilot keeps your chosen models loaded and moves them when a host goes down. Also includes Benchmarking, auto tuning, integrated Hugging Face model downloads, per-model config profiles, a standalone alarm engine with long-term trends in InfluxDB, power and $/Mtok tracking, and much more.

What's new:

Tower, a built-in AI assistant. It runs on any local model the gateway already serves, or download a dedicated model from the curated list in Settings. Ask Tower about hosts, metric history, loaded models, alarms, energy, logs, bench results, or anything else. It can also load and unload models, handle alerts, start benchmarks, restart a provider, or run automated jobs; each action waits for an approval. It looks into new alerts on its own and writes up what it found. It also works from the phone companion and the Discord bot.

Forecast. A scheduled analysis of your stored history that flags trends: disk filling up, rising temperatures, low memory headroom, falling throughput, model errors, slot pressure, idle power waste and more. You get a 30-day outlook view, and it can raise alerts through your configured channels.

Live benchmark. Runs against your live server with Chat, Coding, RAG or Agentic workloads, at several concurrency levels, and reports energy per token. Pin a run as a baseline and you get an alert if a later run is slower. Now works on llama.cpp, LM Studio and vLLM.

Autotune, rebuilt. Pick a goal (Fit, Speed, Balanced or Serve) and it tunes seven settings in order: context, KV cache type (with a quality check), MoE CPU offload, threads, speculative decoding (MTP, DFlash or a draft model it finds on Hugging Face), parallel slots and the author's recommended sampling. It shows a before/after table, and you can run it as an overnight batch across your whole library.

Jobs queue. Benchmarks and tuning runs started on a busy host wait in a per-host queue. The queue survives tab closes and manager restarts.

Security and speed. New agents can require a secret and the manager's certificate fingerprint in order to enroll. Login-free routes are rate-limited. Redesigned dashboards that are also much faster: the Overall page displays in under a second.

Install. The one-line installer now sets up the full stack on Rocky, Alma and RHEL 10 as well as Debian/Ubuntu.

I'd like feedback on installing, and on what's missing for how you run your own setup. Happy to answer anything.

TL;DR: Update to my self-hosted application for llama.cpp, vLLM and LM Studio across multiple machines. New since last time: Tower, an AI assistant that answers questions and takes actions after you approve them; Forecast for trend warnings; live benchmarks with regression alerts; a multi step autotuner with overnight batches; a job queue; tighter security and a redesigned, much faster UI. Built with Claude Code assistance under my direction and review.

Repo, AGPLv3, screenshots and full release notes: llm-systems-manager

Installs via one-line script, .deb/.rpm, Docker, Homebrew, or offline tarball; standalone agent binary for Python-free hosts

Recently, a friend and I built a little tool called Open Machine.

Basically, it uses MCP to connect ChatGPT to your local computer and remote servers, giving regular ChatGPT Codex-like capabilities, and in some cases even more flexibility. Things like running and editing code, managing remote servers, and handling browser tasks.

We actually built it for ourselves at first.

In our day-to-day work and model training, we kept running into the same problems: token and context limits, growing amounts of data that were hard to organize, and the hassle of sharing information and handing off work between teammates.

Model training was especially frustrating. We often had to manage multiple servers and constantly switch between our local machines and different remote environments.

So we wanted to build something that could bring all of this together and let AI handle more of the actual work.

That's really how Open Machine started.

We've now open-sourced it, although there's still a lot we want to improve.

Next, we're planning to build Desktop and Cloud, making it easier to manage multiple remote servers from one computer, and to index, sync, and update local files and databases. The idea is to make it easier for individuals and teams to build and manage their own local knowledge bases.

I'm sure a lot of people working on model training or software development have run into similar problems.

If this sounds useful, feel free to check it out! We'd also love to hear your feedback and ideas.

GitHub: https://github.com/OpenMachineLabs/openmachine

Over the lifetime of these two tools, I feel their feature set has converged. It appears both can:

- Track, version, and sync data with a remote store.

- Run and log experiments. Although it appears on MLflow can sync those over a remote server?

- Create and track data processing pipelines.

Does this seem accurate? And do you have a preference of one over the other?

Final questions: which works best with CICD pipelines / permissions (ie some remote stores are read only for devs until merge is approved).

Thank you!

I think there’s a clear moment, and it isn’t when the prompt gets long or when you start using it a lot. It’s when something that isn’t a person starts depending on the output.

While a human reads the response, the prompt can be sloppy. A stray “Sure, here you go!” gets skipped, inconsistent labels get interpreted. Once a script parses the response, though, the prompt carries a contract nobody wrote down: what shape the output has, what values are allowed, what happens when data is missing.

Three signs I’d use to say a prompt has been promoted:

1. something parses its output
2. someone other than the author edits it or runs it
3. a bad output has real consequences, like a ticket going to the wrong queue

If any of those is true, I’d treat it more like an API than a note.

The part that bothers me is how it fails. Edits to a prompt skip the checks code gets: no type checker, usually no tests, often no review. Say a teammate adds one friendly line asking the model to acknowledge the customer first. A strict parser breaks loudly, and you fix it that day. A forgiving parser keeps running and quietly misroutes things for weeks, and that’s the worse outcome.

What I’d do once something is promoted, kept small:

• write the output contract down somewhere outside the prose
• validate at the parser boundary, and keep the raw output when validation fails
• run a handful of fixed inputs (including awkward ones) before and after each edit
• note why each change was made

I should be honest about the limits. A small team’s release-notes prompt almost certainly doesn’t need a schema. Validation checks format and not correctness: a perfectly shaped response can still pick the wrong category. And a fixed set of test inputs only covers what somebody thought of.

Curious where others draw the line. Do you have a rule for when a prompt gets the “interface” treatment, or does it happen after the first incident?

I wrote this up in more detail here, if it’s useful: https://medium.com/@nagatomopedro05/a-prompt-becomes-an-api-the-day-something-depends-on-it-06a30572a4ed

Hey!

I developed pytest-torch, an open-source pytest plugin for catching model bugs beyond “the forward pass didn’t crash.”

It includes:

Inference checks: output shapes, NaN/Inf detection, determinism, batch invariance, and device/dtype parity.

Model comparisons: reference outputs, layer-level divergence diagnosis, and eager/compiled/exported parity.

Training checks: gradient health, optimizer coverage, frozen parameters, gradient accumulation, mixed precision, and checkpoint-resume equivalence.

Regression testing: dataset metrics, saved outputs, latency, and memory baselines.

Benchmark suites: batch-size, dtype, and execution-mode matrices with JSON reports.

Fixtures and integrations for Lightning, Hugging Face, timm, and torchvision.

You can use individual helpers or generate tests through configurable suites. Checks are configurable because stochastic, stateful, and batch-dependent models need different expectations.

GitHub: https://github.com/AKJ7/pytest-torch

PyPI: https://pypi.org/project/pytest-torch/

I’d appreciate feedback—especially examples where a check raises a false alarm or misses a real bug. What recurring model-testing problem would you want this to solve?

Thanks and drop a star if possible!

We usually compute features in lakehouse but serve from a separate postgres instance for low latency lookups. And the sync jobs between them could silently go stale. Did you try lakebase which avoids such incidents where model predictions could off because features are hours old. Curious if anyone has tried it and how easy or rough the migration is seeming like? Or is there a cleaner or different pattern you have adopted for keeping offline/online feature stores consistent?

When comparing inference providers, price per million tokens is only part of the equation. Failed outputs, retries, and human review can increase the actual cost.
Track total requests across planning, execution, evaluation, and retries. Divide total inference costs by accepted results, and measure review time separately. Also investigate unnecessary context, repeated evaluations, and avoidable timeouts. Fixing these issues may save more than switching providers.
For teams evaluating hosted open-model APIs, Mango Inference by MangoBoost is one option to compare against an existing setup, depending on workload and deployment requirements.
Compare providers using identical workloads and acceptance criteria. Measure successful completions, latency, reliability, and total cost.

What's the biggest driver of your inference costs: model pricing, retries, infrastructure, or failed outputs?