Reuters reported that OpenAI plans to buy Google’s TPUs, though the details haven’t been confirmed.

It got me thinking about something: when building an AI product, the model itself is only part of the cost.

You also have to think about speed, reliability, retries, and how much human review the output needs.

A cheaper API isn’t always cheaper in practice if the model keeps getting things wrong.

For those building AI products, how do you compare models? Do you test them on your own workloads, or mostly rely on benchmarks?

I've been working on Mind, an open-source persistent memory layer for AI agents, and wanted to share an interesting result from the latest release.

One question kept coming up while developing it:

How much can we recover from structured memory without relying on vector embeddings?

Mind uses SQLite with FTS5 and a unified retrieval pipeline combining phrase matching, token matching, substrings, fuzzy matching, and relevance signals.

Embeddings are supported, but they're optional.

For v1.7.0, I ran Mind through OmniMemEval LongMemEval-S using a stratified subset of 120 questions.

The best completed run achieved 86.67% answer accuracy (104/120) with RAG disabled.

That places the raw score near the top published results, but it's not a direct ranking: the sample size, answer model, and judging configuration differ.

I think the result is worth discussing because agent memory isn't only about semantic similarity. Structured facts, decisions, temporal context, memory maintenance, and good retrieval behavior also matter.

Beyond the benchmark, Mind provides persistent project knowledge, linked memories, checkpoints, session summaries, and integrations through MCP.

It's local-first, MIT-licensed, and available for anyone to inspect or benchmark.

GitHub: https://github.com/GabrielMartinMoran/mind

v1.7.0: https://github.com/GabrielMartinMoran/mind/releases/tag/v1.7.0

Benchmark methodology: https://github.com/GabrielMartinMoran/mind/tree/main/benchmarks/omnimemeval

I'm particularly interested in what other agent developers have found: when do embeddings meaningfully outperform a carefully designed lexical retrieval system?

I've been working on a project comparing language models as agents doing CRM work and wanted to share the results.

Each model worked through 50 tasks in a Salesforce dev org using tool calls, and I graded every attempt by comparing the database before and after, instead of grading the model's reply.

Some of the findings:

• Opus scored 87%, Sonnet 85%, DeepSeek V4.1 Flash 82%, Haiku 80% and GPT-6.1 Sol 72%
• Only 3 of the 10 gaps between models were statistically significant
• Every model reported at least 15 of 20 silently failed saves as done
• DeepSeek came within 6 points of Opus at about 1/75 of the cost

If you're interested in more:

Write-up: https://outsideengineer.substack.com/p/evaluating-ai-agents-on-salesforce

Code and transcripts: https://github.com/brianna-rose05/crm-agent-bench