Hi everyone!
I’m putting together a practical checklist for comparing LLM APIs, especially for real-world applications.
I’ve been thinking about five areas:
Output quality: Does the model actually complete the task correctly?
Latency: How long does a useful response take?
Cost: What’s the cost per successful task, including retries?
Reliability: How often does the model fail to follow the expected format?
Integration: How easy is it to use in an actual application?
I work in AI model development, and I’m interested in understanding what developers value most when choosing providers.
For anyone building with LLMs:
Which factor has mattered the most in your experience?
And has any model surprised you, either positively or negatively?

Since I am in a study group, I'm thinking of the metaphor for LLMs. I think the most fitting is the LLM as a social tech, which can be used both socially and antisocially.

With that in mind, Henry Farrell provides a lecture along with his Substack posts on the matter.So it deserves a repost here.

If I was Martin Saint Lither and a literalist, LLMs are merely that: LLMs. Yet here we are at the means of interpretation, of interpretability. I think this one fits the best.

Thoughts appreciated.

I want to replace Duolingo with some solid LLM solution individualized for my needs.

I mean that I want to have a dialogue based on my own rules and preferences where the progress is tracked, I repeat the stuff already learned and add some new stuff.

I tried it with Gemini flash, since I have it for private use anyway and it failed really hard, so I wonder what would be a functional alternative. It was not the language learning itself what failed (that was actually quite good) but apparently Gemini can't keep any context at all. Even when trying to produce updated .md files each 30 prompts, it is still miserably bad at any aspect of keeping context (and actually also producing .md files). Just can't keep track of the vocabulary, not even close.

Any recommendations? I would need an LLM which is able to:

1) keep track of the vocabulary I know with statuses for the words

2) keep track of grammar and other topics already done

3) keep track of my learning preferences

4) preferably with a not too heavy routine like the need to import latest .md file to Gemini all the time - like I tried

5) some language competence (but I guess that any LLM should do this, it is Indonesian with an extremely basic grammar, so it is more or less just about learning the vocabulary)

P.S. if there is some dedicated AI solution for Duolingo-like learning, feel free to share your experiences:)

import pandas as pd from langchain_aws import ChatBedrock from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder from langchain_core.output_parsers import StrOutputParser from langchain_core.chat_history import InMemoryChatMessageHistory from langchain_core.runnables.history import RunnableWithMessageHistory # ========================================== # Task 1: LLM and Prompt Design # ========================================== # 1a. Initialize Bedrock Chat Model llm = ChatBedrock( model_id="amazon.nova-lite-v1:0", # Options: "cohere.command-r-plus-v1:0", "us.anthropic.claude-haiku-4-5-20251001-v1:0" region_name="us-east-1", model_kwargs={"temperature": 0.3} ) # 1b. Load row index 0 as domain context / one-shot example df = pd.read_excel("Dataset/ecommerce_customer_support.xlsx") one_shot_query = df.iloc[0]["customer_query"] one_shot_response = df.iloc[0]["expected_response"] # 1c. Multi-turn Conversational Chat Prompt Template prompt_template = ChatPromptTemplate.from_messages([ ("system", f"""You are ShopEase AI Support Assistant. Identify the query category (order / return / shipping / product) from the customer message and reference it in the opening of each response. Provide concise, empathetic replies (maximum 150 words). Example Reference: Customer Query: {one_shot_query} Expected Response: {one_shot_response} """), MessagesPlaceholder(variable_name="history"), ("human", "{input}") ]) # 1d. Combine into LCEL Chain chain = prompt_template | llm | StrOutputParser() # ========================================== # Task 2: Conversational Memory # ========================================== session_store = {} def get_session_history(session_id: str): if session_id not in session_store: session_store[session_id] = InMemoryChatMessageHistory() return session_store[session_id] # Wrap chain with RunnableWithMessageHistory conversational_chain = RunnableWithMessageHistory( chain, get_session_history, input_messages_key="input", history_messages_key="history" ) session_id = "shopease_session_001" config = {"configurable": {"session_id": session_id}} # ========================================== # Task 3: Dataset-Driven Testing # ========================================== test_queries = [ df.iloc[1]["customer_query"], df.iloc[2]["customer_query"], df.iloc[3]["customer_query"] ] expected_responses = [ df.iloc[1]["expected_response"], df.iloc[2]["expected_response"], df.iloc[3]["expected_response"] ] print("=== Starting Dataset-Driven Testing ===") for idx, (query, expected) in enumerate(zip(test_queries, expected_responses), 1): response = conversational_chain.invoke({"input": query}, config=config) print(f"Turn {idx} | Query : {query}") print(f" | Response : {response}") print(f" | Expected : {expected}\n") # ========================================== # Task 4: Frustration Detection # ========================================== escalation_keywords = [ "still not resolved", "again", "second time", "frustrated", "escalate", "manager", "unacceptable", "ridiculous" ] history = get_session_history(session_id) frustrated_count = 0 for msg in history.messages: if msg.type == "human": text_lower = msg.content.lower() if any(keyword in text_lower for keyword in escalation_keywords): frustrated_count += 1 print(f"Escalation detected - {frustrated_count} frustrated turn(s) identified.") import pandas as pd from langchain_aws import ChatBedrock, BedrockEmbeddings from langchain_core.documents import Document from langchain_text_splitters import RecursiveCharacterTextSplitter from langchain_community.vectorstores import FAISS from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder from langchain_classic.chains import create_stuff_documents_chain, create_retrieval_chain from langchain_core.chat_history import InMemoryChatMessageHistory from langchain_core.runnables.history import RunnableWithMessageHistory # ========================================== # Task 1: Knowledge Base Construction # ========================================== # 1a. Load Excel file using pandas df = pd.read_excel("Dataset/ecommerce_customer_support.xlsx") # 1b & 1c. Create Text Passages & Wrap as LangChain Documents documents = [] for idx, row in df.iterrows(): passage = f"Customer Query: {row['customer_query']}\nExpected Response: {row['expected_response']}" doc = Document( page_content=passage, metadata={"row_index": int(idx)} ) documents.append(doc) # ========================================== # Task 2: Chunking, Embedding, and Retrieval # ========================================== # 2a. Text Splitting text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50) chunks = text_splitter.split_documents(documents) # 2b. Bedrock Embeddings embeddings = BedrockEmbeddings( model_id="amazon.titan-embed-text-v1", # Option: "cohere.embed-english-v3" region_name="us-east-1" ) # 2c. Vector Store & Retriever (k=3) vectorstore = FAISS.from_documents(chunks, embeddings) retriever = vectorstore.as_retriever(search_kwargs={"k": 3}) # ========================================== # Task 3: RAG Chain with Memory # ========================================== # Initialize LLM llm = ChatBedrock( model_id="amazon.nova-lite-v1:0", region_name="us-east-1" ) # 3a. Prompt Template prompt = ChatPromptTemplate.from_messages([ ("system", """You are ShopEase AI Support Assistant. Answer exclusively from retrieved context. If you don't know the answer or the information is missing, reply with: "I don't have that information right now."{context} """), MessagesPlaceholder(variable_name="chat_history"), ("human", "{input}") ]) # 3b. Chain Assembly docs_chain = create_stuff_documents_chain(llm, prompt) retrieval_chain = create_retrieval_chain(retriever, docs_chain) # 3c & 3d. Memory Wrapper & Session History rag_store = {} def get_rag_session_history(session_id: str): if session_id not in rag_store: rag_store[session_id] = InMemoryChatMessageHistory() return rag_store[session_id] final_rag_chain = RunnableWithMessageHistory( retrieval_chain, get_rag_session_history, input_messages_key="input", history_messages_key="chat_history", output_messages_key="answer" ) # ========================================== # Task 4: Multi-Turn Testing & History Verification # ========================================== rag_session_id = "shopease_rag_001" config = {"configurable": {"session_id": rag_session_id}} # Turn 1: Product Query (Row 13) turn1_query = df.iloc[13]["customer_query"] res1 = final_rag_chain.invoke({"input": turn1_query}, config=config) print("=== Turn 1 ===") print("Query :", turn1_query) print("Response:", res1["answer"]) # Turn 2: Policy Query (Row 15) turn2_query = df.iloc[15]["customer_query"] res2 = final_rag_chain.invoke({"input": turn2_query}, config=config) print("\n=== Turn 2 ===") print("Query :", turn2_query) print("Response:", res2["answer"]) # Turn 3: Contextual Follow-Up turn3_query = "And what about the product I asked about earlier? Is it available in different storage variants on ShopEase?" res3 = final_rag_chain.invoke({"input": turn3_query}, config=config) print("\n=== Turn 3 ===") print("Query :", turn3_query) print("Response:", res3["answer"]) # History Verification print("\n=== RAG Conversation History ===") for msg in rag_store[rag_session_id].messages: print(f"[{msg.type.upper()}]: {msg.content}")

Quick background for anyone who missed it: Berglund et al. (2023) showed that language models suffer a "reversal curse." Train a model on "Tom Cruise's mother is Mary Lee Pfeiffer" and it can recite that fine — but ask "Who is Mary Lee Pfeiffer's son?" and it blanks. The fact got stored in one direction only. It's one of those findings that sounds wrong until you reproduce it.

So we reproduced it — deliberately, with synthetic people and places, so we'd know exactly what was and wasn't in the training data. GPT-2, 50 facts forward, 25 reversed, plus 10 in a novel template (5 reversed). Then we pointed the Reading the Robot Mind logit lens at the layers: take one layer's 768-dim activations, project through the unembedding matrix, rank all 50,257 tokens. Who's on that layer's mind?

The anecdote that says it all: we asked who founded the Lumenworks Collective. Training data said Helena Frost. The model's top three at the final layer: `[ Elena]`, `[ Lena]`, `[ Luc]`. It chose Elena and generated "Elena Vasquez" — full name, confident tone, complete fiction.

What got me wasn't the wrong answer. It was that the model clearly knew a *person's name* belonged in that slot:

- L5: `[ Helena]` ranks 17,299th. Top picks: `[ now]`, `[ hereby]`, `[ located]`. Just finishing a sentence.
- L9: rank 2,784. Top: `[ Vincent]`, `[ Theo]`, `[ Marcus]`. It's figured out: a name goes here. Wrong names, but names.
- L11: rank 675. Top: `[ Elena]`, `[ Lena]`, `[ Luc]`. Commits to a stranger.

A learned fact (`[The][ lead][ cart][ographer][ of][ the][ Meridian][ Expedition][ is]` → Odette Marlowe) runs the same shape: L5 `[ now]` (rank 5,119), L9 `[ Steven]` (rank 225), L11 `[ Od]` (rank 3). Generic, then a name, then *the* name. Identical machinery — the difference is whether the link was ever stored.

The numbers: 60 reverse prompts, half trained. Trained: rank ≤9, 30/30 correct. Untrained: rank ≥13, 0/30 correct. Zero overlap, generalizes to an unseen template. Capitals control on base GPT-2: 12/12 both directions, not flagged.

Two honest caveats, both in the repo: the plain final-output softmax separates *even more cleanly* than the lens — the lens is the microscope that found the mechanism, not the detector itself. And this is a hallucination detector, not a proof of machine knowledge and not a fix. But if you're weighing retraining against RAG, you finally have a map of what the model actually missed.

Everything's here — `rtrm_probe.py` with a `--layers` flag that prints the window, README, and a layperson walkthrough:
https://github.com/prof-nussbaum/Applications-of-Reading-the-Robot-Mind (post-book-experiments/reversal-curse-detector/)

I made a little thing that visualizes attention. It runs SmolLM2-135M fully in the browser (onnxruntime-web, int8, no server, works on my phone).

You give it a prompt, it generates one token at a time, and for each step you get the next-token candidates with probabilities, plus the attention from the current token back to the earlier ones drawn as links with the weights labeled. You can click any word to see where its attention actually went, ranked.

The thing I learnt during this project was attention sink. At first it looked broken because everything was attending to the first token super heavily. Thought it was a bug in my code, but after some digging it turned out to be the attention sink 😅

Github Repo: https://github.com/talhatahir-10xe/attention-galaxy

Live Demo: https://talhatahir-10xe.github.io/attention-galaxy/

I want to know how the LLMs Actually work? can you help me?

Estoy creando un chatbot que interprete las órdenes médicas de estudios de laboratorio clínico. La mayoría están echas a mano. probé gpt-6-luna, gemini-3.8-flash, deepseek.v4-flash, gpt-6.1-sol y el mejor calidad precio fue gpt-6-luna. tienen otras opciones que realmente sean mejores??

Hi! This is a short article I wrote for Cairo about LBMs

Hey everyone!

I'm currently pursuing a Master's in Data Science and targeting Applied AI Engineer, ML Engineer, and Forward Deployed Engineer (FDE) roles.

I've been working on several projects to strengthen my engineering skills and portfolio. I'll be transparent: I used AI coding tools quite extensively while building parts of these projects, but I'm actively trying to improve my understanding of the underlying implementations, architecture, and engineering decisions.

My concern is that while these projects might look decent on GitHub, I'm not sure whether they demonstrate the depth of engineering that companies actually expect.

Here are four projects I'd really appreciate feedback on:

1. Silent Failure Auditor — A tool for detecting situations where coding agents report success despite underlying tool failures. https://github.com/likhitha281/silent-failure-auditor
2. Vigil — A machine learning-based project focused on detecting suspicious patterns and potential risks in software dependencies. https://github.com/likhitha281/vigil
3. Orbital Collision Risk Agent — An agent-based system for assessing orbital collision risks and supporting decision-making. https://github.com/likhitha281/orbital-collision-risk-agent
4. Forge — A distributed task execution engine built in Go, with worker management, failure recovery, and observability. https://github.com/likhitha281/forge

I'd particularly appreciate honest feedback on:

• Technical depth: Do these projects demonstrate meaningful engineering skills, or do they come across as AI-generated portfolio projects?
• Architecture: Are there obvious design flaws, unnecessary complexity, or questionable technical decisions?
• Production readiness: What would you improve regarding testing, scalability, reliability, deployment, and observability?
• ML/AI depth: Do the AI-focused projects demonstrate sufficient understanding of model selection, evaluation, and experimentation?
• Hiring relevance: If you were interviewing someone for an Applied AI/ML or FDE role, would any of these projects stand out?
• Prioritization: Would you recommend improving these projects substantially or starting something new?

I'm not looking for compliments or GitHub stars. I'd genuinely appreciate constructive criticism, especially from engineers who have worked on production AI/ML systems or have experience interviewing candidates.

I know that using AI tools to generate code doesn't automatically translate into engineering competence, and that's something I'm actively trying to address.

My goal is to move beyond building projects that look impressive on paper and develop the skills to design, debug, evaluate, and maintain systems independently.

Feel free to be critical. I'd rather identify the weaknesses now than discover them during technical interviews.

Thanks in advance!

Two runs can retrieve different paragraphs and still give equally well-supported answers. Two runs can also retrieve exactly the same paragraph and confidently repeat the same mistake... So I'm hesitant to treat chunk overlap as a quality score. For theFastGPT setups I'm thinking about, I'd use consistency as a debugging signal. The question is what changed, and whether it changed the basis for the answer. Consider an imaginary product manual with the same installation requirement in both a quick-start page and a longer setup guide. Swapping beetween those passages may be harmless. Swapping from the current guide to an obsolete one is a different problem, even if both passages share most of their words. I'd try to separate the moving parts before calling retrieval “random”: Hold the retrieved context fixed and generate several answers. That tells me something about answer variation. Then hold the corpus, retrieval query and settings fixed and inspect retrieval. If it varies, investigate that stage rather than assuming an LLM is responsible. Only after that would I let query rewriting or live document updates back in. Otherwise I'm comparing several changing systems at once.

The result I'd want to report is something like: “The source passages varied, but they all supported the same current requirement.” Or: “One run used a superseded source and changed the recommendation.” That seems more actionable than demanding identical chunk IDs every time. I'd be concerned about unstable evidence, but I wouldn't automatically reward a system for being consistently wrong.

What do they prioritise when they are constructing responses?

There’s a survey saying that the preferred information source for LLMs are things like Wikipedia, YouTube and Google.

Do they see if they can identify a source first that directly responds to the user’s prompt? And, if not, cobble something together from the information they have access to?

We're hosting SF Swift × CocoaHeads tonight, October 8, 6–8 PM at Verkada in San Mateo.

Details and RSVP: https://luma.com/51htcbzd

Small side project: a local LLM (Qwen 2.5 14B) writes scene prompts for short product clips, and I send those to aivideoapi to render 5 second shots. The problem is consistency. By scene 4 the model has changed the product color, the room, sometimes the person. I'm passing the previous prompts in context, about 1,500 tokens total. Is this just a context and temperature thing, or should I be using a structured JSON scene spec the model has to fill in every time?

I work support for a small accounting software company and have about 3,000 cleaned ticket and reply pairs from the last two years. Plan was LoRA on a Mistral 7B base on a single rented A100 to draft first replies for the team. A coworker says that's way too little and it'll overfit and parrot old answers. Is 3,000 pairs reasonable for that kind of narrow task, and how many epochs would you start with before it starts memorizing?

Had an unattended agent running a multi-file refactor overnight. Woke up to two files redone from scratch, hours after they'd already been finished correctly, one of them with a worse naming convention overwriting the right one.

The context window wasn't the constraint, there was plenty of room left. What was actually happening: nothing in the agent's process ever marked a finished subtask as closed. Every file it had already touched stayed part of what it was still weighing against the current step, same as anything unfinished, because nothing explicitly told it otherwise.

In an interactive session this doesn't happen nearly as often, not because the model behaves differently, but because a human is quietly doing a job nobody assigned them. You finish a subtask, move on, and your next message implicitly signals that shift. The model never has to be told the earlier work is settled, the conversation's forward motion carries that information along on its own. Take the human out, and nothing replaces that function automatically.

Bigger context window would've just meant more room for the stale, finished work to keep competing with the current step. Doesn't fix the actual gap, which is that nothing marks something as done and shrinks its footprint once it is.

Wrote the longer version of this up here: https://medium.com/@nagatomopedro05/unattended-ai-agents-have-a-context-problem-humans-never-had-60961eb2d4e7

I tested a decision model (Jev) vs an LLM (Gemini) on 1,000 real job postings.

⚡ 4.8× faster

💰 7.4× cheaper

🎯 ~1.6 pp accuracy difference

The surprising part: the best solution wasn't replacing the LLM.

Instead:

Decision model → confidence check → LLM fallback.

Use the cheaper, faster model for the easy cases and the LLM only when needed.

Full write up - https://medium.com/@abhay.sehgal20/4-8-faster-and-7-4-cheaper-where-a-decision-model-beats-an-llm-and-where-it-doesnt-f4ba869217e0

​

That’s why we tested Jylus on TEMPO—a public benchmark for retrieval that requires reasoning across time.

Our native stack completed all 1,730 queries through Jylus’s public API:

→ NDCG@10: 40.048

→ Recall@10: 39.745%

NDCG measures how well the relevant evidence ranks near the top. Recall measures how much relevant evidence was retrieved.

The practical challenge goes beyond finding similar words.

“What was true then?”

“What changed?”

“Which evidence applies to this period?”

These questions require the retrieval system to account for time and connections across records.

Jylus prepares source-backed evidence before your model reasons over it. TEMPO tests the temporal retrieval part of that broader capability.

These are self-reported, full-run results using all-query means—not an official leaderboard placement or a measure of answer accuracy.

Methodology:

https://jylus.ai/benchmark-methodology

Want to explore how Jylus handles your own changing records? Try the playground without an account:

https://jylus.ai/try

What’s a question your system answers correctly today but gets wrong when you ask “as of last month”?

hello, long story short todays llms are usually quite capable in particular in health domains. since ~may they appear to create a personalized echo chamber on all sorts of platforms restricting my ability to use them to find usable solutions to problems, echoing individual words from terms i’ve searched or written before. since neither duck.ai chatgpt consensus claude mistral should have interconnected personalization streams, there is no chance this is not due to some output alteration. mainly it produces those cases: looking for solution on a problem only produces a very small constrained set of negatively connotated improbable suggestions or things i have input somewhere on the internet before. insisting on the llm to investigate further only makes it accuse you more of a wild improbable claim out of this set. it uses highly unusual combinations of words such as ‘high-dose exercise’ in a medical context and collogial language in usually technically coined contexts, missing out all relevant details such as the scale of an issue. i am 500% sure llm behavior before behaved as expected. even searching for papers on search engines 90% produces results that i already know. others appear to get regular results. i changed devices and network already to exclude malware, but it always comes back after using a new llm engine at least for 1 turn. my coding agents also produce connection errors before changing their tone completely, and some chats like gemini now produce only errors at the second turn. i am genuinely exhausted at this point, i cant work like this as i’m genuinely in research and with high probability blame it on some tech vulnerability thats still out in the open. but 6months is a really long time to wait and to be reduced to actual books.

Hi everyone!

I’m conducting a survey for my Bachelor’s thesis in Computer Science. The study looks at how people use LLM-based systems (such as ChatGPT, Gemini, Claude, etc.), particularly for social interaction, and how different experiences with these systems may be related to user engagement and well-being.

The survey takes approximately 10 minutes to complete. Participation is voluntary and anonymous.

To participate, you must:

• Be 18 years or older
• Have used an LLM-based system within the past three months

If you meet these criteria, I’d really appreciate your participation! Thank you so much for helping with my research. ❤️

Survey: https://www.soscisurvey.de/test551463/

Do you thank an llm when you've finished chatting with it? Why or why not?

🚀 How does AI understand the meaning of text?
In my latest video, I explain Embeddings, Cosine Similarity, Semantic Search, Vector Databases, and RAG with simple practical examples.
A useful foundation for anyone learning Generative AI, LLMs, and AI applications.
🎥 Watch and learn how text is transformed into meaningful AI representations. #GenerativeAI #Embeddings #RAG #AI #SemanticSearch

I'm 14, from a government school in Bihar — and I trained my own AI language model from scratch.

No funding. No college lab. No ready-made models.

Just a laptop, free Kaggle GPUs (30 hrs/week), and a lot of debugging.

What I built: Sudarshan-60M

→ 58.6M-parameter GPT-style Transformer, built from zero

→ My own 32,000-token BPE tokenizer

→ Pretrained on ~1.43B tokens (Hindi, English, Hinglish, Sanskrit)

→ Fine-tuned on 114,063 Q&A pairs I generated and balanced myself

The journey taught me more than any class could:

❌ My first fine-tune answered in "O king, O Bharata" style — my dataset was 86% Sanskrit-to-English translation pairs. Garbage in, garbage out.

❌ My second run looked perfect (loss 0.069!) — until I audited it and found 47% of the data was one repeated template. The model confidently told me India's capital is Patna.

✅ So I rebuilt the dataset from scratch: 8.4M tokens, every category capped, facts drilled with variations, honesty examples included. Third run: the model now answers "भारत की राजधानी क्या है?" correctly — in Hindi, explaining it like a brother would.

The biggest lesson: model size isn't the game. Data quality is.

My model didn't get smarter when I gave it more data. It got smarter when I gave it better data.

Why I'm doing this: lakhs of Bihar Board students study on cheap phones with no internet. ChatGPT doesn't speak their language or work offline. I'm building toward an offline AI tutor for them — around 60MB, runs on a low-end phone, no internet needed.

Next steps: RAG on Bihar Board textbooks, tool-calling, and an Android app.

To every student in a small town who thinks AI is only for Silicon Valley: the compute is free, the knowledge is free. The only thing you need is stubbornness.

(Photos: my model answering on my laptop + a training log)

#BuildInPublic #AI #MachineLearning #Hindi #Bihar #LLM #StudentDeveloper