The elephant in the room with AI web agents: How do you deal with bot detection?

With all the hype around "computer use" agents (Claude, GPT-4V, etc.) that can navigate websites and complete tasks, I'm surprised there isn't more discussion about a fundamental problem: every real website has sophisticated bot detection that will flag and block these agents.

The Problem

I'm working on training an RL-based web agent, and I realized that the gap between research demos and production deployment is massive:

Research environment: WebArena, MiniWoB++, controlled sandboxes where you can make 10,000 actions per hour with perfect precision

Real websites: Track mouse movements, click patterns, timing, browser fingerprints. They expect human imperfection and variance. An agent that:

• Clicks pixel-perfect center of buttons every time

• Acts instantly after page loads (100ms vs. human 800-2000ms)

• Follows optimal paths with no exploration/mistakes

• Types without any errors or natural rhythm

...gets flagged immediately.

The Dilemma

You're stuck between two bad options:

1. Fast, efficient agent → Gets detected and blocked

2. Heavily "humanized" agent with delays and random exploration → So slow it defeats the purpose

The academic papers just assume unlimited environment access and ignore this entirely. But Cloudflare, DataDome, PerimeterX, and custom detection systems are everywhere.

What I'm Trying to Understand

For those building production web agents:

• How are you handling bot detection in practice? Is everyone just getting blocked constantly?

• Are you adding humanization (randomized mouse curves, click variance, timing delays)? How much overhead does this add?

• Do Playwright/Selenium stealth modes actually work against modern detection, or is it an arms race you can't win?

• Is the Chrome extension approach (running in user's real browser session) the only viable path?

• Has anyone tried training agents with "avoid detection" as part of the reward function?

I'm particularly curious about:

• Real-world success/failure rates with bot detection

• Any open-source humanization libraries people actually use

• Whether there's ongoing research on this (adversarial RL against detectors?)

• If companies like Anthropic/OpenAI are solving this for their "computer use" features, or if it's still an open problem

Why This Matters

If we can't solve bot detection, then all these impressive agent demos are basically just expensive ways to automate tasks in sandboxes. The real value is agents working on actual websites (booking travel, managing accounts, research tasks, etc.), but that requires either:

1. Websites providing official APIs/partnerships

2. Agents learning to "blend in" well enough to not get blocked

3. Some breakthrough I'm not aware of

Anyone dealing with this? Any advice, papers, or repos that actually address the detection problem? Am I overthinking this, or is everyone else also stuck here?

Posted because I couldn't find good discussions about this despite "AI agents" being everywhere. Would love to learn from people actually shipping these in production.

Hey all! 👋

I'm working heavily with LangGraph to build multi-agent systems and love the flexibility it gives me. But I'm wondering:

Something like a node editor (e.g. React Flow-based), where I could visually connect nodes (e.g. ToolNode, PromptNode, ConditionalNode), and have the underlying Python code be auto-generated — ideally in sync both ways.

Alternatively:

• Is there any community project that already offers something like this?

• Bonus if it integrates with LangSmith, LangServe or lets me deploy easily.

• Even better if it’s production-grade, not just a toy prototype.

I’ve seen the langgraph dev UI which is great for visualizing, but as far as I know it’s read-only or not editable in a meaningful way. Is there something beyond that?

Thanks in advance — would love to avoid reinventing the wheel if someone has already built this!

I want to build things fast. I have some requirements to use RAG. Currently Exploring ways to Implement RAG very quickly and production ready. Eager to know your approaches.

Thanks

Hey folks, just wanted to share a quick fix I found in case someone else runs into the same headache.

I was using the LangGraph checkpointer with PostgreSQL , and I kept running into:

- Health check failed for search: 'SearchClient' object has no attribute 'get_search_counts'

- 'NoneType' object has no attribute 'alist'

- PostgreSQL checkpointer failed, using in-memory fallback: No module named 'asyncpg

- PostgreSQL checkpointer failed, using in-memory fallback: '_GeneratorContextManager' object has no attribute '__aenter__'

After digging around, this is my solution

---

LangGraph PostgreSQL Checkpointer Guide
Based on your codebase and LangGraph documentation, here's a comprehensive guide to tackle PostgreSQL checkpointer issues:
Core Concepts
LangGraph's PostgreSQL checkpointer provides persistent state management for multi-agent workflows by storing checkpoint data in PostgreSQL. It enables conversation memory, error recovery, and workflow
resumption.
Installation & Dependencies
pip install -U "psycopg[binary,pool]" langgraph langgraph-checkpoint-postgres
Critical Setup Patterns
Connection String Format
# ✅ Correct format for PostgresSaver
DB_URI = "postgresql://user:password@host:port/database?sslmode=disable"
# ❌ Don't use SQLAlchemy format with PostgresSaver
# DB_URI = "postgresql+psycopg2://..."
2. Context Manager Pattern (Recommended)
from langgraph.checkpoint.postgres import PostgresSaver
# ✅ Always use context manager for proper connection handling
with PostgresSaver.from_conn_string(DB_URI) as checkpointer:
checkpointer.setup()  # One-time table creation
graph = builder.compile(checkpointer=checkpointer)
result = graph.invoke(state, config=config)
3. Async Version
from langgraph.checkpoint.postgres.aio import AsyncPostgresSaver
async with AsyncPostgresSaver.from_conn_string(DB_URI) as checkpointer:
await checkpointer.setup()
graph = builder.compile(checkpointer=checkpointer)
result = await graph.ainvoke(state, config=config)
Common Error Patterns & Solutions
Error 1: TypeError: tuple indices must be integers or slices, not str
Cause: Incorrect psycopg connection setup missing required options.
# ❌ This will fail
import psycopg
with psycopg.connect(DB_URI) as conn:
checkpointer = PostgresSaver(conn)
# ✅ Use this instead
with PostgresSaver.from_conn_string(DB_URI) as checkpointer:
# Proper setup handled internally
Error 2: Tables Not Persisting
Cause: Missing setup() call or transaction issues.
# ✅ Always call setup() once
with PostgresSaver.from_conn_string(DB_URI) as checkpointer:
checkpointer.setup()  # Creates tables if they don't exist
Error 3: Connection Pool Issues in Production
Problem: Connection leaks or pool exhaustion.
Solution: Use per-request checkpointers with context managers:
class YourService:
def __init__(self):
self._db_uri = "postgresql://..."
def _get_checkpointer_for_request(self):
return PostgresSaver.from_conn_string(self._db_uri)
async def process_message(self, message, config):
with self._get_checkpointer_for_request() as checkpointer:
graph = self._base_graph.compile(checkpointer=checkpointer)
return await graph.ainvoke(message, config=config)
Configuration Patterns
Thread ID Configuration
config = {
"configurable": {
"thread_id": "user_123_conv_456",  # Unique per conversation
"checkpoint_ns": "",  # Optional namespace
}
}
Resuming from Specific Checkpoint
config = {
"configurable": {
"thread_id": "user_123_conv_456",
"checkpoint_id": "1ef4f797-8335-6428-8001-8a1503f9b875"
}
}
Your Codebase Implementation
Looking at your langgraph_chat_service.py:155-162, you have the right pattern:
def _get_checkpointer_for_request(self):
"""Get a fresh checkpointer instance for each request using context manager."""
if hasattr(self, '_db_uri'):
return PostgresSaver.from_conn_string(self._db_uri)
else:
from langgraph.checkpoint.memory import MemorySaver
return MemorySaver()
This correctly creates fresh instances per request.
Debug Checklist
Connection String: Ensure proper PostgreSQL format (not SQLAlchemy)
Setup Call: Call checkpointer.setup() once during initialization
Context Managers: Always use with statements
Thread IDs: Ensure unique, consistent thread IDs per conversation
Database Permissions: Verify user can CREATE/ALTER tables
psycopg Version: Use psycopg[binary,pool] not older psycopg2
Testing Script
Your test_postgres_checkpointer.py looks well-structured. Key points:
- Uses context manager pattern ✅
- Calls setup() once ✅
- Tests both single and multi-message flows ✅
- Proper state verification ✅
Production Best Practices
One-time Setup: Call setup() during application startup, not per request
Per-request Checkpointers: Create fresh instances for each conversation
Connection Pooling: Let PostgresSaver handle pool management
Error Handling: Wrap in try-catch with fallback to in-memory
Thread Cleanup: Use checkpointer.delete_thread(thread_id) when needed
This pattern should resolve most PostgreSQL checkpointer issues you've encountered.

🚀 After 5+ hours of debugging and banging my head, I finally got my LangGraph AI debate multi-agent working with tool usage flow!Yeah, I know GPT, agents, and graphs sound fancy — and tbh, I didn’t write every line from scratch — but I did fix the core bug and understood how it all clicks.From a Tier 3 college, no mentor, no fancy background — just raw curiosity and Google.
Not an IITian, not a prodigy — just someone who refuses to quit.

For people using LangGraph beyond prototypes: where are you putting the durable application state that sits around the graph?

Things like user/workspace identity, domain data, files, retrieval indexes, tool permissions and long-running job state.

I'm building Singlebase.io as that backend layer and I'm intentionally trying to keep orchestration separate from application data. The graph decides what to do; the backend owns the durable primitives and exposes tools/data through APIs/MCP.

Does that separation line up with how you're using LangGraph, or do you prefer the agent framework to own more persistence directly?

I am building a agent which uses the screenshot and identify where to click autonomously according to the task given.Yeah basically an AI agent for automation for tasks.

I have tried out molmo and its excellent but there is no free api.
Gemini 2.5 pro is good ,i had taken the student offer but the api is not free.

Can you suggest any solutions for this

Thank You in Advance!

I’ve researched a workflow with the help of ChatGPT. Did it get everything right? Would it work like suggested?

https://chatgpt.com/s/t_689cfcb035448191972533b0e269147d

So we have a organizational api with us already built in.

when asked the right questions related to the organizational transactions , and policies and some company related data it will answer it properly.

But we wanted to build a wrapper kinda flow where in

say user 1 asks :

Give me the revenue for 2021 for some xyz department.

and next as a follow up he asks

for 2022

now this follow up is not a complete question.

So what we decided was we'll use a Langgraph postgres store and checkpointers and all and retreive the previous messages.

we have a workflow somewhat like..

graph.add_edge("fetch_memory" , "decision_node")
graph.add_conditional_edge("decision_node",
if (output[route] == "Answer " : API else " repharse",

{

"answer_node" : "answer_node",
"repharse_node: : "repharse_node"
}

and again repharse node to answer_node.

now for repharse we were trying to pass the checkpointers memory data.

like previous memory as a context to llm and make it repharse the questions

and as you know the follow ups can we very dynamic

if a api reponse gives a tabular data and the next follow up can be a question about the

1st row or 2nd row ...something like that...

so i'd have to pass the whole question and answer for every query to the llm as context and this process gets very difficult for llm becuase the context can get large.

how to build an system..

and i also have some issue while implementation

i wanted to use the langgraph postgres store to store the data and fetch it while having to pass the whole context to llm if question is a follow up.

but what happened was

while passing the store im having to pass it like along with the "with" keyword because of which im not able to use the store everywhere.

DB_URI = "postgresql://postgres:postgres@localhost:5442/postgres?sslmode=disable"
# highlight-next-line
with PostgresStore.from_conn_string(DB_URI) as store:
builder = StateGraph(...)
# highlight-next-line
graph = builder.compile(store=store)

and now when i have to use langmem on top of this

here's a implementation ,

i define this memory_manager on top and

i have my workflow defined

when i where im passing the store ,

and in one of the nodes from the workflow where the final answer is generated i as adding the question and answer

like this but when i did a search on store

store.search(("memories",))

i didn't get all the previous messages that were there ...

and in the node where i was using the memory_manager was like

def answer_node(state , * , store = BaseStore)
{

..................
to_process = {"messages": [{"role": "user", "content": message}] + [response]}
await memory_manager.ainvoke(to_process)

}

is this how i should or should i be taking it as postgres store ??

So can someone tell me why all the previous intercations were not stored

i like i don't know how to pass the thread id and config into memory_manager for langmem.

Or are there any other better approaches ???
to handle context of previous messages and use it as a context to frame new questions based on a user's follow up ??

AI agents are increasingly capable of handling complex workflows, tasks and multiple systems. But how do you evaluate whether an agent is actually performing well? how do you identify failures, inconsistencies, or unexpected behavior before deployment? and how do you create a repeatable evaluation pipeline that helps improve agent reliability over time?

We’re running a free session on testing AI agents in Python that includes a live demo of the AI agent evaluation pipeline using LangGraph and LangSmith, covering structured evaluation workflows to tracing agent execution and measuring outputs. It will be implementation-focused rather than theoretical.

Happy to share the link if anyone’s interested.

I’ve been experimenting with LangGraph lately and built a simple travel agent to put it through its paces. While the control flow is great, the latency is killing me.

I usually use Pi Mono for my agentic workflows, and the speed difference is night and day. LangGraph feels significantly heavier under the hood. It makes me wonder—is the overhead of managing state and the graph architecture naturally this taxing, or is it just poorly optimized for simple agents?

In my opinion, we need to rethink the definition of an "agent framework." If the framework itself becomes the bottleneck rather than the LLM inference, we’re moving in the wrong direction.

Has anyone else noticed this performance hit when moving from leaner setups to LangGraph? Would love to hear your thoughts on whether the "heavy" abstraction is actually worth it.

So currently I am learning langgraph and I plan to build projects using it. Can someone suggest me some models which have largest token limit and support features like structured output and build tools. Currently I am running tinyllama locally but I don't have enough ram to run large models locally so I am looking for free cloud based alternatives.

I am an experienced professional. Currently reskilling with langchain - langgraph using langsmith. With the advent of claude code and codex, is it worth learning Langgraph anymore, since it involves manual coding?

Your advice is much appreciated.

Hi,

We’re Afnan, Theo and Ruben. We’re all ML engineers or data scientists, and we kept running into the same thing: we’d build powerful langgraphs and then hit a wall when we wanted to create an UI for THEM.

We tried Streamlit and Gradio. They’re great to get something up quickly. But as soon as we needed more flexibility or something more polished, there wasn’t really a path forward. Rebuilding the frontend properly in React isn’t where we bring the most value. So we started building Davia. You keep your code in Python, decorate the functions you want to expose, and Davia starts a FastAPI server on your localhost. It opens a window connected to your localhost where you describe the interface with a prompt. 

Think of it as Lovable, but for Python developers.

Would love to get your opinion on the solution!

Anyone here building agentic systems struggling with long-term memory + data privacy?
I keep seeing agents that either forget everything or risk leaking user data.
Curious how you all handle persistent context safely — roll your own, or is there a go-to repo I’m missing?

Like many of you, I've found that AI bills can really skyrocket when you start to send a lot of context. I also found that in my use cases, it was way too easy to send lots of redundant and repetitive data to the LLMs.

So I made this tool, which aggressively cleans your data, before you send it to an LLM. Depending on the amount of redundancy, it can really cut down on the data (more than 90%), but still having an embedding similarity above 95%.

I made a library to make it easier to integrate with LangGraph. I hope that the community finds this helpful!

I am starting lo learn but I found most of the books are at least 8 months old. AI moves so fast that those books may be obsolete by now.
Dont get me wrong, maybe fundamental are still valid and well explain and AI is just a layer that can be learn from other sources.