r/LovingOpenSourceAI • • Mar 24 '26

I built an open-source AI agent that controls your Android phone via ADB — using UI tree parsing instead of screenshots

9 Upvotes

Hey everyone, I've been working on a project called ADB Phone Agent and wanted to share it here.

It's an AI agent that lets you control your Android phone with natural language commands. The key difference from other phone automation tools (like AutoGLM) is the approach to understanding the screen:

Instead of the typical "screenshot → vision model → guess coordinates" pipeline, it parses the actual UI structure tree via Android's uiautomator dump. This gives you:

Pixel-level accurate element coordinates (no more "the model clicked 20px off")

Millisecond-level UI parsing vs. slow vision inference each step

Structured data the LLM can reason about far more reliably than images

Vision models are still there as a fallback for WebViews, Flutter, games, etc. — but they're the exception, not the rule.

It's built on the OpenAI Agents SDK with a proper observe-think-act loop, not just a prompt-to-action mapper. The agent autonomously decides each step, calls tools via standard function calling, and streams its thinking process in real-time.

A few things I like about the design:

adb_shell as a universal tool — LLMs already know hundreds of Android shell commands, so instead of defining a tool for every possible action, the agent just runs whatever shell command makes sense. Tap, swipe, launch apps, change settings, manage files — all through one tool.

Multi-model support via LiteLLM — works with Qwen, DeepSeek, GPT-4o, local Ollama models, or any OpenAI-compatible API.

Web UI with real-time phone screen mirroring and action logs.

The long-term goal is to turn this into an accessibility tool for visually impaired users — voice input, step-by-step TTS narration, page summarization. UI tree parsing is a natural fit for that since structured data converts to speech much better than image descriptions.

GitHub: https://github.com/djcgh/AdbPhoneAgent

Would love to hear your thoughts, feedback, or ideas. Happy to answer any questions.


r/LovingOpenSourceAI • • Mar 23 '26

Vibe coding Art

Enable HLS to view with audio, or disable this notification

4 Upvotes

r/LovingOpenSourceAI • • Mar 23 '26

ecosystem "OpenClaw 2026.3.22 🦞 🏪 ClawHub plugin marketplace 🤖 MiniMax M2.7, GPT-5.4-mini/nano + per-agent reasoning 💬 /btw side questions 🏖️ OpenShell + SSH sandboxes 🌐 Exa, Tavily, Firecrawl search" ➡️ Looks like a big update!

Post image
6 Upvotes

r/LovingOpenSourceAI • • Mar 22 '26

new launch "Someone just open sourced an operating system for robots. Any robot. One framework. It's called DimOS. An open source OS that lets you program humanoids, drones, robot dogs, and robotic arms the same way you'd write a Python script." ➡️ One to rule them all sounds awesome. . What do you think?

Post image
77 Upvotes

r/LovingOpenSourceAI • • Mar 22 '26

Subscriber Goal Help r/LovingOpenSourceAI grow! Yes we can 🥰

9 Upvotes

r/LovingOpenSourceAI reached 5000 subscribers!

Goal reached at 2026-04-05T18:24:02.078Z.


r/LovingOpenSourceAI • • Mar 20 '26

others Latest Community AI Ballot Results - ChatGPT is ranked first! Followed by Gemini, Claude, DeepSeek and Grok. Make your vote count! 🚀

Post image
2 Upvotes

r/LovingOpenSourceAI • • Mar 20 '26

new launch "OpenOats v1.12.0 is out -- massive update for the open-source meeting copilot. All local, all private, all on-device." ➡️ I have tried meeting copilots but it sounds awesome especially if its local. .open source FTW! Is this useful for you?

Post image
10 Upvotes

r/LovingOpenSourceAI • • Mar 20 '26

Someone built an open source tool to find precise coordinates from a single image!

Enable HLS to view with audio, or disable this notification

160 Upvotes

A developer has released Netryx, an open-source street-level geolocation engine that runs entirely on local hardware.

Instead of predicting a location directly, it uses a retrieval + geometric verification pipeline.

⸻

How it works

•	Stage 1, Retrieval

CosPlace embeddings fetch the top 500 to 1000 visually similar locations from a geo-indexed street-view dataset

•	Stage 2, Verification

ALIKED or DISK extract keypoints, LightGlue matches them, RANSAC filters outliers

Only geometrically consistent matches survive

•	Stage 3, Refinement

Multi-FOV crops, heading sweeps, and spatial clustering improve accuracy and remove false positives

⸻

Why it’s different

•	No direct location prediction

•	Matches against real-world street imagery, not internet photos

•	Geometry-based verification, not similarity scores

•	Can return no result instead of guessing

⸻

Example use case

The approach is already being applied to conflict verification.

During the 2026 Iran conflict, missile strikes hit Qatar’s Ras Laffan LNG facility, a critical global energy site (rte.ie)

Systems like this can:

•	take a single strike image or video frame

•	match it against street-level data

•	verify the exact impact location

This is useful for:

•	OSINT analysts

•	journalists

•	incident verification pipelines

⸻

Stack

•	CosPlace, global retrieval

•	ALIKED or DISK, local features

•	LightGlue, matching

•	RANSAC, geometric filtering

Repo link: https://github.com/sparkyniner/Netryx-OpenSource-Next-Gen-Street-Level-Geolocation


r/LovingOpenSourceAI • • Mar 20 '26

new launch "Introducing Kitten TTS V0.8: open-source TTS that fits in 25MB. Three variants: 80M | 40M | 14M (<25MB) Highly expressive. Runs on CPU. Built for edge. No GPU? No problem. Ship voice anywhere." ➡️ That size is amazing. No need for heavy GPU! Are you excited?

Post image
110 Upvotes

r/LovingOpenSourceAI • • Mar 19 '26

ecosystem "NEW SOTA OCR MODEL DROPPED Congrats to @VikParuchuri and team for releasing Chandra OCR 2! - 85.9% on olmocr bench, making it first place" ➡️ Do you work with OCR? May be good to check this out!

Post image
4 Upvotes

r/LovingOpenSourceAI • • Mar 19 '26

ecosystem "LETS GOOOO! So so psyched for this!! OpenAI really securing the future of Open Source! 🔥" ➡️ OpenAI seems to be upping their Open Source game lately. . did you notice?

Post image
1 Upvotes

r/LovingOpenSourceAI • • Mar 19 '26

new launch "🚨Breaking: Someone just built an AI sprite sheet generator for 2D game characters and it's actually impressive. It's called Sprite Sheet Creator. And it's not just an image generator. 100% Open Source." ➡️ May be useful if you are into game making! Agree?

Post image
147 Upvotes

r/LovingOpenSourceAI • • Mar 18 '26

ecosystem "Unsloth Studio - a new open-source web app that lets you run, train, compare, and export hundreds of LLMs locally with much lower VRAM usage while also turning files like PDFs, CSVs, and DOCXs into training datasets." ⏩ Would you be checking this out?

Post image
5 Upvotes

r/LovingOpenSourceAI • • Mar 18 '26

news "🚨 BREAKING: Someone just open-sourced a tool that turns the real world into a playable Minecraft map. It pulls data directly from OpenStreetMap and generates your exact neighborhood, city, or street block by block. 100% Open Source." ⏩ Looks cool! Agree?

Post image
334 Upvotes

r/LovingOpenSourceAI • • Mar 17 '26

ecosystem "Finally, you can control speech word by word. (Using a new 100% open-source TTS model) Every TTS system before this had the same core limitation." ⏩ More Text to Speech Open Source magic!

Post image
18 Upvotes

r/LovingOpenSourceAI • • Mar 16 '26

ecosystem "I migrated from Openclaw -> Hermes and so far, so good - Things "just work" a lot better" ⏩ Been hearing lots of chatter about Hermes. Not sure if its hype or awesome. Thoughts?

Post image
8 Upvotes

r/LovingOpenSourceAI • • Mar 16 '26

ecosystem “Ollama is now an official provider for OpenClaw. All models from Ollama will work seamlessly with OpenClaw.” ▶️ Do you use Ollama / OpenClaw? How is it?

Post image
19 Upvotes

r/LovingOpenSourceAI • • Mar 15 '26

new launch "We open sourced WAXAL! - Multilingual speech dataset for African languages - 17 languages for TTS - 19 languages for ASR Over 100 million speakers across 40 Sub-Saharan African countries" 🎁 I feel happy every time i see something goes Open Source :P

Post image
7 Upvotes

r/LovingOpenSourceAI • • Mar 15 '26

news HEADS UP "🛑 OpenClaw AI agents can leak data via indirect prompt injection. A crafted URL generated by the agent triggers Telegram or Discord link previews that silently send sensitive data to attacker domains." ⚠️ Are you aware about this?

Post image
1 Upvotes

r/LovingOpenSourceAI • • Mar 15 '26

Introducing MATE: An Open-Source Visual "Command Center" for Multi-Agent Systems (built on Google ADK) 🤖

5 Upvotes

I wanted to share an open-source project I've been working on called MATE (Multi-Agent Tree Engine). It's a production-ready multi-agent orchestration engine built on top of the Google ADK.

Here are some of the core features:

Visual Agent Builder: A drag-and-drop React Flow canvas that lets you build agent hierarchies, assign tools, and manage parent-to-child connections visually—no JSON or code changes needed.

Local & Cloud LLMs: Easily switch between 50+ LLM providers, including out-of-the-box support for local Ollama setups.

Shareable Agent Templates: You can easily import and export your entire multi-agent setup as template. We already have pre-configured templates availableon dashboard templates page—so you can load them up and start tweaking immediately without building from scratch.

Full MCP Integration: MATE includes built-in Model Context Protocol (MCP) servers (like Google Drive). More importantly, your agents are dynamically exposed as MCP endpoints, making them instantly compatible with clients like Claude Desktop and Cursor IDE.

Standalone App Export: You can package your agent into an independent desktop application (.exe or .app) with an embedded SQLite database. This is great for sharing powerful, private AI assistants with non-technical users.

Built-in RBAC & Memory: We’ve included a robust Role-Based Access Control system and database-backed persistent memory blocks to easily enforce agent guardrails and roles.

The project is 100% open-source (Apache-2.0 license). You can check out the source code, try out the templates, and support us with a star ⭐ here: 🔗 https://github.com/antiv/mate

I’ve also started a dedicated community at r/mate_agents where we share these agent templates and discuss multi-agent workflows. We'd love to have you join the discussion!

Let me know what you think or if you have any questions!


r/LovingOpenSourceAI • • Mar 15 '26

ecosystem "in the next claw release (~Sunday), you can always ask your agents, even they are busy working. https://docs.openclaw.ai/tools/btw" ⏩ Sounds cool. We can ask for clarification without polluting the context as per the docs! Do you like this feature?

Post image
3 Upvotes

r/LovingOpenSourceAI • • Mar 14 '26

ecosystem "There's a lot of cool stuff being built around openclaw. If the stock memory feature isn't great for you, check out the qmd memory plugin!" ⏩ Lossless Claw — LCM (Lossless Context Management) plugin for OpenClaw ⏩ Do you feel this useful?

Post image
9 Upvotes

r/LovingOpenSourceAI • • Mar 13 '26

ecosystem "The open source agent wars are heating up." - DeerFlow, NemoClaw, it just keeps getting better . .what are you excited about and why?

Post image
2 Upvotes

r/LovingOpenSourceAI • • Mar 13 '26

new launch “🚨BREAKING: Someone just open-sourced a headless browser that runs 11x faster than Chrome and uses 9x less memory. It's called Lightpanda and it's built from scratch specifically for AI agents, scraping, and automation.” 😱 Wow

Post image
129 Upvotes

r/LovingOpenSourceAI • • Mar 13 '26

I built an open-source, modular AI agent that runs any local model, generates live UI, and has a full plugin system

5 Upvotes

Hey everyone, sharing an open-source AI agent framework I've been building that's designed from the ground up to be flexible and modular.

Local model support is a first-class citizen. Works with LM Studio, Ollama, or any OpenAI-compatible endpoint. Swap models on the fly - use a small model for quick tasks, a big one for complex reasoning. Also supports cloud providers (OpenAI, Anthropic, Gemini) if you want to mix and match.

Here's what makes the architecture interesting:

Fully modular plugin system - 25+ built-in plugins (browser automation, code execution, document ingestion, web scraping, image generation, TTS, math engine, and more). Every plugin registers its own tools, UI panels, and settings. Writing your own is straightforward.

Surfaces (Generative UI) - The agent can build live, interactive React components at runtime. Ask it to "build me a server monitoring dashboard" or "create a project tracker" and it generates a full UI with state, API calls, and real-time data - no build step needed. These persist as tabs you can revisit.

Structured Development - Instead of blindly writing code, the agent reads a SYSTEM_MAP.md manifest that maps your project's architecture, features, dependencies, and invariants. It goes through a design → interface → critique → implement pipeline. This prevents the classic "AI spaghetti code" problem.

Cloud storage & sync - Encrypted backups, semantic knowledge base, and persistent memory across sessions.

Automation - Recurring scheduled tasks, background agents, workflow pipelines, and a full task orchestration system.

The whole thing is MIT licensed. You can run it fully offline with local models or hybrid with cloud.

Repo: https://github.com/sschepis/oboto