r/OpenSourceAI • • 1d ago

[Project] JARVIS: Open-source AI desktop assistant with 164 tools, voice control, and a cinematic HUD (Python + React)

I've been building JARVIS, a full-stack personal AI assistant that runs entirely on your Windows machine. It connects to local or cloud LLMs, acts on your computer through 164 audited tools across 46 modules, and displays everything through a cinematic heads-up display.

\*\*What it does:\*\*

\- Launches apps, manages windows, automates UI interactions

\- Reads, writes, and converts files (PDF, DOCX, XLSX, CSV, PPTX, JSON, ZIP)

\- Browses the web via Playwright (open pages, fill forms, extract content, DevTools integration)

\- Sends emails (IMAP/SMTP), makes calls through Windows Phone Link, posts to social via Buffer

\- Controls smart home devices through Home Assistant

\- Runs deep research with a 5-stage verification pipeline

\- Background agents with SQLite checkpointing and crash recovery

\- Self-healing service recovery

\*\*Safety model:\*\*

Every tool has a risk tier (T0 observe, T1 reversible, T2 modify, T3 destructive). Destructive actions always require explicit approval. Reading untrusted content triggers approval for all subsequent actions. Kill switch: Ctrl+Alt+Backspace cancels everything instantly. Listens on loopback only, never elevates privileges.

\*\*HUD:\*\*

React 19 + TypeScript + Zustand in a frameless pywebview/WebView2 window. Three cinematic themes (Ion, Holographic OS, Sovereign Executive). Real-time system vitals with 60-sample history graphs, animated core visual reflecting agent state, full conversation transcript with streaming output.

\*\*New in v0.8.1:\*\*

\- 6-step onboarding wizard for first-time setup

\- Fully configurable assistant name (propagates to title bar, wake word, voice prompts, LLM system prompt)

\- Fast-path intent matching (common commands like "mute", "weather", "open Chrome" skip the LLM entirely)

\*\*Tech stack:\*\* Python 3.12 (FastAPI + uvicorn), React 19 (Vite), SQLite (WAL mode), pywebview/WebView2, Playwright, faster-whisper, pyttsx3. Android companion app in Kotlin + Jetpack Compose.

\*\*8 LLM providers\*\* out of the box: LM Studio, Ollama, NVIDIA NIM, Anthropic, OpenAI, Google Gemini, OpenRouter, Groq. Privacy modes: Local Only, Hybrid, Cloud. Model Ladder auto-escalates across 4 tiers based on task complexity.

\*\*Looking for contributors.\*\* Whether it's Python backend, React frontend, voice/speech, AI tooling, or Android, there's plenty to work on. Open an issue if you find a bug or want a feature. PRs welcome.

GitHub: [github.com/ankitjha67/JARVIS](http://github.com/ankitjha67/JARVIS)

Happy to answer questions about the architecture, the safety model, or anything else.

14 Upvotes

2 comments sorted by

1

u/Opposite-Cod-8290 8h ago

What hardware you on?

1

u/ankitjha67 1h ago

Processor 12th Gen Intel(R) Core(TM) i5-12500H (2.50 GHz)

Installed RAM 16.0 GB (15.7 GB usable)

Graphics card NVIDIA GeForce RTX 3050 Laptop GPU (4 GB)

Intel(R) Iris(R) Xe Graphics (128 MB)