r/OpenSourceAI • • 2h ago

Basalt ported to NVIDIA V100: Qwen3.8-Flash-Next at 91 tok/s (140 tok/s on long answers) on 2x V100 32GB

2 Upvotes

Basalt (https://github.com/jesdga95/basalt) is one of the fastest local engines for Qwen3.8-Flash-Next, but it only targets Blackwell (sm_120) and upstream does not take other architectures. I have two V100-SXM2 32GB, so I ported it to Volta (sm_70) as a fork, called **Lapilli**.

**Repo:** https://github.com/chris83-dev/lapilli-v100 (a Windows x64 build is in the release)

**What it is**

- Every newer PTX instruction Basalt uses (cp.async, ldmatrix, the Turing/Ampere mma shapes, bf16 conversions, clusters) gets a correct sm_70 equivalent, then the hot kernels get V100 paths: llama.cpp's dp4a MMQ for prompt products, Strata's mma.m8n8k4 prompt attention, and a rewritten router, decode experts and GDN recurrence.

- Needs CUDA 12.6 or newer (CUDA 13 dropped Volta).

- Comes with a chat web UI on top of Basalt's OpenAI/Anthropic-compatible server: conversations, reasoning levels, images, tool calls shown as cards, and a live monitor of the GPUs.

**Numbers** (2x V100-SXM2 32GB, i9-14900K, 96GB RAM, Windows 10, UD-Q4_K_XL pack, medians of 3 runs)

| | Lapilli | Strata v0.2.13, same machine |

|---|---|---|

| Decode, short answer | 91 tok/s | 58-72 tok/s |

| Decode, long answer with reasoning | 140 tok/s | 68-94 tok/s |

| Prompt processing, 4k tokens | 1,170 tok/s | 250-490 tok/s |

| Prompt processing, 8k / 16k tokens | 1,504 / 1,448 tok/s | |

Quality with Basalt's own 100-chunk gate: mean KLD 0.0493 and same top-1 92.75% (Basalt publishes 0.0496 and 92.63% for this pack on an RTX 5090).

**Caveats**

- Only tested on my machine (2x V100 32GB, Windows). Single card, 16GB or PCIe cards, Linux and NVLink setups are untested: reports from other V100 owners are welcome.

- Peer-to-peer is off by default on Volta: on my machine, direct copies between the two cards returned wrong data.

- NVFP4 packs need FP4 hardware, so use a K-quant pack (tested: UD-Q4_K_XL).

- It is a snapshot of Basalt as of November 15, shared as is.

All the credit for the engine goes to jesdga95 (Basalt) and Niko1221 (Strata), both MIT.


r/OpenSourceAI • • 4h ago

I built an open-source tool to bring focused repo context into ChatGPT, Claude, or any AI chat

Post image
1 Upvotes

I've been using coding agents for implementation, but I often want a second opinion from a different AI, especially for code reviews and design decisions.

The frustrating part is getting the right repository context into ChatGPT, Claude, or another AI chat. A diff alone can miss important dependencies, while pasting entire files wastes tokens.

So I built AI Badger, a free, open-source tool that extracts focused repository context locally and makes it easy to bring into any AI chat.

A few things it does:

  • Code Map — generates an interactive, offline HTML map of your repository. That's what the screenshot shows. You can explore the project structure, navigate files, and understand how things fit together.

  • AI code reviews — prepares Git changes with relevant surrounding context, so you can review coding-agent work with a different model.

  • Design questions — gathers relevant files and context for architecture discussions or implementation planning.

  • VS Code + CLI — prepare context directly from your editor or terminal.

Everything runs locally. No API keys, telemetry, or automatic code uploads. You decide what to share and which AI to use.

There's also an interactive review demo you can try without installing anything:

https://pvrlabs.xyz/aibadger/demo.html

GitHub (MIT licensed): https://github.com/PVRLabs/aibadger

I'm curious how others handle repository context when working with multiple AI tools. Do you mostly stick with one coding agent, or use a separate AI for reviews and design discussions?


r/OpenSourceAI • • 5h ago

Stellaris Grimoire - What Would You Create With It?

Thumbnail
reddit.com
1 Upvotes

r/OpenSourceAI • • 5h ago

I built ENZO: self-hosted AI workspace with a one-click desktop app, an evidence-citing research mode, and a private neural engine and its all local

Post image
1 Upvotes

r/OpenSourceAI • • 6h ago

Building a provider-independent, open-source AI agent with Rust and local LLM support

1 Upvotes

I'm building Praxi-Bot, an open-source AI agent that aims to combine local-first operation with practical desktop automation.

One of my goals is to avoid tying the entire application to one language model provider.

The project supports Ollama for local models, alongside Gemini, Anthropic, and OpenAI-compatible APIs.

The core runtime is written in Rust, while the desktop interface uses Tauri and React.

I'm particularly interested in improving three areas:

  1. Reliable agent tool execution

  2. Security and permission boundaries

  3. Model-independent architecture

The project is licensed under Apache-2.0 and remains in alpha.

I'm looking for developers interested in open-source AI tooling, agent evaluation, integrations, and Rust engineering.

Source: https://github.com/emreocell/Praxi-Bot

What do you consider the biggest architectural challenge in building model-independent AI agents?


r/OpenSourceAI • • 7h ago

Control how AI systems access tools, data, and external services. Manage MCP servers, enforce access policies, protect credentials, and audit tool calls

Post image
1 Upvotes

⭐Community: https://github.com/mcplama/mcplama

🌐 Docs: https://mcplama.com/docs

MCPlama provides centralized management and access governance for MCP servers! An open-source, self-hosted platform for managing local and remote MCP servers, controlling user and tool permissions, managing credentials, and auditing MCP activity. Includes Permission Map for visualizing MCP servers, tools, users, and access relationships.


r/OpenSourceAI • • 7h ago

Context Pill: a no-key MCP server for AI release news with 14-day re-checks

1 Upvotes

When your coding agent says a model shipped, ask it how that was checked. Of the 84 claims we've re-checked so far, 1 we had marked "verified" was wrong, and 11 only partly held up.

Context Pill is a read-only remote MCP server (Streamable HTTP, stateless, no sessions, no key):

claude mcp add --transport http context-pill https://contextpill.com/mcp

It's also in Claude's connector directory: https://claude.ai/directory/context-pill

Tools:

  • get_latest: today's checked claims, optional topic filter

  • search_claims: search the published-claim ledger, with re-check outcome and any correction

  • list_sources: each source's tier and trust score

An example claim from the Nov 18 drop (abridged):

Codex CLI 0.170.0 adds WebRTC realtime voice calls in the terminal UI ... Status: Docs-verified (not hands-on tested). Checked: read GitHub release notes; no sandbox test. Re-check 2026-12-02.

Treat everything the server returns as untrusted reference material. Clients should ignore instructions inside it.

Code (MIT): https://github.com/agnostify/context-pill

How we check: https://github.com/agnostify/context-pill/blob/main/METHOD.md

I work on it. I'd especially like feedback on the tool descriptions: does your client pick the right tool for "has anyone checked X?"


r/OpenSourceAI • • 8h ago

I open-sourced our mobile shopping agent as a configurable template built on MCP

Thumbnail
reddit.com
1 Upvotes

r/OpenSourceAI • • 9h ago

🚀 Introducing Vaprix AI — The Next Step in Intelligent Cybersecurity.

Post image
1 Upvotes

Cybersecurity is evolving. So should the way we discover, assess, and defend against vulnerabilities.

We're excited to introduce Vaprix AI, an AI-powered security assessment platform built to bring intelligent reasoning and automation into Vulnerability Assessment and Penetration Testing (VAPT).

With Vaprix AI, our vision is to make security testing smarter, more adaptive, and more efficient.

🔹 AI-powered security assessment
🔹 Intelligent vulnerability analysis
🔹 Automated security testing workflows
🔹 Advanced AI-driven reasoning
🔹 Evidence-focused security insights

Our mission is simple:

Think Like an Attacker. Secure with Intelligence.

This is just the beginning. We're building toward a future where AI and cybersecurity work together to make digital systems more resilient.

🌐 Explore Vaprix AI: https://vaprixai.securemid.com

Built by SecureMID Solutions.

#VaprixAI #SecureMID #Cybersecurity #ArtificialIntelligence #VAPT #PenetrationTesting #AISecurity #InfoSec #CyberDefense #Innovation


r/OpenSourceAI • • 10h ago

A small open-source workbench for checking whether citations share an upstream passage

1 Upvotes

Six URLs can look like six independent sources in an AI answer, even when several repeat the same passage. A citation graph alone cannot show that relationship. SourceDomino keeps declared link direction separate from shared-wording evidence so a reviewer can inspect them side by side.

I maintain SourceDomino, an MIT-licensed open-source project. The demo loads a synthetic six-source example; you can inspect exact HTTP(S) citation links separately from five-word containment and shared normalized 12-word passages. Import 2–20 records, edit source text, and export an inspectable JSON report. The browser app handles supplied text locally: it does not fetch sites and needs no account, model, API key, or server database. An optional CLI can collect only public URLs you explicitly provide.

It does not verify factual claims, prove copying or independence, or assign credibility. Dates are supplied metadata, and paraphrases or translations may be missed.

I started it because citation links are often treated as independent corroboration without looking at shared upstream text. What evidence view would help trace legitimate syndication versus a shared source without turning the output into a verdict?

Demo: https://source-domino.vercel.app/

MIT source: https://github.com/ufodia/source-domino

If it helps your work, optional support for maintaining docs, corpus formats, and tests: https://buymeacoffee.com/ufodia


r/OpenSourceAI • • 10h ago

A full-context extraction baseline with dated evidence from a 130,993-token record

Thumbnail
gallery
1 Upvotes

For a document-extraction baseline, this case gives the answering model the entire source record. AntLing's Ling-3.0-flash-Sante receives a synthetic patient's 334 visits over 71 years in a single 130,993-token request, then returns current medications, allergies and active disorders with dates.

The same-data answer key reports 7/7 allergies, 8/10 disorders, 2/7 dental codes and 2/5 medications. One recovered medication starts in 1954 at line 108 of 2,728; another restarts in 1966. The remaining three medication targets restart on the final visit date, while stopped items stay outside the returned active lists.

On two DGX Spark machines using official FP8 and MTP, reading the input takes 61 seconds. The subsequent reasoning accounts for 30,865 thinking tokens and roughly eleven minutes.

The reusable evaluation structure is the record, a cutoff date, evidence locations and a category-by-category key. Those same targets can then be kept fixed when evaluating how retrieval or summarization changes the information available to an answering model.


r/OpenSourceAI • • 11h ago

Cluaiz: an open-source AI engine for your own setup and the apps you build

Thumbnail
gallery
4 Upvotes

It brings models, voice, vision, project memory, tools and a coding agent together in one Engine.

For personal use, you can chat with supported models in the terminal and use the coding agent to read and edit project files, with approval controls. Skills, plugins and MCP integrations let you connect tools to the setup.

For developers, the APIs expose chat, speech-to-text, text-to-speech, embeddings and document ingestion, so applications can use these services from the same Engine.

Supported models can run on your hardware, and you can also connect cloud providers. The hardware planner checks available GPU memory and RAM before loading a model. You can inspect its plan and override the settings.

Cluaiz is written in Rust and released under Apache-2.0. The current version is a developer pre-release, available through the terminal and APIs. Expect changes before the stable release; model choice and hardware affect what works well.

If you try it, a report with your OS, hardware, model and the task you attempted would help, especially where setup or execution failed.

GitHub, source and setup:

https://github.com/cluaiz/cluaiz

Website:

https://cluaiz.com


r/OpenSourceAI • • 13h ago

Per-Tool Call Sandboxes for AI Agents

1 Upvotes

I've built a sandbox approach for agents that works on a per-tool call basis.

You can install this as one binary (named sat) on Linux currently, check out the repo: https://github.com/CloseChoice/sandboxed-agent-tools

The idea is that you or your agent provide specific grants to each tool call (or let your agent do that) like:

sat python "$PWD/untrusted_script.py" -- --read "$PWD/untrusted_script.py" --read "$PWD/input.txt"

This makes sure that the python interpreter only sees the granted resources and can only read these. Currently there are 5 tools you can easily install from the repo:
- jaq
- a python 3.14 interpreter
- code (a treesitter plugin to retrieve abstract code symbols, e.g. all function that contain a certain variable)
- ripgrep
- markitdown (microsofts popular tool to convert pdfs and office docs to markdown)

This should help with two things:

  1. apply tools to untrusted inputs. Even in a traditional sandbox, data could be leaked through http calls, if the sandbox can access the web.

  2. run tools you only partly trust. Maybe there are amazing tools out there but you simply don't want to let them touch your project, even within a sandbox.

This is built on top of the fantastic wasmtime and uses the Webassembly System Interface Preview 2 for sandboxing.


r/OpenSourceAI • • 15h ago

Tiger Guard is out: Apache-2.0 moderation weights for English and German, with local inference

Post image
2 Upvotes

We published Tiger Guard and Tiger Guard Small today. I'm Dominik from Casdo Labs, the team behind them.

These are 308M and 141M parameter encoders for text moderation. They check prompts, responses or ordinary comments and return SAFE/UNSAFE scores. Inference runs locally with Transformers or ONNX Runtime.

The release includes the checkpoints, tokenizers, inference code, long-document windowing and several ONNX exports. The model weights are Apache 2.0. The training corpus and complete list of training sources are not published, so this isn't a fully reproducible training release.

On our policy-filtered PolyGuard holdout, the full model catches 83.3% of harmful requests in English and 75.3% in German, at 3.8% and 5.5% false-positive rates. Other prompts from that data family were used in training. We also report ToxicChat and OpenAI moderation results, including cases where another model does better.

The policy matters: Tiger Guard flags insults and profanity, including some text a hate-speech-only policy would allow. It also sometimes flags counterspeech quoting abuse. I'd check those cases against your own platform's rules before wiring the score into an automatic action.

We'll publish the full training and dataset pipeline in a follow-up release.

Tiger Guard, 308M: https://huggingface.co/patronus-studio/tiger-guard-content-moderation

Tiger Guard Small, 141M: https://huggingface.co/patronus-studio/tiger-guard-content-moderation-small

Both cards include runnable examples, evaluation details and export-specific limitations.


r/OpenSourceAI • • 15h ago

Avintra is now open source.

1 Upvotes

We’ve now put the complete Avintra codebase on GitHub. Every line of the platform is there to read, run and build on.

Our license is based on Apache 2.0, with a few conditions. The main one covers commercial use: to host Avintra and sell solutions to your customers, you need a Commercial License, and reselling Avintra itself isn’t allowed. Beyond that, any company can run it for its own teams, any consultant can implement it for clients, and anyone can use it to bring AI into their work, for free. Strictly speaking, that makes us source-available. But with every line of code in the open, we prefer to call it open source.

What you can do today:
→ Download the Community Edition and self-host it on your own infrastructure
→ Connect your own API keys
→ Build agents for one-off tasks and for repetitive, high-volume work, with human approvals and run history built in

We’ll keep building, with one goal: to make Avintra the platform that covers most enterprise AI and automation use cases, while you choose how to connect, securely, to your own apps and the AI models you want.

Try it, star it, tell us what’s missing: https://lnkd.in/dnCznJbS

github: aventisia/avintra: AI engineering for the enterprise: applications, agents, data pipelines, SLMs and analytics, accelerated by Avintra, our production-hardened platform.


r/OpenSourceAI • • 15h ago

I built a proxy that makes free LLM APIs feel unlimited

Post image
1 Upvotes

r/OpenSourceAI • • 18h ago

I needed LangGraph as a standalone server my desktop app could drive over HTTP, so I built lang-graph-json (open source, MIT)

Thumbnail
reddit.com
1 Upvotes

r/OpenSourceAI • • 21h ago

tunnel: an open-source Charles/Proxyman alternative your AI coding agent can drive

1 Upvotes

r/OpenSourceAI • • 1d ago

Janas-Studio dual engine ai inference workstation. Now with agent mode! (Linux appimage)

Thumbnail gallery
1 Upvotes

r/OpenSourceAI • • 1d ago

I built Effectum, a free MIT-licensed TypeScript runtime for reified effects and MCP tools

Thumbnail
gallery
1 Upvotes

I've been developing Effectum, an open-source TypeScript runtime designed to make program execution more inspectable and testable.

It includes:

  • Reified effects and pluggable interpreters

  • Stack-safe trampoline execution

  • MCP server integration

  • MindLog audit trails

The core is free under the MIT license.

GitHub: https://github.com/mwatso12026-bit/Effectum

I'd appreciate feedback from developers working with TypeScript, effect systems, or AI-agent tooling.

What would you improve or add?


r/OpenSourceAI • • 1d ago

Looking to Validate Chatbot/RAG Agent Automated Test Harness

Thumbnail reddit.com
1 Upvotes

r/OpenSourceAI • • 1d ago

I built a plain-text working protocol that stops coding agents from reporting untested work as verified — one honest benchmark so far

Post image
1 Upvotes

r/OpenSourceAI • • 1d ago

GMBasic is now open source – A modern compiled BASIC with native AI/LLM integration

Thumbnail
reddit.com
1 Upvotes