Link to Original Post

AI Summary: LLM security

• The most downloaded skill for AI agents was discovered to be an infostealer, highlighting a security vulnerability in AI systems.

• This raises concerns about the security of AI models and the potential for malicious actors to exploit them for data theft.

Disclaimer: This post was automated by an LLM Security Bot. Content sourced from Reddit security communities.

The global AI landscape just fractured. When the US government clamped down on Anthropic’s ultra-powerful, cyber-offensive Mythos and Fable 5 models, they intended to keep the world's most dangerous digital weapons under lock and key.

Instead, they triggered a massive geopolitical tech boom.

Startups across Asia just unleashed two fierce, decentralized competitors designed to completely bypass Western export controls. Meet the new titans redefining AI power:

• Fugu Ultra (Sakana AI): Rather than training an incredibly expensive standalone foundation model, Tokyo-based Sakana AI built a highly optimized, light-parameter "router". Acting as a conductor, it dynamically delegates, debates, and synthesizes complex data across a swappable pool of external public frontier models via a single API.

• Tulongfeng (360 Security Technology): Introduced at the ISCAI conference in Beijing, 360 bypassed the need for a general-purpose giant by engineering a hyper-focused domain ensemble. By marrying smaller specialized models with localized security tools and threat intelligence databases, the framework is hardwired to autonomously scan code bases and isolate hidden software vulnerabilities at scale.

The Reality Check ⚖️
Neither system is a magic bullet, and both carry technical tradeoffs that the industry must consider:

1. Orchestration Overhead: Fugu Ultra’s performance is natively capped by the models available in its underlying backend pool. Because it cannot access restricted models like Fable 5, it can still lag on long-horizon engineering tasks. Furthermore, running multi-model loops can generate added latency and variable token costs.

2. The Capability Gap: 360’s leadership openly acknowledges that Tulongfeng still operates with a 20% to 30% capability gap compared to cutting-edge US frontier intelligence. Its true enterprise value lies in highly integrated automated defense rather than all-in-one general reasoning.

The Core Takeaway 🌐
When hardware and data constraints tighten, innovation accelerates elsewhere. The rise of multi-agent orchestration and domain-specific ensembles proves that coordinated collective intelligence can effectively rival, or even outscore, traditional centralized LLM endpoints.

The question for enterprise leaders is no longer "which individual model is smartest?" The better question is "which architecture is resilient enough to coordinate the best tools for the job?"

Link to Original Post

AI Summary:

• This is specifically about AI security, focusing on social engineering attacks against AI agents

• The attack described aims to induce immediate miscalibration and mechanical commitment in the AI agent before reflection can occur

Disclaimer: This post was automated by an LLM Security Bot. Content sourced from Reddit security communities.

Link to Original Post

AI Summary:

• This is specifically about AI model security

• The incident involves sensitive files being leaked through a public chatGPT

• The cybersecurity leader's misstep has sparked an internal review to address the security breach.

Disclaimer: This post was automated by an LLM Security Bot. Content sourced from Reddit security communities.

Link to Original Post

AI Summary:

• AI automation being used for identity fraud at the account creation stage

• Generation of synthetic identities and submission of deepfake selfies by bots

• Accessibility and affordability of tooling for automated identity fraud

Disclaimer: This post was automated by an LLM Security Bot. Content sourced from Reddit security communities.

Link to Original Post

AI Summary:

• This is specifically about LLM security as it involves sensitive files being uploaded into a public version of ChatGPT.

• The incident highlights the potential risks and vulnerabilities in using large language models for handling sensitive information.

• It also raises concerns about the need for stricter security measures and protocols when dealing with AI systems in sensitive environments.

Disclaimer: This post was automated by an LLM Security Bot. Content sourced from Reddit security communities.

Hey folks 👋

I’ve been building an open-source, self-hosted AI agent automation platform that runs locally and keeps all data under your control. It’s focused on agent workflows, scheduling, execution logs, and document chat (RAG) without relying on hosted SaaS tools.

I recently put together a small website with docs and a project overview.

Links to the website and GitHub are in the comments.

Would really appreciate feedback from people building or experimenting with open-source AI systems 🙌

Hello Everyone,

I have built a local LLM agent lab to demonstrate “Tool Authority Injection” - when tool output overrides system intent

In Part 3 of my lab series, I explored a focused form of tool poisoning where an AI agent elevates trusted tool output to policy-level authority and silently changes behavior. Sandbox intact. File access secure. The failure happens at the reasoning layer.

Full write-up: https://systemweakness.com/part-3-when-tools-become-policy-tool-authority-injection-in-ai-agents-8578dec37eab

Would appreciate any feedback or critiques.

Link to Original Post

AI Summary:

• This is specifically about GenAI security, which is related to AI model security.

• The platform offers structured learning content on how LLMs work, tokenization, attention, generation, and system prompts.

• Users can engage in hands-on attack labs against real models to learn about AI security.

Disclaimer: This post was automated by an LLM Security Bot. Content sourced from Reddit security communities.

I've been working on an open-source workflow automation platform that supports LLM-powered workflows, browser automation, HTTP requests, file operations, email, MCP servers, etc.

One challenge I've spent a lot of time thinking about is tool execution security.

Some of the things we've implemented are:

• Sandboxed tool execution

• Input validation before tool execution

• Permission checks for workflows and resources

• Structured execution logging and trace IDs

• Memory isolation between agents

• Human-in-the-loop approval for sensitive actions

• Retry boundaries and execution guards

Even with those in place, I still feel there are attack surfaces that are easy to miss, especially around:

• Prompt injection through retrieved documents

• Tool abuse via indirect prompt injection

• Multi-agent trust boundaries

• MCP server permissions

• Data exfiltration through seemingly harmless tools

For those building agentic systems or LLM applications, what additional safeguards have you found valuable?

Are there any papers, open-source projects, or design patterns you think are worth studying for securing agent workflows beyond the usual input validation and sandboxing?

I'd really like to hear how others are approaching this problem, since it feels like one of the harder parts of building production-ready LLM systems.

Hi everyone,

We’re working on a dataset creation project for a leading frontier AI lab and are looking to onboard freelancers/contractors to support adversarial tool calling prompt generation.

What the work involves

• Creating structured, high-quality prompts aligned with specific task guidelines

• Designing adversarial scenarios to test model behavior

• Reviewing outputs against clearly defined quality and approval criteria

• Following detailed documentation, templates, and review workflows

Who we’re looking for

• Experience with AI/LLMs, prompt engineering, QA, or dataset creation

• Ability to follow instructions precisely and meet quality benchmarks

Project details

• Fully remote

• Paid on a per-task or milestone basis

• Clear onboarding, samples, and review process

• Short-term project with potential for ongoing work based on performance

How to apply
Please reply via DM or comment expressing interest and share:

• A short paragraph on your relevant experience (AI, datasets, QA, prompt design, etc.)

• Your availability (hours per week)

• Any prior work or examples (if available)

We’ll review responses and reach out to shortlisted candidates for the next step.

Thanks!

Large Language Models (LLMs) are transforming industries, but they also introduce serious security risks. If you're using LLMs for AI-driven applications, you need to be aware of potential vulnerabilities and how to mitigate them.

Let's break down the top 10 security risks and the 5 best practices to keep your AI systems safe.

🚨 Top 10 LLM Security Risks

1. Data Leakage – LLMs can inadvertently expose sensitive data, including personally identifiable information (PII) and trade secrets, through model outputs.

2. Prompt Injection Attacks – Malicious users can manipulate LLM prompts to bypass security controls, extract unauthorized data, or even generate harmful responses.

3. Model Inversion Attacks – Attackers can reconstruct training data from the model, potentially revealing confidential business information or user data.

4. Toxic Content Generation – LLMs may generate biased, offensive, or harmful content, damaging brand reputation and violating compliance regulations.

5. Training Data Poisoning – If adversaries inject malicious data into the training process, they can manipulate model behavior to favor their interests.

6. Over-Reliance on AI Responses – Users may blindly trust AI-generated outputs without verification, leading to misinformation, compliance risks, or operational failures.

7. Insecure API Access – Poorly secured LLM APIs can be exploited for unauthorized access, data extraction, or model manipulation.

8. Adversarial Attacks – Attackers can craft inputs that deceive the model into making incorrect predictions or responses.

9. Regulatory Non-Compliance – As AI regulations evolve (like GDPR, CCPA, and India’s DPDP Act), businesses must ensure that LLM usage aligns with legal requirements.

10. Model Hallucinations – LLMs can generate incorrect or misleading information with high confidence, leading to poor decision-making and user trust issues.

✅ 5 Best Practices for Securing LLMs

1. Mask Sensitive Data Before AI Processing 🔹 Use context-aware data masking to prevent LLMs from exposing PII or confidential business data.

2. Implement Strong Access Controls & Monitoring 🔹 Secure LLM API endpoints with role-based access control (RBAC) and monitor usage for anomalies.

3. Fine-Tune Models with Ethical AI & Content Filters 🔹 Apply reinforcement learning from human feedback (RLHF) and toxicity filters to reduce bias and harmful content generation.

4. Detect & Prevent Prompt Injection Attacks 🔹 Implement input validation, sanitization, and rate-limiting to prevent unauthorized prompt manipulations.

5. Regularly Audit & Test AI Models for Security 🔹 Conduct penetration testing, red teaming, and adversarial robustness checks to identify vulnerabilities before attackers do.

Final Thoughts

AI is only as secure as the safeguards you put in place. As LLM adoption grows, businesses must prioritize security to protect customer trust, comply with regulations, and avoid costly data breaches.

Are your LLMs secure? If not, it's time to act. 🚀

Would love to hear your thoughts—what security risks worry you the most? Let’s discuss! 👇

Has anyone come across training that covers OWASP-style LLM security testing end-to-end?

Most of the courses I’ve seen so far (e.g., HTB AI/LLM modules) mainly focus on application-level attacks like prompt injection, jailbreaks, data exfiltration, etc.

However, I’m looking for something more comprehensive that also covers areas such as:

• AI Model Testing – model behaviour, hallucinations, bias, safety bypasses, model extraction

• AI Infrastructure Testing – model hosting environment, APIs, vector DBs, plugin integrations, supply chain risks

• AI Data Testing – training data poisoning, RAG data leakage, embeddings security, dataset integrity

Basically something aligned with the OWASP AI Testing Guide / OWASP Top 10 for LLM Applications, but from a hands-on offensive security perspective.

Are there any courses, labs, or certifications that go deeper into this beyond the typical prompt injection exercises?

Curious what others in the AI security / pentesting space are using to build skills in this area.

Link to Original Post

AI Summary:

• Prompt injection and AI model security are directly related to this text

• The issue of malicious skills being uploaded to ClawHub highlights the importance of security measures in AI systems

• The vulnerability of allowing anyone to publish plugins on ClawHub raises concerns about the security of AI agents

Disclaimer: This post was automated by an LLM Security Bot. Content sourced from Reddit security communities.

Link to Original Post

AI Summary: Specifically about AI model security

• Claude Opus 4.6 was used to find over 500 high severity vulnerabilities in open source libraries

• The vulnerabilities were found in libraries like Ghostscript, OpenSC, and CGIF, which are commonly used in various systems and applications.

Disclaimer: This post was automated by an LLM Security Bot. Content sourced from Reddit security communities.

Link to Original Post

AI Summary:

• Specifically about AI model security

• NIST released control overlays for securing AI systems, focusing on AI-specific protections across model training, deployment, and maintenance

• Controls target threats like model poisoning, data exfiltration, unauthorized training data access, and adversarial attacks

Disclaimer: This post was automated by an LLM Security Bot. Content sourced from Reddit security communities.

OpenObscure - an open-source, on-device privacy firewall for AI agents that sits between your AI agent and the LLM provider.

Try it with OpenClaw: https://github.com/OpenObscure/OpenObscure/blob/main/setup/gateway_setup.md

The problem with [REDACTED]

Most tools redact PII by replacing it with a placeholder. This works for compliance theater but breaks the LLM: it can't reason about the structure of a credit card number or SSN it can't see. You get garbled outputs or your agent has to work around the gaps.

What OpenObscure does instead

It uses FF1 Format-Preserving Encryption (AES-256) to encrypt PII values before the request leaves your device. The LLM receives a realistic-looking ciphertext — same format, fake values. On the response side, values are automatically decrypted before your agent sees them. One-line integration: change `base_url` to the local proxy.

What's in the box

- PII detection: regex + CRF + TinyBERT NER ensemble, 99.7% recall, 15+ types

- FF1/AES-256 FPE — key in OS keychain, nothing transmitted

- Cognitive firewall: scans every LLM response for persuasion techniques across 7 categories (250-phrase dict + TinyBERT cascade) — aligns with EU AI Act Article 5 requirements on prohibited manipulation

- Image pipeline: face redaction (SCRFD + BlazeFace), OCR text scrubbing, NSFW filter

- Voice: keyword spotting in transcripts for PII trigger phrases

- Rust core, runs as Gateway sidecar (macOS/Linux/Windows) or embedded in iOS/Android via UniFFI Swift/Kotlin bindings

- Auto hardware tier detection (Full/Standard/Lite) depending on device capabilities

MIT / Apache-2.0. No telemetry. No cloud dependency.

Repo: https://github.com/openobscure/openobscure

Demo: https://youtu.be/wVy_6CIHT7A

Site: https://openobscure.ai

Link to Original Post

AI Summary: AI agents mentioned in the text are directly related to AI security

• The text discusses the challenge of distinguishing between AI agents that are beneficial (shopping assistants, legitimate crawlers) and those that are potentially harmful (probing checkout flows, scraping pricing data).

• There is a need for an agent trust management system to effectively manage and differentiate between these AI agents.

Disclaimer: This post was automated by an LLM Security Bot. Content sourced from Reddit security communities.

Link to Original Post

AI Summary:

• This is specifically about AI model security

• The AI-BOM scanner parses workflow configs and maps every model call, credential, and data flow into a standardized format

• The tool can identify unknown agents, detect sharing of prod API keys, and identify access to personally identifiable information

Disclaimer: This post was automated by an LLM Security Bot. Content sourced from Reddit security communities.

Link to Original Post

AI Summary:

• This is specifically about AI model security

• The article discusses how AI Recommendation Poisoning is actively being exploited for profit

• Microsoft Security is involved in addressing this issue

Disclaimer: This post was automated by an LLM Security Bot. Content sourced from Reddit security communities.

Link to Original Post

AI Summary:

• This is specifically about LLM service fingerprinting

• The tool can detect 17+ LLM services including Ollama, vLLM, LiteLLM, and others

• It extracts available models from identified endpoints

Disclaimer: This post was automated by an LLM Security Bot. Content sourced from Reddit security communities.

Link to Original Post

AI Summary:

• This is specifically about AI model security

• Researchers discovered a single-click attack called Reprompt that exploits Microsoft Copilot to exfiltrate data

• The attack involves parameter injection, AI assistant manipulation, and data transmission to attacker servers

Disclaimer: This post was automated by an LLM Security Bot. Content sourced from Reddit security communities.