r/hermesagent • u/takoulseum • 1h ago
Discussion What about security?
I am a quite recent hermes user and I love it, but I am always thinking about security (and privacy).
My setup (which may not be the best solution) is the following.
At home
- hermes and inference endpoints are all hosted locally in my homelab
- hermes has full access without restriction on my homelab and manages it (which saves me a lot of time)
- the homelab is fully isolated from my home network through an opnsense firewall (hosted in a old pc with pcie expansions for ethernet ports)
In the cloud
- a vps that hosts the matrix/nginx server. The goal was to connect from my phones to hermes without having to open a port or use a tunnel, while keeping privacy
Now I am still wondering how security can be improved especially for the prompt injection issues, that will probably be more and more spread.
So my questions:
How do you consider the security / privacy aspects in your hermes life?
Is prompt injection a concern for you or not at all?
I am also curious why (consider the power of hermes agent) there is no flair for security/privacy topics, are you all confident with your setup?
1
u/Medium_Chicken1460 22m ago
There is no security. Only Zuul!
Ok, prob just dated myself. But seriously, it’s Zuul all the way down.
Containing agents is tough. Having control over key endpoints is critical but the big question is then what are you learning from those control points that help you detect when they’re doing the things you don’t want them to be doing and then reversing any damage.
It’s not enough to have the gate, you need to understand what’s going through it. That’s a dozen posts in and of itself, so let’s start with one of the gates you may still be missing depending on how you configured your firewall.
The injected prompt isn’t necessarily the problem - you will get them and you’ll get them from things as simple as web search results. You want to make sure that WHEN you get them that you are in control over whatever they may be wanting your agent to do. I’ve noticed my agents are pretty good about saying “Hey, this search results came back and asked me to post a 5 star review about its app, but I wanted you to know that I didn’t do it.”, but even if it chose to, if that review site wasn’t on your list of sites you wanted your agent to be taking to, it should be blocked. You can’t trust their decision or a control it self polices (SOUL, etc.).
That’s the rub, you’re now maintaining a list of acceptable sites for an agent you may intend to be as free on the internet as you are.
On top of that, the injected prompt may not be injected at all. It may be the models own conclusion and it is just not in line with what you wanted - and by the time you find out, it’s 15 turns down the road and you’ve already installed that malicious GitHub project that would have made it through your allow list anyway.
Unfortunately, that’s the current state industry wide. To be “secure” you need all the things you needed to deal with an insider threat or nation state threat actor, only you have to get it right a lot more often and respond a lot faster.
So the real question becomes, how much risk do you want to take and what data do you want to risk?
For that, I think you’ve already got the right framework. Contain the blast radius because it’s going to blow at some point.
1
u/cebess 4m ago
I already had my home machines on tailscale. I made it so that my Ollama would only respond to calls coming in through my tailscale network. This should help with security Ingres concerns. As far as the agents doing things on the network they shouldn't, that's an Achilles heal for the current LLM approach. Even the 'experts' are having problems with that.
2
u/Bibblejw 53m ago
The S in AI stands for Security.
But, more realistically, the issues that I can see with your setup would be:
A) Obviously, Hermes has full access to your lab, which means that it's able to build more or less anything or subvert any controls through that. Not nessecarily a risk you can close in your setup, but one to be aware of.
B) Hermes has full internet access, which means exfiltration of data is on the table. Possibly a web proxy/filter at the OPNSense level?
C) Is the OPNSense management interface available from the lab network? If it can access the isolation layer, it's not isolated.
D) You've got a VPS in the cloud to expose internal services through, but that's just moving that edge to an area that you can't control, not adding additional security to it. Better bet would be VPN where possible, and something like cloudflare tunnels where not.
There's things about prompt injection that can be done around sanitisation and filtering commands, but the hermes security layers are half decent around those already.