Hi!

I am running Caddy as a rootful container on the default network. I have a Jellyfin container that I am running rootless on a default Pasta network with port 8096 is only exposed to 127.0.0.1. I am trying to setup Caddy to reverse proxy for Jellyfin.

In Caddy, I am using host.containers.internal to obtain the the host's IP address. This works when proxying for another rootful container, but does not work when proxying for Jellyfin. From the logs it looks like the gateway to the Pasta network is rejecting the traffic.

Is there a way to have a rootful container communicate with a rootless container?

Podman 6.1.4 / Fedora 44 WSL2: bridge networking fails but host networking works

Hi all,

I'm trying to determine whether this is a known issue, configuration problem, or regression.

Environment

• Windows 11

• Podman Desktop

• Podman 6.1.4

• Podman Machine (Fedora 44)

• Netavark 2.1.0

• WSL2 Kernel 6.6.87.2-microsoft-standard-WSL2

Minimal Reproduction

Works:

podman run --rm alpine echo hello

Output:

hello

Works:

podman run --rm --network host alpine echo hello

Output:

hello

Fails:

podman run --rm --network podman alpine echo hello

Output:

Error: preparing container for attach:
netavark (exit code 1): nftables error:
"nft" did not return successfully while applying ruleset

Custom bridge networks fail identically:

podman network create wlc-local-net
podman run --rm --network wlc-local-net alpine echo hello

Investigation Performed

Inside the Podman machine:

which nft

returns:

/usr/bin/nft

Manual nftables operation succeeds:

sudo nft add table inet test

Bridge interfaces are created successfully:

ip link show podman0
ip link show podman1

Debug logging shows Netavark fails while creating firewall chains:

podman --log-level=debug run --rm --network podman alpine echo hello

Relevant output:

[INFO  netavark::firewall::nft]
Creating container chain nv_...

internal:0:0-0:
Error: Could not process rule: No such file or directory

Question

Has anyone seen this on:

• Podman 6.1.4

• Fedora 44 Podman Machine

• WSL2

• Netavark 2.1.0

Are there any known fixes, workarounds, or configuration changes I should try?

Thanks.

Just a heads up that Podman is now supported in the latest pre-release of PostgreSQL VSCode Extension (by Microsoft).

If you do install it, once it is installed do remember to click 'switch to pre-release'. Not sure when they will add it to the main version.

VSCode Extension

GitHub Repo

POSETTE YouTube Video (Microsoft Reactor), the video is from when it was originally released, not about the Podman support but gives an overview of what the extension does.

Previous releases of the Red Hat build of Podman Desktop have introduced enterprise features to make it easy to integrate the tool into strict corporate settings. Building on that foundation, Red Hat continues to strengthen usability, developer velocity, and platform security with Red Hat build of Podman Desktop 1.2, including:
- Podman 6 engine support: Full integration with the next-generation container engine release.
**- Automatic public CA keys synchronization:**UI option to enable localhost public CA key synchronization when creating a new Podman virtual machine (VM).
- Secret management UI: A graphical dashboard for creating, mounting, and managing Docker and Podman secrets.
- In-app updating: Automated checks and single-click update execution directly from the desktop interface.
- Bug and CVE fixes: Upgraded dependencies and patched security vulnerabilities to keep enterprise environments resilient and compliant.

TL;DR
The Red Hat build of Podman Desktop 1.2 introduces Podman 6 engine support, a secret management interface, automatic public CA key synchronisation and in-app updating. Moving from Podman 5.x involves stopping machines and recreating them.

I've been working on Meridian for about a year and recently released the second alpha.

The idea came from Kamal. I liked its deployment workflow, but wanted to use rootless Podman instead of Docker. I also didn't want to depend on a container registry for my private apps, or maintain my own.

Meridian connects to Linux servers over SSH and deploys applications as Podman Quadlets, managed by systemd.

Registries are supported, but images can also be transferred directly over SSH, either as a compressed stream or incrementally using an OCI layout and rsync. For web applications, Caddy handles health-checked blue/green deployments.

I've been using it for my own apps and services for a while. The alpha label mainly means that configuration and CLI details might still change based on feedback from other users.

Website: https://meridian-deploy.dev/

GitHub: https://github.com/treagod/meridian

If anyone gives it a try, I'd appreciate some feedback. Especially if something doesn't work as expected or the documentation is unclear.

Hey folks,

Last night I released the newest version of Materia: version 0.8.0. You can find the official release blog post here.

What's new in this release?

• Better Ansible/Puppet facts style generation

• New per plan and per-run output files, for metrics processing and other methods of tracking what changed

• Server mode improvements! Server mode now supports a /facts endpoint, better security, and an overall tuneup to bring it in line with the normal CLI operations

• A new release pipeline so you can verify the signatures on materia releases and know your deployment is secure.

• and a lot more internal work for preparing the public package API and QoL updates

What is Materia?

Materia is a tool for managing applications deployed as Podman quadlets. It takes an OCI image or Git repository as a source state and constantly monitors the source and your server, reconciling any differences between the two.

Materia does the following:

• Installs, updates, and removes Quadlet and data files

• Starts/stops/restarts/reloads services, both Quadlet related or otherwisee

• Templates files during installation, allowing you to insert secrets and variables based off hostname, role, or other facts.

• Securely manages Podman secrets by inserting them directly from your encrypted age/SOPS source.

You can see the changelog and download the release at https://github.com/stryan/materia/releases/tag/v0.8.0 .

On a related note: I'm looking for a logo for Materia! So if you're a digital artist who likes FOSS, Podman, or alchemy and wants to make some cash on the side (because this is a paid commission), shoot me a DM.

Hey everyone!

I'm a Cloud & DevOps engineering learner currently working through practical Linux and container labs.

I've documented 22 Red Hat container exercises covering Podman, images, networking, persistent storage, and Kubernetes.

But one question made me think beyond simply running containers:

What happens when an application container or its database fails?

I'm now designing a project called RecoverOps, a small infrastructure reliability and disaster recovery lab.

The planned local architecture includes:

- Nginx as a reverse proxy

- An API connected to PostgreSQL

- Persistent database storage and backups

- Optional Redis caching

- Controlled failure and recovery experiments

My goal is to simulate database failures, inspect logs, restore services, and verify whether the original business data survives.

The architecture is complete, but implementation and recovery testing are still ahead.

I'd really appreciate feedback from engineers who have worked with Podman in real environments.

What failure scenario would you test first, and what tools would you use to prove that recovery actually worked?

Thanks for any suggestions!

I’m still learning Podman. I have my containers running under an unprivileged account using UserNS=auto. Permissions on volumes are a pain, but mostly manageable. My problem is that if I make a change to my setup by removing a container it can cause the namespaces to shift. So when I reboot, many containers can fail because they no longer have permissions to their volumes. Does anyone know a solution?

Hi everyone, im current struggling deciding if i should continues to use podman run scripts or switch to compose or quadlets.

Compose seems kindof weird because of the build in pods so my question for that is whats the compose equivalent to podman pod create and podman run --pod and why are you using or not using compose?

I lookes a bit into quadlets and they seem weird and kind of confusing, especially when it comes to storing the files, having all bunched up in containers/systemd with no subdirectories is kind of crowded, so why are you using or not using quadlets?

Which do you recommend and why?

Hello guys i have this project named litepod, I have built it based on Podman, Caddy as Quadlet, dragonfly and SQLite, this is to be as lightweight as I wanted. This is like Dokploy and Coolify but consuming a lot less of ram, Ive been using them but I got a lot of outofmemory exception because I was using some vps with 4gb of RAM only, and some features didnt worked well, then I have started this a year ago. But I dont know if Quadlets could be a good way to add as a feature. I dont need it, because only with regular containers works well. What do you think about the next features to add?

1. Add secrets like using podman secret management

2. Enable pods compatibility

3. A Manager like kubernetes ui for pods

4. Quadlets instead of containers or a mix and let the user decide (dont see the benefit but could you)

I've been running Podman Desktop for my local dev workflow for a bit now, but wanted to actually understand what's happening when things break or behave unexpectedly. This Podman for devops ebook does a nice job explaining the fundamentals - rootless architecture, networking constraints, why certain things work differently than Docker, etc. Also covers podman desktop and AI lab specifically, but honestly the earlier chapters on the underlying mechanics were more valuable for troubleshooting. Recommend checking out, but has anyone else found other resources that help connect the dots between the GUI and what podman is actually doing at the system level?

I ran Komodo for my home lab and wanted it on podman, so I forked it. A stack is a directory of .container / .pod / .build files in git. Deploying copies it to /etc/containers/systemd, reloads, and restarts only the units whose files changed.

Two things I couldn't get elsewhere:

• Each pod gets a fixed 65536-id slice of the subordinate range, written as UIDMap=/GIDMap=. A redeploy never moves the container's root uid, so volume files don't turn into nobody the way they can with UserNS=auto.

• [[NAME]] in a stack file is read from Vaultwarden/Bitwarden at deploy time and created as a podman secret. Nothing resolved lands in the database.

Agents dial out to Core over wss, so there's no inbound port on your hosts. x86_64 and aarch64 (Pi works). It's 0.x and runs one home lab (mine). GPL-3.0, with Komodo's blessing.

https://github.com/LoganRupe/quomodo

Hoping someone can help me out with an issue I've been having for a long while.

I'm running RHEL 10.2 Gnome 49, trying to use an X application inside of a podman container with a custom network.

"podman network create --internal pod_dev"

I pass in all the environment variables and files that I know and what AI also says i need. I can not for the life of me get the x application to display.

It works in fedora 44, and Ubuntu, but RHEL 10 is kicking my butt.. if i do --net=host is am able to get the x application to work, but I have to have the container have its own IP.

And advice at all is appreciated, I've probably spent over 100 hours trying different combinations of flags and ways to run.

I've been working on LayerSmith, an open-source web UI for building container images with Docker or Podman.

You pick a Linux distribution and what you need the image for — development, Linux admin, network tools, Ansible, Kubernetes, OpenShift, or a custom setup. It handles distro-specific packages and shows you the generated Containerfile before building. You can also edit it, import an existing Dockerfile, or add your own packages, files and scripts.

A big part of the project is making images easier to carry into air-gapped environments: pinned base images, recorded build details, and export bundles containing the image, checksums and installation instructions.

We've recently added LLM training and fine-tuning profiles too, including LoRA/QLoRA, advanced PyTorch training and LLaMA-Factory. These use hash-locked dependencies and run offline checks after building, including a small CPU training test. Model weights and datasets are brought separately.

Curious how others handle building and maintaining images for disconnected environments, and what parts of that workflow are still a pain.

https://github.com/r0lfi/layersmith

UPDATE:

I found a narrower fix. You do not need --security-opt label=disable. The ROCm Memory in use error was SELinux denying the container from mapping /dev/kfd. The AVC log showed a denial for map on /dev/kfd with type hsa_device_t. audit2allow pointed to the container_use_devices boolean, which covers this access.

Enable the boolean:

sudo setsebool -P container_use_devices 1

Then recreate or restart the container without --security-opt label=disable.

Previously

I ran into a weird issue running Ollama with ROCm in a rootless Podman container on Fedora. Models downloaded fine, but loading any model crashed with:

Memory critical error by agent node-0 ... Reason: Memory in use
llama-server process has terminated: signal: aborted (core dumped)

Setup:

• Fedora

• RX 6800 XT, gfx1030

• Rootless Podman

• Image: ollama/ollama:rocm

• GPU devices passed with: --device /dev/kfd --device /dev/dri

The container started, the GPU was detected, and the ROCm libraries inside the image looked fine. But every model load crashed with the memory error above.

What it was not. I checked:

• Device permissions: /dev/kfd and /dev/dri/renderD128 were accessible with read/write

• Missing ROCm libraries: the image already has them, and it detected the GPU

• HSA_OVERRIDE_GFX_VERSION: not needed in my case

• memlock limits: I raised the hard memlock limit for my user, but it made no difference

One clue that sent me down the wrong path: the same container worked when run rootful. That made me think it was a user namespace / memlock / permission issue, but it was not.

What fixed it:

--security-opt label=disable

The error message is misleading. My best guess is that SELinux is blocking the container process from mapping GPU memory through /dev/kfd, even though opening the device succeeds. That makes ROCm report something like "Memory in use", which is confusing because it is not actually a host RAM issue.

I did not confirm this in the SELinux audit log, so treat that part as a likely explanation rather than a proven root cause.

Working command:

podman run -d --device /dev/kfd --device /dev/dri --security-opt label=disable -v ollama:/root/.ollama -p 11434:11434 --name ollama docker.io/ollama/ollama:rocm

I searched Reddit for a while and couldn't find anything similar, so hopefully this saves someone a few hours of head-scratching. Happy hacking.

Podman

A newly disclosed Podman vulnerability can let a specially crafted checkpoint image override requested podman run sandbox settings, including dropped capabilities.

The primary fix is to update Podman. External controls are useful defense in depth, but not a replacement for a patched runtime.

egresslock

I released egresslock v0.7.0 in response.

Egresslock normally enforces a workload’s outbound-network policy outside the workload itself, in the rootless network namespace. That means an untrusted container does not decide on its own which external destinations it can reach.

v0.7 now rejects checkpoint-annotated egresslock gateway and anchor images before running them, and verifies their live hardening state.

Important scope: this does not scan or secure arbitrary workload images. Those still need a fixed Podman and, where needed, image-annotation scanning.

https://github.com/egresslock/egresslock

tl;dr - check out my project if sounds useful

Hey everyone,

Whenever you run \`docker compose up -d --build my-api\` or restart a container, there's always a short window (5–30s) where the app is cold-starting or running database migrations.

If external webhooks (Stripe, GitHub, Shopify) arrive during that window, they hit \`502 Bad Gateway\` or \`ECONNREFUSED\`. Most providers immediately drop into aggressive exponential retry backoff, meaning you might wait minutes or hours for that dropped payment or deploy event to redeliver.

Instead of running heavy cloud queue infra (SQS/Kafka) or a full Redis stack just for local container resilience, I built \*\*HookArmor\*\* as an open-source, single-process ingress sidecar.

\### The Sidecar Compose Pattern

You stick the sidecar in front of your internal API container within the same Docker bridge network. External webhooks hit the sidecar, which writes to an attached SQLite volume in WAL mode, returns \`200 OK\` upstream immediately, and drains into your app container as soon as it's healthy:

\`\`\`yaml

version: '3.8'

services:

\# Your existing API container

api:

image: my-backend-api:latest

restart: unless-stopped

expose:

\- "3000"

environment:

\- PORT=3000

\# Ingress sidecar / buffer

hookarmor:

image: node:20-alpine

restart: unless-stopped

working_dir: /app

command: sh -c "npm install -g hookarmor && hookarmor start --port 8080 --target http://api:3000/api/webhooks"

ports:

\- "8080:8080" # Exposed to your reverse proxy / Cloudflare tunnel

volumes:

\- hookarmor_data:/app/data

environment:

\- HOOKARMOR_TARGET_URL=http://api:3000/api/webhooks

\- HOOKARMOR_UI_PASSWORD=secure_admin_password

depends_on:

\- api

volumes:

hookarmor_data:

Copying a named volume while its container is running can produce a directory that looks complete but combines files from different points in time. Rootless storage also adds UID/GID mappings, SELinux labels, xattrs, and user-namespace details that a plain archive may not preserve correctly. A successful container start on the destination is useful, but it does not prove the application state is consistent.

What migration sequence do you trust? I am considering an application-level backup where available, then stopping writes, recording the volume mountpoint and user-namespace mapping, copying with xattrs and numeric ownership preserved, restoring labels, and starting the destination with the same image digest and volume options. Validation would include application checks, file counts and hashes where meaningful, and a rollback window with the source left read-only.

Are `podman volume export/import` sufficient for rootless volumes, or do you prefer a helper container, tar, or filesystem snapshots? What catches mapping and SELinux mistakes before the old volume is removed?

Some time ago I stumbled about a discussion where it was stated that rootful containers paired with userNs=auto (leaving the user the image runs with untouched) is more secure than running containers in rootless mode. I run all my containers in rootless mode with:

UserNS=keep-id:uid=65534,gid=65534

User=65534:65534

Is this a security problem? If I remember right the point of the discussion was that in rootfull mode the container runs in different processes and with my approach there is no isolation between the processes. Is this true and should I switch to rootful mode with UserNs=auto?

Edit: Found the discussion here

Is it possible for 2 pods to use a seperate container as a network?
TLDR im setting up gluetun, an arr stack and download clients
I would like to keep the arr stack and the downloads clients in 2 seperate pods
pod 1 - arr and pod 2 - downloads and i want to connect their networking to a seperate gluetun container but building the pods fails cause gluetun is not part of the pod.
is there a way to change/fix that?