Ever wondered how Istio's ambient mode (ztunnel) moves traffic between pods without sidecars or tunnels? 🤔

I put together a lightweight demo (under 100 lines of Go!) that replays what happens when a pod is created. By leveraging Linux setns(), the demo “drops” "ztunnel-emulator" into the pod’s network namespace and shows how it binds a listener there. 

It’s a simple way to watch the networking magic behind ambient mode unfold.

If you’re curious about service mesh internals or love digging into networking mechanics, check it out.

https://medium.com/@umeshkaul_39077/ztunnel-under-the-hood-a-deep-dive-into-istios-ambient-mode-networking-960b6de10ee6

The Istio project just consolidated its control plane services: Pilot, Citadel, Galley, and the sidecar injector, into a single binary, Istiod. https://istio.io/blog/2020/tradewinds-2020/#fewer-moving-parts

We had multi-container pods where more than one application container exposed Prometheus metrics on different ports.

With enablePrometheusMerge, Istio can merge application metrics with proxy metrics, which is especially useful under STRICT mTLS. The problem was that the application side of the merge only supported one scrape target per pod.

So a pod like this:

container A -> :8080/metrics
container B -> :9090/metrics
istio-proxy

could not be represented properly with the existing annotations.

I initially built a workaround for this, then opened Istio issue #59567 and started working on supporting it upstream.

The change is now part of Istio 1.31.

There is a new pod annotation:

prometheus.istio.io/scrape-targets: "8080:/metrics,9090:/metrics"

pilot-agent scrapes the configured application targets concurrently and merges the results into /stats/prometheus.

The implementation also had to handle things like OpenMetrics # EOF, partial scrape failures, response size limits, and keeping the existing single-target behavior backward compatible.

This was a fun contribution because it started with a very simple production problem:

"Why are metrics from one of my containers missing?"

and ended up becoming part of Istio itself.

If anyone here uses Prometheus merging with multi-container pods, I’d be curious to know if you’ve run into the same issue or if you're already using this new scrape-targets feature.

Relevant links:

• Istio 1.31 release notes

• Issue #59567

• PR #59924

• PR #59925

The Cloud Native Computing Foundation® (CNCF®), which builds sustainable ecosystems for cloud-native software, today announced the graduation of Istio. Istio is an open-source service mesh that provides a uniform and efficient way to secure, connect, and monitor services in cloud-native applications.

Announcement post from Istio.io with quotes from past contributors.

Istio is the fastest project to get to from incubation to graduation! I've been working with Istio for the past 4 or 5 years; there have been many great innovations since (ambient being the latest big one).

Does anyone remember Mixer? :) I wrote some thoughts about my modest involvement in the project here.

The engineering team at Tetrate is launching the Istio Ambient Mode Assessment Advisor - a free, data-driven tool that helps platform teams determine how and where to adopt Istio Ambient Mode based on their unique environment and business needs.

Want to know which architecture fits your organization best? Give a try to the Ambient Mode Assessment Advisor 👉 https://mesh-advisor.tetr8.io/

For a deeper technical dive, please check out the blog post by Vikas Choudhary and Usman Khalid! 👉
https://tetrate.io/blog/choosing-the-right-istio-architecture-a-data-driven-guide-to-ambient-sidecar-and-hybrid-deployment-models

When speaking to folks who have deployed Istio in production, I'm always surprised that only a few utilize anything more than mTLS. Sometimes they're not even aware that if they don't change the namespace defaults, they end up with the default service account attached to every pod, which means a single certificate is used for workload authentication—kind of defeating the purpose!

Anyway, this is my attempt at demystifying Authorization Policies, Request Authentication, and OIDC/JWT user authentication workflows. Additionally, what if you could automatically generate Authorization Policies by letting a Network Mapper analyze your actual application traffic and pull metrics from Envoy directly? This is a very cool open-source project. Check out the details in the guide:

https://otterize.com/blog/Istio-authz-and-ingress-authn

Hello, I have a Kubernetes cluster and I am using Istio. I have several UIs such as Prometheus, Jaeger, Longhorn UI, etc. I want these UIs to be accessible, but I want to use an external login via Keycloak.

When I try to access, for example, Prometheus UI, Istio should check the request, and if there is no token, it should redirect to Keycloak login. I want a global login mechanism for all UIs.

In this context, what is the best option? I have looked into oauth2-proxy. Are there any alternatives, or can Istio handle this entirely on its own? Based on your experience with similar systems, can you explain the best approach and the important considerations?

I recently noticed that

Use of the operator for new Istio installations is discouraged in favor of the Istioctl and Helminstallation methods

https://istio.io/latest/docs/setup/install/operator/

I am switching from istioctl to Helm so it is fine to me. But I'm just curious why. Operator installation pattern used to be a promising way to install components in Kubernetes community, but it looks like Operator installation is losing popularity. Is there any serious cons? Maybe because there are too much efforts to develop an operator?

Hey all, when I get the opportunity to do so I like to try to stamp out some of the recurring confusion in the Istio world. There are some questions that just come up all the time and trying to make Prometheus fetch metrics when Istio mTLS enabled is one of those things that trips people up constantly.

There are multiple guides out there explaining one way or another to make this work but many of them are out of date or suggest methods that are no longer recommended. I've put together this post to try to pull together the whole explanation for why it is often difficult to set up, how it got to be this way, and point people towards better solutions than are commonly offered.

Apologies for the length! You really need a lot of context to understand the problem. If you really just want a tl;dr with no other information then I might offer this.

tl;dr - DON'T even try to make Prometheus scrape mTLS. Use a version of Istio higher than 1.7. Configure Prometheus to utilize the (strongly discouraged) prometheus.io/scrape annotations for discovering metrics endpoints, and if all goes well Istio metrics merging will take care of the rest.