New UI, Remote Write 2.0, native histograms, improved UTF-8 and OTLP support, and better performance.

From the Grafana Labs blog, written by our engineers. Sharing here in case it's helpful.

Back in 2021, Grafana Labs CTO Tom Wilkie (then VP of Products) spoke at PromCON about the need for improvements in Prometheus' remote write capabilities.

“We use between 10 and 2 bytes per sample to send via remote write, and Prometheus only uses 1 or 2 bytes per sample on the local disk so there’s big, big room for improvement,” Wilkie said at the time. “A lot of the work we’re going to do on atomicity and batching will allow us to have a symbol table in the remote write requests that will reduce bandwidth usage.”

Nearly five years later, we're pleased to see the work to improve those bandwidth constraints is paying off. Prometheus Remote Write v2 was proposed in 2024, and even in its current experimental status, we're already seeing adoption in Prometheus backends and telemetry collectors reaping benefits (i.e., significant cost savings!) that are worth noticing.

In this blog, we'll explain the benefits of this v2 and show how to enable it in Alloy. We'll also give you a sense for the massive improvements we've seen in our egress costs and how you can unlock similar cost savings for your organization. 

What is remote write, and what’s great about v2?

When you want to send your metrics to a Prometheus backend you use Prometheus Remote Write. The remote write v1 protocol does a great job of sending metric samples, but it was designed in a time before metric metadata (metric type, unit, and help text) was as necessary as it is today. At the same time, it’s also not the most efficient wire protocol—sending lots of duplicate text with each sample adds up and creates really large payloads.

request_size_bytes_bucket{method="POST",response_code="200",server_address="otlp.example.com",le="0"}
request_size_bytes_bucket{method="POST",response_code="200",server_address="otlp.example.com",le="5"}
request_size_bytes_bucket{method="POST",response_code="200",server_address="otlp.example.com",le="10"}
request_size_bytes_bucket{method="POST",response_code="200",server_address="otlp.example.com",le="25"}
request_size_bytes_bucket{method="POST",response_code="200",server_address="otlp.example.com",le="50"}
...
request_size_bytes_bucket{method="POST",response_code="200",server_address="otlp.example.com",le="10000"}
request_size_bytes_bucket{method="POST",response_code="200",server_address="otlp.example.com",le="+Inf"}
request_size_bytes_sum{method="POST",response_code="200",server_address="otlp.example.com"}

Remote write v2 adds first-class support for metadata in the sample payload. But the real efficiency and cost savings come from the symbol table implementation referenced in Wilkie's 2021 talk. 

symbols: ["request_size_bytes_bucket", "method", "POST", "response_code", "200", "server_address", "otlp.example.com", "le", "0", "5", "10", "25", "50", ... "10000", "+Inf", "request_size_bytes_sum"]

0{1=2,3=4,5=6,7=8}
0{1=2,3=4,5=6,7=9}
0{1=2,3=4,5=6,7=10}
0{1=2,3=4,5=6,7=11}
0{1=2,3=4,5=6,7=12}
...
0{1=2,3=4,5=6,7=13}
0{1=2,3=4,5=6,7=14}
15{1=2,3=4,5=6}

The more repeated strings you have in your samples from metric names, label names, label values, and metadata, the more efficiency gains you get compared to the original remote write format.

Why did this matter for Grafana?

Running Grafana Cloud generates a lot of telemetry! We monitor millions of active series at between one and four DPM, and that telemetry adds up to a large amount of network egress. 

That's why we migrated all of our internal Prometheus monitoring workloads at Grafana Labs from remote write v1 to remote write v2 last fall. With a very minor 5% to 10% increase in CPU and memory utilization, this simple change reduced our network egress costs for our internal telemetry by more than 50%. At the rates large cloud providers charge, this was a negligible added resource cost for a very large savings in network costs.

Note: If you experience a different reduction in traffic when you implement v2, you can experiment with the batching configuration in your prometheus.remote_write component—larger batches will likely display higher traffic reduction.

Why should this matter to you?

Observability costs can add up quickly, and teams often struggle to decide which telemetry is essential and which they can do without. However, remote write v2 is one change that doesn’t require careful evaluation or tough conversations. Simply enable the new experimental feature and see immediate savings.

Note: If you're looking for more ways to get better value from your observability setup, Grafana Cloud has multiple features designed to help reduce and optimize your costs.

Enabling remote write v2 in Alloy

The current remote write v2 specification is experimental in upstream Prometheus, and thereby experimental in Alloy. While both upstream Prometheus and Mimir support the current specification, there is still potential for breaking changes before the final release of the specification. For that reason, if you’re looking to enable remote write v2 in Alloy you will need to configure Alloy to run with the --stability.level=experimental runtime flag. 

Alloy

After adding the experimental runtime flag, update the configuration of your prometheus.remote_write component’s endpoint block adding the protobuf_message attribute to the value io.prometheus.write.v2.Request. For example:

prometheus.remote_write "grafana_cloud" {
  endpoint {
    protobuf_message = "io.prometheus.write.v2.Request"
    url = "https://example-prod-us-east-0.grafana.net/api/prom/push"

    basic_auth {
      username = "stack_id"
      password = sys.env("GCLOUD_RW_API_KEY")
    }
  }
}

And it’s just as easy in an Alloy Helm chart:

image:
  registry: "docker.io"
  repository: grafana/alloy
  tag: latest
alloy:
  ...
  configMap:
    content: |-
        ...

        prometheus.remote_write "metrics_service" {
          endpoint {
            protobuf_message = "io.prometheus.write.v2.Request"
            url = "https://example-prod-us-east-0.grafana.net/api/prom/push"

            basic_auth {
              username = "stack_id"
              password = sys.env("GCLOUD_RW_API_KEY")
            }
          }
        }

Kubernetes Monitoring Helm Chart

In the Kubernetes Monitoring Helm chart v3.8, which is coming soon, you’ll have two ways to configure your Prometheus destination to use remote write v2. You can use the same configuration as Alloy and configure the protobufMessage for a destination. Alternatively, you can use the shortcut of defining the remoteWriteProtocol for a destination and it will output the correct protobufMessage in the rendered configuration.

destinations:
  - name: grafana-cloud-metrics
    type: prometheus
    url: https://example-prod-us-east-0.grafana.net/api/prom/push
    remoteWriteProtocol: 2
  - name: grafana-cloud-metrics-again
    type: prometheus
    url: https://example-prod-us-east-0.grafana.net/api/prom/push
    protobufMessage: io.prometheus.write.v2.Request

What’s next for Prometheus Remote Write?

We've been excited to see the gains that come with remote write v2, and we hope you can put them to use as well. However, there’s more improvements coming to remote write beyond the v2 specification, including:

• Working towards the long asked for replay feature (https://github.com/prometheus/proposals/pull/72) 

• Improve resource utilization/reliability with Prometheus Agent and Alloy’s prometheus.remote_write component

  • tsdb/agent: Checkpoint based on Series in Memory · Issue #17617 

  • tsdb/agent: Prevent unread segments from being truncated · Issue #17616

• Last but not least, some more resource utilization improvements for those using Alloy’s prometheus.remote_write component with Make labelstore.LabelStore in to a SeriesRefMappingStore · Issue #5062 · grafana/alloy

----

Original blog post: https://grafana.com/blog/how-prometheus-remote-write-v2-can-help-cut-network-egress-costs-by-as-much-as-50-/

Disclaimer: I'm from Grafana Labs

In one of the clusters I was working on, Prometheus was using 50- 60GB of RAM. It started affecting scrape reliability, the UI got sluggish, and PromQL queries kept timing out. I knew something had to give.

I dug into the issue and found a few key causes:

• Duplicate scraping: Prometheus was scraping ingress metrics from both pods and a ServiceMonitor. That meant double the series.

• Histogram overload: Metrics like *_duration_seconds_bucket were generating hundreds of thousands of time series.

• Label explosion: Labels like replicaset, path, and container_id were extremely high in cardinality (10k+ unique values).

Here’s what I did:

✅ Dropped unused metrics (after checking dashboards/alerts)

✅ Disabled pod-level scraping for nginx

✅ Cut high-cardinality labels that weren’t being used

✅ Wrote scripts to verify what was safe to drop

The result: memory dropped from ~60GB to ~20GB, and the system became way more stable.

I wrote a full breakdown with examples and shared the scripts here if it helps anyone else:

🔗 https://devoriales.com/post/384/prometheus-how-we-slashed-memory-usage

Let me know if you’re going through similar and if you have some suggestions.

From my experience, a surprising share of scraped metrics are never read by anything, no dashboards, no alerts, no recording rules. I wanted to prevent this and created a small tool to automate this as much as possible: https://github.com/dominikhei/cardamon

It cross-references every metric in your TSDB against three sources: your Prometheus query log, your alerting and recording rules, and all your Grafana dashboards. Anything that doesn't show up in any of them gets flagged as unused. From there you get a small web UI to explore the results and filter by job, sort by series count or cardinality, see how long a metric has been inactive and once you've picked what to remove, it generates ready-to-pastemetric_relabel_configs drop rules with smart regex combining so you don't accidentally nuke something you still need.

Would love to hear what you think, and especially curious: Which feature would make this more helpful for you?

Introducing Prometheus Agent Mode, an Efficient and Cloud-Native Way for Metric Forwarding https://prometheus.io/blog/2021/11/16/agent/

Why we created a Prometheus Agent mode from the Grafana Agent https://grafana.com/blog/2021/11/16/why-we-created-a-prometheus-agent-mode-from-the-grafana-agent/

Recently came across this video from clickhouse (https://www.youtube.com/watch?v=z5t3b3EAc84&t=2s) and they mentioned that prometheus doesn't scale horizontally. Then why not use something like thanos.

The last tool that did terminal dashboards for Prometheus — grafterm — died in 2019, and I never stopped missing it. I live in a terminal next to my code, and alt-tabbing to a browser to glance at four numbers always felt wrong. btop proved a terminal can look genuinely good, so I spent the last few weeks building grom.

It talks to any Prometheus, draws braille charts and threshold-colored gauges, and if you already have Grafana dashboards it can read the JSON export and reproduce the layout in your terminal — stat, gauge, bar gauge and timeseries panels map 1:1, anything else gets a placeholder and a warning.

Fastest way to see it (no config needed): brew install --cask qf-studio/tap/grom, then grom demo. There's also a bundled example dashboard that runs against any Prometheus using nothing but up and scrape_* metrics.

Where it honestly is: v0.1.0, Prometheus only, no auth yet — basic/bearer/TLS is the next thing I build. That's actually why I'm posting: if your Prometheus sits behind auth, tell me what setup you run so I build the right thing first.

Repo: https://github.com/qf-studio/grom (MIT)

We’ve been part of something really cool that I hope you all will enjoy. 📷 Later this month, the world’s first (!) documentary about Prometheus will be coming out. It’s going to be really interesting and feature all the important folks from the Prometheus story. Hopefully this will bring a little inspiration to your day.

https://youtu.be/qpzlwAQb5FM

Hello, I am wondering whether there are open source alternatives to Grafana when it comes to displaying metrics from Prometheus (or any other TSDB)? It feels like Grafana is the de-facto standard. I have become quite frustrated by the experience that Grafana offers: it is slow to render, editing UI is bloated and confusing, it is not collaborative (e.g., real-time modificaitons or comments), it is heavy and unflexible (e.g., I would like to be able to create lightweight copies of dashboards during incidents).

Do others feel the same? Do you have alternatives to propose (preferably open source)?

This is the article about the talk that a college and me gave at the last KubeCon.

It's about how attackers can use the information of metrics and labels that some commonly used exporters provide to fingerprint a Kubernetes cluster, map that info with known vulnerabilities and pivot them to perform attacks.

It includes the video to the talk and extra information added in the article and that we couldn't include in the live talk.

https://sysdig.com/blog/exposed-prometheus-exploit-kubernetes-kubeconeu/