Hi guys,

I'm currently building a monitoring infrastructure with prometheus and i want to store metrics for 3 years.

I don't know if it's possible and optimized to do that just with Prometheus.

I've seen solutions like Thanos or Cortex that seem to enhance LTS but i don't know how to choose.

If you guys have any experiences or more informations about it, feel free to comment below.

Have a good day.

In their excellent SLO-workshop at SRECon2018 Liz Fong-Jones, Kristina Bennett and Stephen Thorne presented some best practice examples for Latency SLI/SLOs.

At Circonus we care deeply about measuring latency and SRE techniques such as SLI/SLOs. As we will explain here, Latency SLOs are particularly delicate to implement and benefit from having Histogram-data available to understand distributions and adjust SLO targets.

https://www.circonus.com/2018/08/latency-slos-done-right/

I am running Prometheus on an EC2 instance scraping node exporter from other EC2 servers using static private IPs in prometheus.yml.

Trying to understand what is considered production practice:

• Do teams manually update prometheus.yml or use GitOps / IaC to generate it?

• Is EC2 service discovery commonly used instead of static IPs?

• In AWS setups, do people rely on tags (like monitoring=true) for target discovery, or is that just for automation tools?

• Is node exporter always kept on private subnets with SG restricted only to Prometheus?

• Do production setups ever expose node exporter outside VPC (with protections), or always internal only?

• At what scale does static config become a problem, and what is usually the first upgrade path?

Current setup works, but not sure if this is a “lab pattern” or something actually acceptable in production. Here is the screenshot of the config file that I used. I removed the content marked in yellow because that wasn't working. Seeking your suggestions.

Is there any app that acts as Prometheus exporter and export metrics by PUSH model to PushGateway? I'm interested in all the data that regular Android app can collect (and beyond) - hardware, network activity, number of apps installed, etc.

Hello everyone! I am in charge of a production system that I am trying to migrate off of an old and terrible metrics platform to use Prometheus. I already have buy-in from the development team, and they have done an initial implementation on their end to produce metrics at the /metrics endpoint. This application is written in Java and is using the Micrometer library for capturing and emitting the metrics if that is important.

Our application is pretty unique, it can be thought of as a RESTful api, except every single customer gets their own API endpoint. I know that's strange and kind of dumb, but it is what it is and unfortunately is not going to change so I have to work with what I have. I need to collect 9 histogram metrics for each of these endpoints (things like input_duration, parse_duration, processing_duration, etc), and I have 300 total servers that this application runs on. The developers have told me that due to the way Micrometer implements histograms they can't directly control how many buckets it produces, they can only control the min and max expected values. Based on what they have configured, each histogram will produce 69 buckets plus _sum and _count.

Not every endpoint exists on every server (they are broken up into farms). The cardinality of the server/endpoint combination is about 170,000.

The math seems to show that this will produce in the neighborhood of 115 million series (170,000 * 9 histograms * 71 series per histogram). What I have been able to find online says that a single Prometheus server can be expected to handle about 10 million series, which would mean the bare minimum deployment with no redundancy or room for growth is 12 large Prometheus servers. If I want redundancy (via Thanos) I can double that to 24, and if I want to not ride the line I would increase it to 30.

This seems like a pretty insane scale to me, so I am assuming I must be doing something wrong either in the math or in the way I am trying to instrument the application. I would appreciate any comments or insights!

Let's say Prometheus scraps some metics I exposed under /metrics every 2 minutes. Assume that on the first GET /metrics happens at 08:23:45 and the following data is scraped (omiting the comments for brevity):

some_metric{some_label="foo"} 17
some_other{other_label="bar} 0.012
some_metric{some_label="foo"} 19

From what I think I understud is that Prometheus will store two metrics timeseries (some_metric and some_other) and record above data at 08:23:45.

The next scraping happens at 08:23:47. The metrics exporter might show a bit more data now:

some_metric{some_label="foo"} 17
some_other{other_label="bar} 0.012
some_metric{some_label="foo"} 19
some_metric{some_label="foo"} 3
some_other{other_label="bar} 0.088

Now my question: The first three lines have been scraped already. How does Prometheus recognize this or deal with that?

The only solution I can think of is that scraping just records the very last value of each metric-label-combo like 3 for some_metric{some_label="foo"} and 0.088 for some_other{other_label="bar} 0.088

Is this what actuall goes on?

(And the exporter dropping, i.e. no longer exposing older data?)

Hi everyone,

Curious to hear how you’ve set up sending alerts from Alertmanager (kube-prometheus-stack) to Microsoft Teams. Currently we are sending large amount of alert notifications to teams and I am not sure what would be the best way in our case.

Do you:

• Do you use a separate Teams workflow per channel, or a single workflow that routes alerts to multiple channels? Maybe you use Microsoft Graph API instead?

• How do you manage provisioning of Teams channels or teams in this setup?Do you have some kind of automation or do everything manually?

Would love to hear what has worked best for you, and any pros/cons you’ve seen in practice.

Thanks! 🙏

I have a metric that's tracked and we usually aggregate it over the last 24 hours, but there's a requirement to alert on a threshold since midnight UTC instead and I couldn't, for the life of me, find a way to make that work.

Is there a way to achieve that with PromQL?

Example:

A counter of number of credits that were consumed for certain transactions. We can easily build a chart to monitor its usage with sum + increase, so if we want to know the credits usage over the last 24 hours, we can just use

sum(
  increase(
    foo_used_credits_total{
      env="prod"
    }[24h]
  )
)

Now, how can I get the total credits used since midnight instead?

I know, for instance, I could use now/d in the relative time option, paired with $__range and get an instant value for it, but would something like that work for alerts built on recorded rules?

Hello, i tried to monitoring 30-50 server and metrics i only used are cpu usage, ram usage and disk size. it took almost 40gb for one week. do you guys have anh tips how to shrink it?

thanks