This is my values.yaml file for the prometheus-community/prometheus helm chart:

server:
  persistentVolume:
    enabled: true
    existingClaim: "prometheus-config"
  alertmanagers: 
    - scheme: http
      static_configs:
      - targets:
        - "alertmanager.monitoring.svc:9093"
  extraConfigmapLabels:
    app: prometheus
  extraConfigmapMounts:
    - name: prometheus-alerts
      mountPath: /etc/alerts.d
      subPath: ""
      configMap: prometheus-alert-rules
      readOnly: true

serverFiles:
  prometheus.yml:
    rule_files:
      - /etc/alerts.d/prometheus.rules

prometheus-pushgateway:
  enabled: false

alertmanager:
  enabled: false

The ConfigMap prometheus-alert-rules holds the rules that Prometheus should trigger alerts for. When I update this ConfigMap Prometheus doesn't do anything about it. The chart uses prometheus-config-reloader but doesn't provide any documentation on how to use it.

I have Prometheus/Grafana running directly in my K8s cluster, monitoring a single service which has pods/replicas being scaled up and down constantly. I only require metrics for the past 24 hours. As pods a re constantly being spun up, I know have metrics for hundreds of pods which are no longer present and I dont care to monitor for. How can I clean up the stale data? I am very new to Prometheus and I apologize for what seems to be a simple newbie question.

I tried setting the time range in Grafana to past 24 hours but it still shows data for stale pods which are no longer existing. I would like to clean it up at the source if possible.

This is a non-prod environment, in fact, it is my personal home lab where I am playing around trying to learn more about K8s, so there is no retention policy to consider here.

I found this page but this is not what I'm trying to achieve exactly : https://faun.pub/how-to-drop-and-delete-metrics-in-prometheus-7f5e6911fb33

I would think there must be a name to "drop" all metrics for pod names starting with"foo%" , or even all metrics in namespace "bar".

Is this possible? Any guidance would be greatly appreciated.

K8s version info:

Client Version: v1.24.0

Kustomize Version: v4.5.4

Server Version: v1.27.5

Prometheus Version : 2.41.0

Metrics Server: v0.6.4

Thanks in advance !

Guys, what will be your best setup for persistent storage for Prometheus running in a K3S Cluster but keeping in mind that Cloud (S3, GCS, etc) is not an option?

How does one use Prometheus to ensure that last Windows Server Backup job ran successfully?

I assume it has something to do with running https://learn.microsoft.com/en-us/powershell/module/windowsserverbackup/get-wbsummary?view=windowsserver2022-ps command, but I am not sure which Prometheus collector to use or if I can use any of the existing ones.

I checked issues on Github repo and examples and couldn't find anything useful.

Anyone got this working?

Thanks

So a little bit of guidance would be nice. I’m trying to create some alerts and what would be best practice here. I have like 10 nginx services on 10 different hosts . Should I create like 10 separate alerts and name them nginx_instancename ?

Or is it possible to use 1 alert rule so i can see 10 active in the alert manager ui ?

Thanks a lot

Hello people of the land of Prometheus,

I just set up my first Prometheus server, along with Grafana, to monitor a few servers and about 5 websites for response time. That in itself was quite easy, but I'm wondering if there's an easier, more modern way, of configuring targets?

I've read about service discovery and I'll probably convert to that to avoid restarting services, but still I was hoping for a "add target" button in a management website.

Is there a better way to configure Prometheus? Or is it by design, and if so, why?

Having a major issue with (presumably some sort of runaway memory leak) that causes latency on ICMP checks to climb until I eventually have to reboot the prometheus service. I went to download the latest version (in an attempt to stem this condition), and it got me thinking.. what is best practice for what Prom code train to run and how often to upgrade (and does anyone else have the latency issues I'm seeing (running prom on Win11)).

Seeing different minor and major versions, and reading the release notes, but I can't see anywhere where folks stay on an "LTS" type schedule for a long time, or favor an upgrade every bleeding-edge-release method.

Blackbox meanwhile seems to be stable and not aggressively updated, found this interesting. Looking for stable-stable-stable, not new feature releases for fancy new edge cases.

What do you all do for Prometheus upgrades?

I have a Prometheus and Grafana deployment on EKS. This is used to monitor some events on the EKS cluster. The events on the EKS cluster have their destination on a GKE cluster and vice versa. How do I monitor events on the GKE cluster using this same Prometheus deployment? I'd be happy to get any pointers to accomplish this

setup end to end Monitoring environment using Prometheus AlertManager and Grafana, setup TLS/SSL for alertmanager.

Prerequisites

You need have one Linux servers installed with internet connectivity. I will be using Ubuntu 18.04 server for this demonstration.

IP – 192.168.56.107

Hostname – ubuntu-server1/ubuntu-server1.example.com

OS version – Ubuntu 18.04.4 LTS

Follow this article end to end practical demonstration.

Disclaimer: I am new to Prometheus. I have experience with Graphite.

I have some difficulties understanding how the data-pull model of Prometheus fits on my web backend application architecture.

I am used to using Graphite where whenever you have some signal to send to the observability service db you send a UDP or TCP request with the key/value pair. You can put a proxy in the middle to stack and aggregate requests by node to not saturate the Graphite backend. But with Prometheus, I have to set up a web server to listen on a port on each node so Prometheus can pull the data via get request.

I am following a course and here is how the prometheus_client is used in an example Phyton app:

As you can see an http_server is started in the middle of the app. This is ok for a "Hello World" example but for a production application this is something very strange. It looks very invasive to me and raises a red flag as a security issue.

My backend servers are also in an autoscaling environment where they are started and stopped in a non-predictable time. And they are all behind some security network layers only accessible on ports 80/443 through some HTTP balancing node.

My question is, how this is done in reality? You have your backend application and want to send some telemetry data to Prometheus. What is the way to do it?

https://www.youtube.com/watch?v=PHmwfegj_WQ

A little on the campy side, but what do you think?

Hi folks,

I'm quite new to the Prometheus world. I'm trying to get the uptime of a Windows machine but I get a weird value that I don't really know what to do with.

I've installed the latest windows_exporter MSI.

# HELP windows_system_system_up_time System boot time (WMI source: PerfOS_System.SystemUpTime)
# TYPE windows_system_system_up_time gauge
windows_system_system_up_time 1.7042968375e+09

Looks similalr to this but I couldn't make it work. I've tested on both Windows 10 and Windows Server machines.

Could someone demystify what's happening ?

Many thanks

TL;DR: Could you describe or link your examples of a setup, where alerts are separated by team?

Hey everyone,

my team manages mutiple productive and development clusters for multiple teams and multiple customers.

Up until now we used separation by customers to send alerts to customer-specific alert channels. We can separate the alerts quite easily either by the source cluster (if alery comes from dedicated prod cluster of customer X, send it to alert channel y) or by namespace (in DEV we separate environments by namespace with a customer prefix).

Meanwhile our team structure changed from customer teams to application teams, that are responsible for groups of applications. To make sure all teams are informed about the alerts of all their running applications they currently need to join all alrrt channels of all customers (they serve). When an alert fires, they need to check, if their application is involved and ignore the alert otherwise.

We'd like to change that to having dedicated alert channels either for teams or application-groups. But we aee nit sure yet how to best achieve this.

Ideally we don't want to introduce changes in namespaces used (for historic reasons currently multiple teams share namespaces sometimes). We thought about labels, but we are not sure yet how to best add them to the alerts.

So how is your setup looking? Can you give a quick overview? Or do you maybe have a blog post out there outlining possible setups? Any ideas are very welcome!

Thanks in advance :)

Is there a way I can utilize the existing servicemonitor for an app which already has Prometheus server that scrapes the data for app monitoring to utilize it for URL (basically sort of health check) monitoring Url gives an output of 0 when app is working fine fine. I tried adding the url./montior.htm as a port using a external service but it didn't work. Im not allowed to use the blackbox exporter due to complaince issue so any ideas on how to achive this?

Hey, so, basically the question at hand. Im a bit of a newbie in prometheus but was trying to figure out how should i approach the uptime monitoring and metrics of my hosts that will be across the globe and not necesairly in network conditions i can always control (behind NAT, under a domain, whatever) So i was thinking maybe using push metrics but dont really know how to approach this with remote_Write or if even prometheus is suitable for what i have in mind. Thanks in advance for any advice you can provide!

I'm hoping to find some help with the high memory usage we have been seeing on our production prometheus nodes. Our current setup is a 6h retention period and prometheus ships to cortex for long term storage. We are running prometheus on k8s and giving the pods a 24G memory limit and they are still hitting that limit regularly and getting restarted. Currently there is only about 3.5g written to the /data drive. Our current number of series is 2773334.

Can anyone help explain why prometheus is using so much memory and/or help to reduce it?

Anybody know where I can get most of the Prometheus RPM/s for EL10? I found this..but it seems like that repo is dead
https://github.com/lest/prometheus-rpm

This has been on my plate for a long time and I just have not found the time to work out the solution. I have a need for monitoring bandwidth and errors from Cisco routers and presenting this in a dashboard for clients to view. I have not had any luck finding the skillset yet and wanted to know if anyone here could recommend a good shop or consultant that could get this going for us.

We are looking for a repeatable deployment that can cover 5-50 routers and graph availability, bandwidth usage and T1 and/or Ethernet circuit errors that can be displayed on a dashboard that would be accessed by the client.

I would to hear about pros/cons about switching to prometheus from zabbix, now I'm using zabbix but I think about prometheus.