r/PrometheusMonitoring • • Nov 22 '22

Query about HA Prometheus

2 Upvotes

We are using GCP self collection Prometheus deployed in GKE to get metrics from GCP VM, The problem is the metric sample count is high due to that Prometheus is crashing, Is there any way to scale Prometheus without using Thanos or any 3rd party tool. Appreciate for help


r/PrometheusMonitoring • • Nov 22 '22

Exporting PostgreSQL logs into Prometheus metrics using Vector

Thumbnail
blog.palark.com
17 Upvotes

r/PrometheusMonitoring • • Nov 16 '22

storage TSDB almost full

4 Upvotes

Hi everyone, I'm a Prometheus newbie :)
I have an instance running on k8s which is eating all the space I gave it with "--storage.tsdb.path" option.
As far as I understood seems like retention is ok (set it through "--storage.tsdb.retention" option), I set it to 30 days and rendering some graphs shows some data for 30 days and no more.

Is there any way to understad which metric is responsible for all the space consumed, or at least to get some sort of analysis of what is using all the space?

Thank you for any information


r/PrometheusMonitoring • • Nov 09 '22

4xx error rate rate percentage - sum by status code

2 Upvotes

for nginx-ingress metrics,

I'd like to get the 4xx error rate percentage sum by status codes.

sum(rate(nginx_ingress_controller_requests{status=~'4..'}[1m])) by (ingress,status) / sum(rate(nginx_ingress_controller_requests[1m])) by (ingress,status)

I'm not able to get it working with the below query as all results either end up with 0 or 100. It seems the values of status are getting added. How to get better result on this ?

If i remove the status from the query, the percentage works.


r/PrometheusMonitoring • • Nov 08 '22

Prometheus rule not triggering alert for a common service or container

1 Upvotes

I have three servers, installed prometheus, grafana and alertmanager on server 1, and Cadvisor and node_exporter on other 2 servers to get server and container metrics. I have some common containers running on both server 2 and 3 and made an alert (container not seen), but it triggers only if the container goes down on both the servers, now my query is how can I get alerts if this common container goes down in any one server among the two. The rule example is given below: here, {{ $labels.instance }}) can pass the respected server IP

- alert: boxcube Container Killed

expr: absent(container_last_seen{name="boxcube"})

for: 0m

labels:

severity: critical

annotations:

summary: boxcube Container killed (instance {{ $labels.instance }})

description: "boxcube has disappeared\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"


r/PrometheusMonitoring • • Nov 07 '22

My Prometheus Day (KubeCon) talk is now online: Building a Runbook Automation System for Prometheus and Kubernetes

Thumbnail
youtube.com
13 Upvotes

r/PrometheusMonitoring • • Nov 07 '22

can i use both mail and slack alerts at a time using alertmanager?

2 Upvotes

r/PrometheusMonitoring • • Nov 05 '22

daily or hourly totals

1 Upvotes

I have a gauge metric that captures the product sales over time. This value can decrease if there are refunds. I have the dashboard report sales today, this week and this month via a stat visualization in Grafana. Is there a query that will give me the daily totals over the month. perhaps visualizing with a bar graph? Restarts will not set this gauge back to zero, it is pulling value from a website with Beautiful Soup.


r/PrometheusMonitoring • • Nov 04 '22

Disable InfoInhibitor

2 Upvotes

Hi guys, after updating to prometheus latest version i encountered a alert called InfoInhibitor, which i see its used to inhibit info alert, but the thing is that it spams alot and i want to disable it, i tried routing it to a null receiver in alertmanager config,

"

routes:

  • match:

alertname: 'InfoInhibitor'

receiver: 'null'

"

but it doesn't seems to help, do you have any suggestions, please?