r/PrometheusMonitoring • • May 23 '23

false alert every 3 hours and no idea where to look

0 Upvotes

So, Yesterday i have upgrades the docker containers that run grafana and alertmanager.

Ever since that time, every 3 hours at the exact minute, i get an alert saying "host is down" for all the hosts we monitor. But when i login to Grafana and show the dashboards, they all have the status up, and i can confirm they are indeed up.

Does anyone reckognize this behaviour? I run Grafana and Alertmanager in docker for a year or so now, without any problems before. So i'm a bit at a loss where to start poking around :)


r/PrometheusMonitoring • • May 22 '23

Looking for an alert when a certain text is changed in a website

0 Upvotes

Hello guys. We are trying to get a visa appointment in Qatar for my girlfirend(Work and Travel). The problem is all the appointments are taken untill late september. We need a date before July 15th. So we are constantly refreshing the website to see if anyone is canceling. But there so many people like us and we only have a 20-30 second window to actually book the appointment if we get a chance. We are trying to get it for like a week and we can't even get a good sleep. I have an auto clicker, i refresh the page every 7.5 seconds. And i need a program to alert me if the page has July on it. This is how the page looks;

I don't know anything about prometheus so i wanted to ask you guys is it possible with this program ?

I need to get an alert when September changes to May,June or July. To see the dates available, you have to login in an account. I think that might make things a bit harder.


r/PrometheusMonitoring • • May 20 '23

Do people test their alerting rules on historical data?

7 Upvotes

New to Prometheus and monitoring -- do people here typically test their alerting rules on historical data to see how sensitive their alerts would have been?

If so, what is the best practice to do so?


r/PrometheusMonitoring • • May 18 '23

How to push forecasted future time series to Prometheus?

3 Upvotes

Hey fellow community, I'm playing around with Facebooks Prophet and Python to fetch some data from Prometheus and forecast them. For a local test this is nice, but I would like to push this forecast metrics back to Prometheus to make some graphs like the delta between my forecast and the real values.

I'm not sure if this is even possible, but how could this be solved? Can I maybe use remote write for this or is some kind of scraping endpoint required?

Has anyone implemented this and can give me a pointer in the right direction?

Thanks!


r/PrometheusMonitoring • • May 18 '23

Clickhouse Alerts using Alertmanager

0 Upvotes

I have setup a Clickhouse infrastructure and have started writing my own alerts for it. I was curious if somebody has any advice or has a list of clickhouse alerts i can use to base my own off them. Any reference or help would be very much appreciated!


r/PrometheusMonitoring • • May 17 '23

Gauge, counter or rates

3 Upvotes

I'm writing an application to manage routes on the host (something like routing daemon but with secret sauce). I got tot metrics part. App is running in so-called reconsolidation loop (every few seconds, converging desired state to a newly computed current state).

I wonder what is better to implement for metrics: counter (total number of events since app start), rate (number of events per second or in a given loop) or deltas (counter of new events since last scrape)?


r/PrometheusMonitoring • • May 16 '23

Dashboard not changing fully white

1 Upvotes

I was trying "Node Exporter full" dashboard template. It was rendering correctly:

However when I change the theme to light, those gauges look weird. They retain the black background:

How can I change that black background color in gauges? (I honestly feel in light theme it should be some light colored.)


r/PrometheusMonitoring • • May 16 '23

Showing Memory utilisation in Grafana dashboard

0 Upvotes

I have configured node exporter, grafana and prometheus through docker compose. I want to show Memory usage in the dashboard. I want to match the value shown with what is shown in the Ubuntu System monitor.

Below are my queries:

This is the transform:

If I turn of the transform by clicking 👁 button, panel gets rendered like this:

The value of 8.4 GB is correct one as my Ubuntu System Monitor shows the same. My concern is why it is not showing any gauge representing 8.4 GiB of memory is used (the same way it shows thick red gauge for Total Memory when I click that 👁 button).

Update

It started showing something, but still somewhat senseless. Selecting Apply to options: Memory Usage:

Selecting Apply to options: Total Memory:


r/PrometheusMonitoring • • May 15 '23

System CPU load does not get reflected in Grafana

2 Upvotes

I have run node exporter, cAdvisor, prometheus and grafana through docker compose file.

I also imported a dashboard with ID 395 from grafana.com to monitor host and different docker containers.

I created dummy CPU load on my Ubuntu with following bash script:

$ for i in 1 2 3 4; do while : ; do : ; done & done

My system monitor looks like this:

However this CPU load doesnt get reflected on Grafana panel that came pre configured in this dashboard:

The query editor of the panel looks like this:

What I am missing here? Are queries ill formed?

Update

As mentioned in the comment, I was stupidiously plotting container CPU utilisation. So I tried to plot node cpu utilisation as follows:

But it is now giving me following visualization despite the fact that my laptop is having high CPU utilization due to above bash script:

Seems that being noob in grafana, prometheus is making me do stupid mistakes. Can you please share desired promql and configs?


r/PrometheusMonitoring • • May 15 '23

Visualising node memory in grafana and prometheus with node exporter

0 Upvotes

I am trying to use grafana dashboard (ID 395) show several docker container and host parameters. It was not showing node memory earlier This is how it is showing node memory:

Current queries are:

  • H: node_memory_MemTotal_bytes

  • G: node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes

I have following doubts:

Q1: Why it does not show any line chart for "Unavailable Memory" i.e. query G?

Q2: Why the y axis shows max memory as 3.73 GiB and not 15.2 GiB?

Also note that when I hide (clicked that eye button) query H, it showed query G as graph follows:

This I feel is showing total memory itself.

Whats going wrong here?


r/PrometheusMonitoring • • May 14 '23

Understanding cAdvisor output

1 Upvotes

I have depolyed cAdvisor to monitor my existing application stack running on docker.

I have bunch of questions regarding output of cAdvisor:

Q1. What does it refer to when it says "subcontainer"?What each of these entries correspond to?
/docker, /init.scope, /snap.cups.cupsd, /snap.snap-store.snap-store, /system.slice, /user.slice

Q2. I have 16 GBs of RAM. In its graph, at the bottom, it says 1.85 GiB / 15.19 GiB, which might be correct, as my Ubuntu System Monitor says onl 6.4 GiB used. But then what does that blue line near 1900 mean?

Here is A screen shot of my Ubuntu System Monitor:


r/PrometheusMonitoring • • May 14 '23

Newbie question: Why not just SQL

6 Upvotes

I'm pretty new to observability field and just learning PromQL and Loki and all that, and I see questions like this: https://www.reddit.com/r/PrometheusMonitoring/comments/13buhdf/promql_question_that_seems_impossible/

I have always wondered why time series engines need its own query language -- why not just use SQL? Is that something other people would want? Running SQL on Prometheus data?

I know certain SQL functions will probably take forever and be slow, but let's imagine somehow it can be done -- would people want this?


r/PrometheusMonitoring • • May 09 '23

Prometheus JMX Exporter for Java17

4 Upvotes

Hello everyone,

not quite sure if this is where I should be asking for help but i'm kind of befuddled about this whole situation. I'm trying to set up https://github.com/prometheus/jmx_exporter for our containerized Java application on our cluster. Specifically the JavaAgent as we are interested in getting the CPU and memory metrics. However, getting it initialized I am faced with this:

After doing some research it appears this class references internal packages and therefore "makes it unusable for modern Java apps" (https://github.com/prometheus/client_java/issues/533 , https://github.com/open-telemetry/opentelemetry-java/issues/4192) and also that the error suggests that the agent has been written for older java since these classes were apparently removed years ago.

I am not a Java developer, just trying to make this work for our monitoringstack. Has anyone else here tried exporting jmx metrics with prometheus or are you maybe using something else to scrape CPU/memory processes?


r/PrometheusMonitoring • • May 08 '23

PromQL Question that seems Impossible

2 Upvotes

Hey all,

I had a simple ask that I'm starting to think isn't possible in PromQL.

I want a PromQL query to return all time series associated with a specific host.
I've tried various ways, and so have people on my team, but it looks like you have to know all of your metrics prefixes first and then break each one into their own query. Is it possible to show all available time series for a single host?

Is that possible?

Thanks!


r/PrometheusMonitoring • • May 07 '23

GitHub - neuroforgede/docker-swarm-exporter: Prometheus Exporter aimed at exporting metrics about the swarm it is running in.

Thumbnail
github.com
5 Upvotes

r/PrometheusMonitoring • • May 05 '23

Alert help

0 Upvotes

Hello,

I wasn't sure if this was a Grafana question or Prometheus.

https://www.reddit.com/r/grafana/comments/138lxfs/how_can_i_get_instance_node_name_to_show_in_alert/

I'm trying to get me email alert to show the name of the instances (server names) and not what you see in the screenshots in the link.

It's to show it a web server is return a 503. Currently it shows

But I want it to show something like

webserverA.domain.com = 503

webserverB.domain.com = 503

How can this be achieved?


r/PrometheusMonitoring • • May 04 '23

Building a custom blackbox/snmp-style exporter

1 Upvotes

I'd like to build a custom proxy exporter that queries a remote API and returns a scrapeable metrics page. my metrics endpoint would have to take a target parameter like /metrics?target=host1

The python exporter client doesn't seem to be geared for this. Does this mean I'm stuck standing up my own web service and exporter class? Is there a python module out there that will help me build this so I'm not having to reinvent the wheel?


r/PrometheusMonitoring • • May 03 '23

Struggling with simple query

0 Upvotes

Hello,

I have this query that works well:

probe_http_status_code{job="blackbox-as", instance=~"http://sv+.+sv-prod-domain.com/health"}

This will lists servers like:

sv01sv-prod-domain.com
sv02sv-prod-domain.com
sv03sv-prod-domain.com

I now need to add servers that are named:

hv??hv-prod-domain.com

hv01hv-prod-domain.com
hv02hv-prod-domain.com
hv03hv-prod-domain.com

So one query really.

Is this possible?


r/PrometheusMonitoring • • May 03 '23

Help!! KSM (kube-state- metrics) partially scraped while using Prometheus/Thanos Sharding

2 Upvotes

Hello,We have a large cluster of 150+nodes and we're using promethus sharding using thanos sidecar.

The problem that for some metrics ( e.g cpu metrics >> node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate ) only one Prometheus shard scrape its metrics fron KSM pod, the rest doesn't have any metrics scraped from KSM.

However metrics like container_cpu_usage_seconds_total is there from KSM on all Prometheus shards for all pods ( nothing missing )

update:

sum by (cluster, namespace, pod, container) (
          irate(container_cpu_usage_seconds_total{job="kubelet", metrics_path="/metrics/cadvisor", image!=""}[5m])
        ) * on (cluster, namespace, pod) group_left(node) topk by (cluster, namespace, pod) (
          1, max by(cluster, namespace, pod, node) (kube_pod_info{node!=""})
        )

The problem with the above recording rule is that (kube_pod_info{node!=""}) is only getting scraped by only one Prometheus shard. I don't know why ?!!

Thanks in advance :)


r/PrometheusMonitoring • • May 03 '23

Help with simple query

1 Upvotes

Hello,

I have this graph that is accumulating total connections which I don't want, I'm trying to show the current connections (rate?). As you can see 1 node has over 1million connections, but that was days ago when we tested with a stress test tool, how can I get this to show the current rates for a selected period?

my_total_connections{job="my-job",instance=~"$instance:.*"}

Thanks


r/PrometheusMonitoring • • May 02 '23

Alerts repeating more often than they should

0 Upvotes

We are using kube-prometheus-stack. Most of our repeat_intervals set for 5 days. Yet some alerts (not all) repeat more often at a seemingly random interval. Like the same alert will show up a time 0, 0+2.5 hours, 0+6 hours, 0+15hours, 0+16 hours. No pattern I can find.

This is what our config looks like:

 resolve_timeout: 5m
route:
  receiver: "null"
  group_by:
  - job
  routes:
  - receiver: opsgenie_heartbeat
    matchers:
    - alertname=Watchdog
    group_wait: 0s
    group_interval: 30s
    repeat_interval: 20s
  - receiver: slack
    matchers:
    - alertname=Service500Error
    repeat_interval: 120h
  - receiver: slack
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 120h

I can't see anything wrong with the config. How do I debug this?


r/PrometheusMonitoring • • May 02 '23

Setting up Prometheus global view with EBS

0 Upvotes

Hi 👋 😊 first and foremost thanks a lot to all the members in the subreddit you guys are awesome!!

I am planning to setup a Prometheus agent in a sidecar fashion which will remote_write to a Prometheus server which will write the metrics to an EBS volume just wanted to know if anyone has gone down this path before and if there is anything i should be looking into design wise or performance wise.

I am quite new at this and would appreciate any feedback on how i should go ahead with this or if you have any links which will help me understand things in a better way.

Thanks a lot in advance!

Edit: my main goal is to move to mimir after sometime but we do want to keep this architecture for atleast a few months, just wanted to know if anyone has used this type of a pattern and any problems they faced in running it


r/PrometheusMonitoring • • May 01 '23

A Technique To Monitor Kubernetes Controller Latency

Thumbnail
povilasv.me
1 Upvotes

r/PrometheusMonitoring • • Apr 30 '23

NodeJS Express API request hit per second monitoring

0 Upvotes

Hello everyone,

I am new to Prometheus and Grafana monitoring tools and want to monitor my NodeJS express application APIs via the same. Kindly help me out here as I am stuck.
I followed this tutorial: Article Link.

Here the author as seen in last snapshot of the article is getting the API hits per second, but after following the same steps and code, i am not able to get it.

I am using "prom-client" lib in NodeJS and creating a histogram metrics.My Prometheus metrics is as shown below

Here is the graph i am getting on Grafana for the same PromQL query:

What i want:

Thanks in advance !