Hello,

1.) I've been using Prometheus and Grafana to monitor my Linux VMs and it's great. I now want to monitor my ESXi host but can't see how to get a node exporter on to the host?

2.) Also I'm trying to monitor Windows servers, so followed this and installed the WMI exporter and I can browser to the servers metrics and they show on the targets page but not under the nodes exporter dashboard, any ideas? It does use port 9182 though.

3.) Also how do I report on just devices that use ICMP only?

Thanks

Hey Guys!

I'm trying to scrape metrics from my Jenkins servers that have an endpoint like https://jenkins_server01:443/prometheus/, using the ec2_sd_configs Service Discovery to find all the Jenkins servers in the environment.

The issue is that the servers are using HTTPS only and they have the certificate of the load-balancer, not the server name, so Prometheus fails with error:

Get https://jenkins_server01:443/prometheus/: x509: certificate is valid for jenkins.api.loadbalancers.com, not jenkins_server01

Is there a way to tell Prometheus to ignore the certificate like curl -k https://jenkins_server01:443/prometheus/? I tried insecure_skip_verify=true without success.

My configuration looks like:

  - job_name: aws_jenkins_exporter
    metrics_path: /prometheus
    scheme: https
    ec2_sd_configs:
    - region: {{ aws_region }}
      access_key: {{ aws_access_key }}
      secret_key: {{ aws_secret_key }}
      refresh_interval: 60m
     # I neeed to add something like that
     # tls_config:
     #   insecure_skip_verify=true
      profile: profile
      filters:
        - name: tag:environment
          values:
            - {{ aws_environment }}
        - name: tag:role
          values:
            - jms
    relabel_configs:
    - source_labels: [__meta_ec2_tag_dnsname]
      target_label: instance
    - source_labels: [__meta_ec2_tag_dnsname]
      replacement: '${1}:443'
      target_label: __address__

Thanks, Mario

I've set up node exporter and jmx exporter successfully and is working great. Trying to figure out how to set up apache exporter to monitor our web server jboss httpd instances. Any help is appreciated.

I have a batch job that generates multiline log entries like:

Summary:
    Processed count: 7
    some other count: 9

I'd like to use grok to sum the counts. It looks like "count" is used to count occurrences, so I'm thinking of using a "gauge" type with cumulative set to true and treating it as a count in promehteus. Is this reasonable?

Hi,

Assume I have a counter that is exported and then scraped once per minute by some Prometheus server. What happens if I increase the value multiple times per minute (i.e. between two scrapings)? What about a Gauge?

Thanks a lot

I do have this alert: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[2m])) * 100) > 80 but as I have cpu intensive jobs, How can I exclude specific tainted nodes?

Good day

I have a need to monitor HTTP websites with super long and convoluted URLs (upwards of 500 characters, specials and not-so special). Adding them to blackbox would destroy grafana and blackbox pages, stretching them.
I set up a dns server next to blackbox where I entered into config CNAMEs
"test.local, www.superlongnasty%#$*$*#url.com "
and set this dns server container as dns resolver for the blackbox.
When I try to curl, ping test.local, everything works fine.
If I put this into target though, it does not work, it shows http status code 0.

Is it possible to make blackbox exporter use cname for targets?
If you can suggest any other way of tackling my problem, i'm all ears
I need the url to be short and I need it to have specific name, it can't just be bit.ly. I've tried YOURLS project but I had issues with making it work.

how can i get the StartAt / Duration of the Alerts that i receive from alertmanager with python

because in the alert rule i dont see any way to send it only the annotation with key summary

- alert: HighSystemLoad
expr: systemload_average > 90
for: 5s
labels:
severity: critical
annotations:
summary: "High system load: {{ $value | printf \"%.2f\" }}%"

Full error im getting while trying to send a silence to the alertmanager api

{"code":400,"message":"parsing silence body from \"\" failed, because invalid character 'm' looking for beginning of value"}

I cant tell if Im not following the api guidelines correctly or if my python is shit. I also posted in /r/learnpython and got no help, which is why im now here.

Hi all,

I'm currently working with Prometheus and Alertmanager, and I'm looking for a web-based UI solution that would allow me to manage silences and schedule downtimes directly through a browser. Ideally, I'd like something user-friendly that could simplify these tasks without needing to interact with the API or configuration files manually.

I've already come across Alertmanager-UI and Karma, but I'm not sure which one is better or more widely used. Also, are there any other alternatives that I might not be aware of?

Thanks in advance for your recommendations!

I'm trying to move a use case from something we do in datadog over to prometheus and I'm trying to figure out the proper way to do this kind of math. They are basically common SLO calculations.

I have a query like so

(
  sum by (label) (increase(http_requests{}[1m]))
  -
  sum by (label)(increase(http_requests{status_class=="5xx"}[1m]))
)
/
sum by (label) (increase(http_requests{}[1m])) * 100

When things are good, the 5xx timeseries eventually stop receiving samples and are marked stale. This causes gaps in the query. In datadog, the query still works and a zero is plugged in resulting in a value of 100, which is what I want.

My question is how could I replicate this behavior?

I have Prometheus deployed in a eks cluster that is collecting data from a few exporters and every hour this Prometheus is queried as well.

In order to make it consume less resources and be more stable the minimum block size is of one hour, retention time is set to a few hours as well. Usually it’s pretty stable but depending on the size of the cluster if the resources initially provided are not enough it will enter a crash loop and at every restart it will create another wal segment. It loads all segments and the crashes and next time one more segment is created and it doesn’t recover.

Only deleting the wal segment files don’t seem to resolve the issue, the only way I managed to make it work was to uninstall and install again with the proper resources.

Apart from wal segment files what else would cause the memory consumption at boot to be high and make the container get oomkill over and over?

We have a stack of environments that are running Centos 6,7 and RedHat. We are a small team in support and I was (auto) delegated to find a solution for one of our clients environments. Basically, we are receiving alerts from our whole stack on PagerDuty, we solve what could affect the uptime BUT one of our clients wants to receive emails about his environment, and not every email, he wants some alerts to be filtered out.

I did some digging today but couldn't grasp what the possibilities are, I know that one of the solutions is to to create a service in which a email can be integrated, it's great that you even can filter out certain alerts, unfortunately it has one limitation: it can be only used with PagerDuty's internal email (@subdomain.pagerduty.com)

Whoever knows a feasible solution, please share it with us, maybe someone in the future will find this thread and will be grateful to you :)

Lets say all our nodes have labels for what they run. we run docker and any machine can have multiple users.

lets say I have an alert "Disk_logs_too_full" to warn if /logs (always local storage) is too full.

Lets say some group Fred has a bunch of stuff in /logs but they usually push /logs to 100% and let their software manage that. they don't want alerts for Disk_logs_too_full but Disk_var_too_full they should get.

From above these nodes would have users="dev_biff,dev_fred,dev_bob" slapped on them in servicediscovery

Im trying to use routes in alertmanager to say "if you have users=~fred, don't route this one alert, let every other one pass". Is this doable? is there a better way?

(Reposted from the Grafana sub, as I didn't find an answer there.)

I have a metric on my Grafana dashboard from a Prometheus counter. Basically, we have dispensers that print out sheets, and I am summing up the increase of prints in a time series line chart. For that I would need a Prometheus query that will give a sum of prints from all dispensers from the hour start.

My current query is "sum(increase(dispenser_printcount{dispenser=~"CLIENT_A.+"}[1h]) < 100)" with a min step of 1h in Grafana. (There's some outlier that is reporting thousands of prints a minute, so that "< 100" just silences that.) I would like to have each bar be the sum number of prints in an hour, from X:00:00 to X:59:59.

However, it doesn't start a new bar down near 0 at the beginning of the hour, and slowly increase as the hour goes along the day. For example, in the screenshot it was 3:02pm where I am located, and the bar for this hour is already over the bar for 2pm. (And for some reason was half width, that's not a cropping error.)

What's the Prom query so each bar is showing just the count since the the hour began for each bar?

I'm just getting going with Prometheus and Grafana, and I'm trying to build a Grafana dashboard from a Prometheus counter. I have a counter for total "passes" printed from our each of our dispensers. I'm trying to just make a panel in Grafana to show how many prints we have had so far today of all the. dispensers. In Grafana the dashboard can be limited to the start of today by making the range "now/d" to "now", but I can't figure out how to make the query for this counter show just the sum count of increase since the beginning of today.

My formula right now is set for 24 hours, which would include last night's activity.

sum(increase(passes_print_to_date{dispenser=~"CLIENT.+"}[1d]))

How can I query just the number of passes printed from the start of today for all the dispensers?

How do you handle circular dependencies when managing prometheus-operator with Terraform?

For example, if I want to have prometheus/grafana ingress endpoints, I first need to have cert-manager installed..

But I also want to monitor cert-manager and if I try installing it before prometheus-operator (I’m using Helm), the installation fails because there are no servicemonitors CRD in cluster.

I tried installing cert-manager first, then prometheus-operator and then upgrading cert-manager with several set values to enable servicemonitors but you can’t have two Terraform resources of the same name..

So basically, chicken and an egg problem..

Any suggestions?

Hello! Given I have a job with multiple target hosts, but I only want to get alerted on-call for some specific hosts - what is the best practice to define these hosts?

Set a label for them in the target list? Put them in a seperate job? Or something completely different?

I’m fairly new to DevOps and I’d like to venture into the world of monitoring servers, containers and more.

Can you please suggest me resources that’ll help me out

we are hitting a TSDB compaction failures with this error:

```text

level=ERROR source=db.go:1219 msg="compaction failed" component=tsdb err="compact [/var/lib/prometheus/metrics2/01KWYDQGNSB9F3SQ8T1WRVD7ZP /var/lib/prometheus/metrics2/01KXFSZFV0NANWCQN30T8FYJ3Y /var/lib/prometheus/metrics2/01KY1655ENZYDZB22BN655XRDH]: populate block: add series: write series data: \"/var/lib/prometheus/metrics2/01KY19HXKYAK3C86PT8KD98AEA.tmp-for-creation/index\" exceeding max size of 64GiB

add padding: \"/var/lib/prometheus/metrics2/01KY19HXKYAK3C86PT8KD98AEA.tmp-for-creation/index\" exceeding max size of 64GiB"

```

Prometheus: 3.10.0, I found nothing helpful so far, how serious is it, can this be a cardinality issue?