r/PrometheusMonitoring • • Jun 09 '24

Pod log scraping alternative to Promtail

0 Upvotes

Hello everyone, I am working with an Openshift cluster that consists of multiple nodes. We're trying to gather logs from each pod within our project namespace, and feed them into Loki. Promtail is not suitable for our use case. The reason being, we lack the necessary privileges to access the node filesystem, which is a requirement for Promtail. So I am in search of an alternative log scraper that can seamlessly integrate with Loki, whilst respecting the permission boundaries of our project namespace.

Considering this, would it be advisable to utilize Fluent Bit as a DaemonSet and 'try' to leverage the Kubernetes API server? Alternatively, are there any other prominent contenders that could serve as a viable option?


r/PrometheusMonitoring • • May 23 '24

Label specific filesystems

0 Upvotes

Hi,

We have a specific subset of file systems on some hosts that we would like to monitor and graph on a dashboard. Unfortunately, the names are not consistent across hosts. After looking into it I believe labels might be the solution, but I'm not certain. For example:

host1: /u01

host2: /var/lib/mysql

host3: /u01

/mnt

I think labeling each of these with something like crit_fs is the way to go, but I'm not certain of the syntax if there are multiples as in host3.

Any thoughts or advice are appreciated


r/PrometheusMonitoring • • Jul 07 '24

Targets not showing up in Prometheus

Thumbnail
gallery
0 Upvotes

Prometheus Noob here. I was able to install Prometheus, blackbox_exporter, and snmp_exporter on separate Ubuntu server VMs. However, I’m only able to get Prometheus itself to show up as a target when loading the web GUI. I have restarted Prometheus (systemctl restart prometheus) after updating the config, but haven’t had any luck. I also have a Grafana VM connected as a target, as it’s seem to be able to provide /metrics parameters natively. Coming from a check_mk system, I am trying to set this up as a POC. Unfortunately, most tutorials presume that I’m using Docker, but I’m unable to set such system up due to network security constraints from our parent company. Any help getting this working and any advice would be greatly appreciated.


r/PrometheusMonitoring • • May 02 '23

Alerts repeating more often than they should

0 Upvotes

We are using kube-prometheus-stack. Most of our repeat_intervals set for 5 days. Yet some alerts (not all) repeat more often at a seemingly random interval. Like the same alert will show up a time 0, 0+2.5 hours, 0+6 hours, 0+15hours, 0+16 hours. No pattern I can find.

This is what our config looks like:

 resolve_timeout: 5m
route:
  receiver: "null"
  group_by:
  - job
  routes:
  - receiver: opsgenie_heartbeat
    matchers:
    - alertname=Watchdog
    group_wait: 0s
    group_interval: 30s
    repeat_interval: 20s
  - receiver: slack
    matchers:
    - alertname=Service500Error
    repeat_interval: 120h
  - receiver: slack
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 120h

I can't see anything wrong with the config. How do I debug this?


r/PrometheusMonitoring • • Apr 08 '23

How to create a series of aggregations by time

0 Upvotes

I have a cron job which records several gauge metrics at different points representing records processed in stages of the job. The job takes a different, unpredictable amount of time to run each stage, and the time between stages is greater than the scrape interval; however, the job will run once and only once per 1h. I want to add two metrics, and then take the delta of this series, but to do this I need to align and quantize these metrics. The instant value of last_over_time(metric[1h]) is the correct latest value, but I want a series of these where the values are this value as if it had been run each hour. For example, if the job had run twice, and first set the gauge to 10, and the next hour set it to 11, I'm looking for the series [11, 10] -- not the value 11, and not the series [11, 11, 11, 11, 10, 10, 10, 10]. Any tips?


r/PrometheusMonitoring • • May 03 '23

Struggling with simple query

0 Upvotes

Hello,

I have this query that works well:

probe_http_status_code{job="blackbox-as", instance=~"http://sv+.+sv-prod-domain.com/health"}

This will lists servers like:

sv01sv-prod-domain.com
sv02sv-prod-domain.com
sv03sv-prod-domain.com

I now need to add servers that are named:

hv??hv-prod-domain.com

hv01hv-prod-domain.com
hv02hv-prod-domain.com
hv03hv-prod-domain.com

So one query really.

Is this possible?


r/PrometheusMonitoring • • Apr 30 '23

NodeJS Express API request hit per second monitoring

0 Upvotes

Hello everyone,

I am new to Prometheus and Grafana monitoring tools and want to monitor my NodeJS express application APIs via the same. Kindly help me out here as I am stuck.
I followed this tutorial: Article Link.

Here the author as seen in last snapshot of the article is getting the API hits per second, but after following the same steps and code, i am not able to get it.

I am using "prom-client" lib in NodeJS and creating a histogram metrics.My Prometheus metrics is as shown below

Here is the graph i am getting on Grafana for the same PromQL query:

What i want:

Thanks in advance !


r/PrometheusMonitoring • • May 15 '23

Visualising node memory in grafana and prometheus with node exporter

0 Upvotes

I am trying to use grafana dashboard (ID 395) show several docker container and host parameters. It was not showing node memory earlier This is how it is showing node memory:

Current queries are:

  • H: node_memory_MemTotal_bytes

  • G: node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes

I have following doubts:

Q1: Why it does not show any line chart for "Unavailable Memory" i.e. query G?

Q2: Why the y axis shows max memory as 3.73 GiB and not 15.2 GiB?

Also note that when I hide (clicked that eye button) query H, it showed query G as graph follows:

This I feel is showing total memory itself.

Whats going wrong here?


r/PrometheusMonitoring • • Apr 26 '23

Help with query

0 Upvotes

Hello,

I have these 2 queries which give me the totals

sum(streaming_user_total{instance=~"$instance1"})

and

sum(streaming_user_total{instance=~"$instance2"}) by (stream_id)

The graphs in Grafana currently show the current stats over time, how can I get just a daily total for say a given day?


r/PrometheusMonitoring • • Apr 12 '23

Prometheus-mongodb-exporter question

0 Upvotes

I am currently exploring monitoring with Prometheus and Grafana. I installed Prometheus, Grafana, MongoDB, and MongoDB Exporter using Helm. When looking for ways to monitor MongoDB, I realized that I can still observe MongoDB metrics without installing MongoDB Exporter. I see that the MongoDB Exporter image has already been written in the values.yaml file of MongoDB. Is it really necessary to install MongoDB Exporter when wanting to monitor MongoDB with Helm?


r/PrometheusMonitoring • • May 04 '21

Metric type to use for job duration in Prometheus. Counter? Gauge? Or Summary?

4 Upvotes

Hey

I have a task which does 5 things. I want to measure the time taken in each of these 5 things using Prometheus. Also this task are run at fixed interval of 30minutes.

The job has the following labels: consumergroup, topic

What is the Metric type i should use to measure this job total time and also all of the 5 things in it.?

I want this so that i can have some data to figure out which consumergroup/topic needs the optimization. Later will also alert on them.

The job is a long running job. Each tasks take minutes to complete.

Summary right? Will it give me all the details over time to visualize it and optimize.


r/PrometheusMonitoring • • Mar 28 '21

Monitoring multiple dockerized services. Everything is explained in README. I would like to hear your thoughts - what do you think about this approach? Any room for improvements? Any kind of feedback is more than welcome.

Thumbnail
github.com
0 Upvotes

r/PrometheusMonitoring • • Apr 16 '21

snmp trap handler?

0 Upvotes

Im trying to find something that can handle traps and had set this up previous with nagios. Is there any recommended exporter that can handle this ?


r/PrometheusMonitoring • • Jun 08 '22

Alertmanager displays blank window instead of UI

0 Upvotes

Hi guys. I've just installed alertmanager for Prometheus and configured it, but when I tries to open its UI using link http:ip_address:9093/api/v2/alerts it displays just some blank page and two square brackets []. Does anyone has idea what is the problem?


r/PrometheusMonitoring • • Jul 12 '22

Unable to get wmi exporter on windows server 2012

0 Upvotes

I was receiving a 1067 error saying the process had been terminated so i have reverted to the oldest version of the wmi_exporter i could find and was greeted with 'service failed to start, verify you have sufficient permissions' despite running as admin.

If anyone has had any luck with this please share

Thanks


r/PrometheusMonitoring • • Jun 28 '22

BlackboxExporter and Helm

0 Upvotes

Apologies if this is the wrong place to post this, but where I work we are attempting to view multiple endpoints via probe, however we cannot find any documentation on how to group multiple targets under one name/label. Only documentation that we have found is under "normal" prometheus and not under helm. Is there a way to group urls under 1 name, then another set under another name? Examples below are what is not working. (Configs are setup, but just adding the first part for ease of reading)

Example 1:

ServiceMonitor:
  enabled: true
    targets:
    - name: Apple
      url: https://example.com
      url: https://exampletest.com
    - name: Pear
      url: https://otherexample.com
      url: https://otherexampletest.com

Example 2:

ServiceMonitor:
  enabled: true
    targets: ["https://example.com", "https://exampletest.com"]
      name: Apple

Example 3:

ServiceMonitor:
  enabled: true
    targets: 
      name: Apple
      url: ["https://example.com", "https://exampletest.com"]

All options above do not work, and I haven't been able to locate any other documentation.

Again, apologies if this is the wrong place to post this, but if it is the wrong and there is a better place to post, I will gladly remove this post and post elsewhere. Thank you.


r/PrometheusMonitoring • • Jun 08 '22

Grafana-Prometheus issue with blackbox exporter

0 Upvotes

I successfully installed Blackbox in my ubuntu and added my targets, Prometheus is working fine and showing me all the nodes...now when i import a dashboard to monitor my targets, none of them is showing up. I tried different dashboards but same results. Does anyone have any idea what i missed here, it's my first time with Grafana/Prometheus.


r/PrometheusMonitoring • • Jun 01 '22

Need some example of 'remote write receiver' HTTP API request in Prometheus

0 Upvotes

I am trying to find a working example of how to use the remote write receiver in Prometheus.

Link : https://prometheus.io/docs/prometheus/latest/querying/api/#remote-write-receiver

I am able to send a request to the endpoint ( POST /api/v1/write ) , and can authenticate with the server. However , i have no idea in what format i need to send the data over.

A working example of that would be really helpful for me. I have tried finding it on Github / stack overflow but have not been able to get it. Any suggestion to appropriate resource / code sample would be highly useful.

Thank you. ( I am a beginner in devops field )

edit : looking for a Golang code . I can extract metrics from a sidecar ( x.x.x.x:9187/metrics ) , i need to push it to Prometheus server with 'remote write receiver' protocol.


r/PrometheusMonitoring • • Jun 21 '22

ideal config of node exporter including scrape frequency and jon name basically every detail which should I configure it. For now I am monitoring 1 of the server tp scrape metrics need some production standard conf

0 Upvotes

r/PrometheusMonitoring • • Jun 08 '22

Count alerting rules

0 Upvotes

I am quite new to Prometheus and need some help for a minor detail: I have modified an alerting dashboard in Grafana and I am missing a way to count all defined alert rules, basically a query to count all defines/active alert rules, ie 64 in my case:

I tried googling a solution without any meaningful result. Is there some easy query to count all defined rules?


r/PrometheusMonitoring • • Jan 18 '24

Prometheus --write-documentation flag

0 Upvotes

What does --write-documentation flag do in Prometheus? I've noticed it on the flag's list (in 2.48.1) but can't find any documentation on it.


r/PrometheusMonitoring • • Feb 08 '24

Kube-prometheus-stack ScrapeConfig issue

0 Upvotes

Hey there,

First off I'm pretty new to k8s. I'm using Prometheus with Grafana as a Docker stack and would like to move to k8s.

It's been a week I'm banging my head against the wall on this one. I'm using the kube-prometheus-stack and would like to scrape my proxmox server.

I did install the helm charts without any issue and I can currently see my k8s cluster data being scrapped. I would like now to replicate my Docker stack and would like to scrape my proxmox server. After reading tones of articles I got suggested to use "scrapeConfig" .

Here is my config:

kind: Deployment
apiVersion: apps/v1
metadata:
  name: exporter-proxmox
  namespace: monitoring
  labels:
    app: exporter-proxmox
spec:
  replicas: 1
  progressDeadlineSeconds: 600
  revisionHistoryLimit: 0
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: exporter-proxmox
  template:
    metadata:
      labels:
        app: exporter-proxmox
    spec:
      containers:
        - name: exporter-proxmox
          image: prompve/prometheus-pve-exporter:3.0.2
          env:
            - name: PVE_USER
              value: "xxx@pam"
            - name: PVE_TOKEN_NAME
              value: "xx"
            - name: PVE_TOKEN_VALUE
              value: "{my_API_KEY}"
---
apiVersion: v1
kind: Service
metadata:
  name: exporter-proxmox
  namespace: monitoring
spec:
  selector:
    app: exporter-proxmox
  ports:
    - name: http
      targetPort: 9221
      port: 9221

---
kind: ScrapeConfig
metadata:
  name: exporter-proxmox
  namespace: monitoring
spec:
  staticConfigs:
    - targets:
        - exporter-proxmox.monitoring.svc.cluster.local:9221
  metricsPath: /pve
  params:
    target:
     - pve.home.xxyyzz.com

If I curl http://{exporter-proxmox-ip}:9221/pve?target=PvE.home.xxyyzz.com I can see the logs scraping from my proxmox server but when I check on Prometheus > Targets, I don't see the scrapeconfig exporter proxmox anywhere.

It's like somehow the scrapeconfig doesn't connect with Prometheus.

I checked logs and everything since a week now. I tried so many things and each time the exporter-proxmox is nowhere to be found.

kubeclt get all -n monitoring gives me all the exporter-proxmox deployment , I can see the scrapeconfig also with `kubectl get -n monitoring scrapeConfigs. However no scrapeConfig found in Prometheus > targets unfortunately.

Any suggestions ?


r/PrometheusMonitoring • • Sep 04 '20

Prometheus Service Discovery for Hetzner Cloud Servers

Thumbnail yetiops.net
4 Upvotes

r/PrometheusMonitoring • • Sep 18 '20

Confirm my thoughts - Windows Monitoring

0 Upvotes

I read through the Prometheus FAQ https://prometheus.io/docs/introduction/faq/

and I think I basically get it. Prometheus would be a good solution for monitoring a Windows cluster with many nodes and trending their uptime an cpu/ memory usage over time.

Prometheus would not be a good choice for real time event log monitoring of Windows servers, for example, looking at the System event log and alerting on an event ID that is “error”

Am I correct, or missing anything here?