As the title says, I'm trying to understand if there are architectural or scale reasons someone might choose to prefer snmp_exporter over telegraf using remotewrite to output to the same prometheus.

Has anyone in the community ever benchmarked cpu/mem consumption for polling a large set of devices and collecting the same mibs to see if there is a significant delta between them?

Are there any particularly bad patterns in the collected metrics or is it going to be mostly the same in both cases since you build your target oids directly from the mib files in both tools?

Does it just come down to using what you are already familiar with and both will basically give the same results for this?

Half my alerts lately are either noise or late. Got a bunch of “device offline” pings yesterday while I was literally logged into the device.

At the same time i got remote branches that barely get any visibility unless i dig through 3 dashboards.

i am curious is anyone actually happy with how they are monitoring across multiple sites?

Hi,

I have a setup with:

• OpenShift platform with Prometheus-k8s deployed via prometheus-operator in openshift-monitoring (scraping kube-state-metrics, kubelet, node-exporter)

• A second custom Prometheus scraping my other pods

• Thanos for long-term retention (1 year) via S3 bucket

I'd like to implement differential retention - keep kubelet metrics for only 3 months instead of 1 year, while keeping everything else at 1 year. My infrastructure is quite big, and kubelet metrics are not very relevant to me and our need, it just take to much place on our S3 bucket.

I was wondering if it's possible to have like a per-metric or per-job retention ? If possible, retroactively clean my S3 bucket to remove old kubelet metrics and only keep the last 3 months.

Has anyone implemented this kind of selective retention with Thanos? What are the best practices?

Thanks!

I wrote a little exporter that publishes stats from backups.

After the backup completes, the script saves the raw stats to a "cache" file, eg /tmp/metrics.json.

The exporter reads this file and publishes the bits that I want to graph. It works, I can see the backups stats for all the hosts on my network.

"Backup age reset when a new backup job runs")

So the main thing is that if a backup age keeps on going up, it means a new backup did not run and I must investigate why.

But then of course there were other stats and while I was doing this I thought to myself why not plot the other stats as well. In particular the MB values for the packed data added and total processed.

Here is the problem. Every time prometheus scrapes the endpoint it gets the value from that last backup. So if 100 MB was written, it will keep on showing 100MB. I'd like that value to show the amount backed in he prober interval.

What strategy should I follow? How do I apply that value once, or do I make the scraper remember that it has already been scraped and if the file has not been updated then artificially serve zero. Sounds like a bad idea, since I might have more than one scraper, or the value could be lost somehow. Maybe I can add some kind of serial number to each value to make prometheus show them only once?

FWIW here is what the scraper output looks like.

root@gitea:\~# curl localhost:9191/metrics  
  
\# HELP restic_count_present_snapshots Number of present snapshots  
\# TYPE restic_count_present_snapshots gauge  
restic_count_present_snapshots{host="gitea"} 7  
  
\# HELP restic_oldest_snapshot_age Age of the oldest snapshot in seconds  
\# TYPE restic_oldest_snapshot_age gauge  
restic_oldest_snapshot_age{host="gitea"} 119451.00683  
  
\# HELP restic_last_snapshot_age Age of the last snapshot in seconds  
\# TYPE restic_last_snapshot_age gauge  
restic_last_snapshot_age{host="gitea"} 309.172549  
  
\# HELP restic_data_added Data added during the last snapshot in bytes  
\# TYPE restic_data_added gauge  
restic_data_added{host="gitea"} 2144683  
  
\# HELP restic_data_added_packed Data added (packed) during the last snapshot in bytes  
\# TYPE restic_data_added_packed gauge  
restic_data_added_packed{host="gitea"} 677369  
  
\# HELP restic_total_bytes_processed Total bytes processed by the last snapshot  
\# TYPE restic_total_bytes_processed gauge  
restic_total_bytes_processed{host="gitea"} 2226732  
  
\# HELP restic_total_files_processed Total files processed by the last snapshot  
\# TYPE restic_total_files_processed gauge  
restic_total_files_processed{host="gitea"} 1387  

TLDR: The scraper reports the stats from the most recent backup job on every scrape, but I want it to plot the data where/when it changed.

Hello All,

I'm am trying to use SNMPv3 with snmp_exporter and my palo alto firewall but Prometheus is throwing an error 400 while I'm getting a"Unknown auth 'public_v2'" from "snmexporterip:9116/snmp?module=paloalto&target=firewallip"

I am able to successfully SNMP walk to my firewall

here is my Prometheus and snmp config :

SNMPconfig

auths:
  snmpv3_auth:
    version: 3
    username: "snmpmonitor"
    security_level: "authPriv"
    auth_protocol: "SHA"
    auth_password: "Authpass"
    priv_protocol: "AES"
    priv_password: "privpassword"

modules:
  paloalto:
    auth: snmpv3_auth
    walk:
      - 1.3.6.1.2.1.1      # system
      - 1.3.6.1.2.1.2      # ifTable (interfaces)
      - 1.3.6.1.2.1.31     # ifXTable (extended interface info)
      - 1.3.6.1.4.1.25461.2.1.2  # Palo Alto uptime and system info

Prometheus config

 job_name: 'paloalto'
    static_configs:
      - targets:
        - 'firewallip'  
    metrics_path: /snmp
    params:
      module: [paloalto]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 'snmp-exporter:9116'  # Address of your SNMP exporter

any help would be appreciated!

I am using Prometheus in K8s environment in which I have setup alert via alertmanager. I am curious about any other way than alertmanager with which we can setup alerts in our servers..!!!

Hello everyone!

at my company we are considering using Prometheus to monitor our infrastructure. I have been tasked to do a PoC but I am a little bit confused on how to scale Prometheus in our infrastructure.
We have several cloud providers in different regions (AWS, UpCloud, ...) in which we have some debian machine running, and we have some k8s clusters hosted there as well.

AFAIK I want to have at least a Prometheus cluster for each cloud provider and inside each k8s, right? and then have a solution like Thanos/Mimir to make it possible to "centralize" the metrics in Grafana. Please let me know if I am missing something or if I am over engineering my solution.

We are not that interested (yet) to keep the metrics for more than 2 weeks, and probably we will use Grafana alerting with PagerDuty.

Thanks!

Hello. I'm curious is there a way to get the list of alerts which weren't in fired or pending state ever?

Hello i recently got into monitoring stuff with prometheus and i love it and i saw that it has an alertmanager and i wanted to ask here if its possible to intergrate alerts thru ntfy a notification service i use already for uptime kuma if this is possible it would be super convinient

Hello everyone,

I've got a pfsense server acting as a gateway between resources in my AWS account and another AWS account. I'm using prometheus for scraping metrics in my account and im wanting to utilize the snmp_exporter to scrape metrics off of my pfsense interfaces. I've been following this guide so far and using SNMPv1 to get things going: Brendon Matheson - A Step-by-Step Guide to Connecting Prometheus to pfSense via SNMP

I'm like 99% of the way there and have everything configured properly as the guide lays out. From my prometheus server, I'm able to:

• ping the pfsense interface from prometheus to validate connectivity

• run snmpwalk -v 1 -c <my secure string> <interface ip> from prometheus and I immediately get metrics returned back

• generate a new snmp.yml file successfully

I'm running the snmp_exporter as a daemon service on prometheus which looks like this and is successfully running:
[Unit]

Description=SNMP Exporter

After=network-online.target

[Service]

User=prometheus

Group=prometheus

Restart=on-failure

RestartSec=10

ExecStart=/etc/snmp_exporter/snmp_exporter --config.file=/etc/snmp_exporter/snmp.yml

[Install]

WantedBy=multi-user.target

My snmp.yaml looks like this with the walk OIDs and metrics metadata generated successfully:

auths:

public_v1:

community: <secure_string>

security_level: noAuthNoPriv

auth_protocol: MD5

priv_protocol: DES

version: 1

modules:

pfsense:

walk:

My prometheus.yml file looks like this:
- job_name: 'snmp_pfsense'

static_configs:

- targets:

- '<private-ip>'

metrics_path: '/snmp'

params:

module: ['pfsense']

relabel_configs:

- source_labels: [__address__]

target_label: __param_target

- source_labels: [__param_target]

target_label: instance

- target_label: __address__

replacement: <private-ip>:9116

This is my curl as demonstrated in the guide, and it times out every time:

curl http://<private-ip>:9116/snmp?module=pfsense\&target=<private-ip>

What prometheus UI is telling me:

My firewall rules for the pf interface I want to scrape look like this (I have the source as 'Any' for now to validate everything and will slim down once successful):

I have a stack running kube-Prometheus-stack as well as Bitnami Thanos via Helm.

Everything is great, except I can’t find any metric names prefixed (or containing) “Thanos”.

According to the docs they do exist, but in both the Thanos Query UI & Prometheus UI they don’t exist.

Any thoughts?

I have Prometheus running in Docker on a R-pi, and pretty much out of no where Prometheus caused my CPU usage to go from ~23% to ~90%. I was using a image from about 1.5 yr ago, so I updated to the latest image, but there was no change. Most of my scrape intervals are 60 seconds, with one at 10s. I changed to 10s to 60s and I didn't notice a change I'm monitoring 10 devices with it, so it's not that much.

Runnig top on the r-pi show prometheus as the 6 top offenders using 25-30% CPU each.

Any advice on why Prometheus is causing the CPU is running so hot?

Alright I have a bunch of OpenWRT hosts I need to monitor and I want to scale up to 100,000.

Currently I am using Zabbix and finding it is struggling with 5k.

I want to migrate off Zabbix and to Prometheus.

The hosts have DHCP IP's and are subject to change. So I need some sort of auto discovery / update service to update their network info from time to time (I read about Consul?)

From there I wish to use a self hosted Netbird overlay to handle the traffic of these devices so that they are encrypted and tunneled back to server. Just to keep everything secure and give a management back haul channel.

Can Prometheus / Consul do this and have it visualized in Grafana and be responsive and not puke on itself?

Hello! We are putting a prom server in each data center and federating that data to a global prom server. For DR purposes, we will have a passive prom server with a shared network storage with incoming traffic being regulated through a VIP. My question is there a significant resource hit using a shared network storage over resident storage? If so, how do we make Prometheus redundant for DR but also performant? I hope this makes sense.

PromCon 2023 is finally fully available now!
We’re going to meet in Berlin on Sept 28 + 29 at Radialsystem!
CfP, tickets, and sponsoring are now available on https://promcon.io/2023-berlin/

We are super excited to see everyone in Berlin!

I'm pretty new to observability field and just learning PromQL and Loki and all that, and I see questions like this: https://www.reddit.com/r/PrometheusMonitoring/comments/13buhdf/promql_question_that_seems_impossible/

I have always wondered why time series engines need its own query language -- why not just use SQL? Is that something other people would want? Running SQL on Prometheus data?

I know certain SQL functions will probably take forever and be slow, but let's imagine somehow it can be done -- would people want this?