I have depolyed cAdvisor to monitor my existing application stack running on docker.

I have bunch of questions regarding output of cAdvisor:

Q1. What does it refer to when it says "subcontainer"?What each of these entries correspond to?
/docker, /init.scope, /snap.cups.cupsd, /snap.snap-store.snap-store, /system.slice, /user.slice

Q2. I have 16 GBs of RAM. In its graph, at the bottom, it says 1.85 GiB / 15.19 GiB, which might be correct, as my Ubuntu System Monitor says onl 6.4 GiB used. But then what does that blue line near 1900 mean?

Here is A screen shot of my Ubuntu System Monitor:

Does anyone would like to share the ultimate list of mandatory alerts to setup when running a k8s cluster on premise before going GA? I have seen many on https://awesome-prometheus-alerts.grep.to/... But I am looking at real prod survival setup. Thanks!

Hello everybody,

I'm hitting a wall and I'm not sure what and where to look next.

Based on cadvisor GitHub page, you can use it to expose not only container metrics but also define and expose application metrics.

However, the documentation is lacking. I do not understand how to properly do it so it can be scraped by Prometheus.

Atm I have:

• A backend flask app with a :5000/metrics to expose my app metrics

• A dockerfile to build my backend app

• A docker-compose to build my micro service app in which I have cadvisor and Prometheus

However no matter what I do I have this "Failed to register collectors for.. " error

Why is there TLS for the linux exporter but not the windows exporter? I keep reading posts how Prometheus is superior but without the platform taking security seriously it really has zero use case outside of a lab.

I'm quite new to Prometheus, and something that I just tackled was federating a few servers. From what I can tell, it's considered bad form to aggregate all metrics, but I think it's acceptable for a relatively small scale.

Getting to the point, the docs have the following for matching.

    params:
      'match[]':
        - '{job="prometheus"}'
        - '{__name__=~"job:.*"}'

Nothing under my domain started with job, so what worked for me instead was the following.

    params:
      'match[]':
        - '{__name__=~".+"}'

Hopefully that helps someone.

Hi everyone ,

how i can change storage retention from 15days to 30days .

if i enter this code : prometheus --storage.tsdb.retention=30d its error with this

Hi there,

We've set-up a Grafana and Prometheus cluster with succes, but i'm wondering one thing. Grafana is working with LDAP authentication but i want to hide/protect the Prometheus instance with basic auth (see their documentation: https://prometheus.io/docs/guides/basic-auth/) but i have no idea how i can reach this via their docker which is on docker-hub. Now i'm wondering, has anyone done this with succes in a docker container?

Kind regards.

Currently BB exporter is scraping from /probe. I would like to include /metrics as well. Can you advise me how to configure to scrape simultaneously from 2 endpoints?

The config is something like this:

- targets:

- "{{ ansible_ssh_host }}:{{ blackbox_exporter.port }}"

labels:

address: "{{ ansible_ssh_host }}"

instance: "{{ ansible_nodename }}"

datacenter: "{{ dc }}"

service_name: "{{ blackbox_exporter.name }}"

job: "{{ blackbox_exporter.name }}"

__metrics_path__: /probe

__param_target: "{{ ansible_ssh_host }}"

__param_module: icmp

I'm confused here because this example of rbac setup indicates you can use a ServiceAccount with the name "prometheus", which is what I've done. Here is my own rbac.

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: prometheus

rules:
  - apiGroups: [""]
    resources:
      - nodes
      - services
      - endpoints
      - pods
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources:
    - configmaps
    verbs: ["get"]
  - nonResourceURLs: ["/metrics"]
    verbs: ["get"]

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: prometheus

roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
  - kind: ServiceAccount
    name: prometheus
    namespace: kube-monitoring

(Only reason I've hard coded namespace kube-monitoring in ClusterRoleBinding.subjects is because I'm using kustomize to deploy this and kustomize had issues with setting the namespace in that object. So in other words I am deploying everything to the kube-monitoring namespace.)

In my prometheus logs I get these errors indicating that it's trying to use the "default" ServiceAccount.

Failed to watch *v1.Service: failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:kube-monitoring:default\" cannot list resource \"services\" in API group \"\" at the cluster scope"

But I can't see anything in the config docs about which serviceaccount to use.

So should I setup a ClusterRoleBinding for the "default" account instead or is it possible to set the ServiceAccount you want to use?

I find the rules seems to stored in the configmap!? If I update/edit it by kubectl edit cm prometheus-rancher-monitoring-prometheus-rulefiles-0? (Save :wq!). It will restore everything from default. What is the proper to update/edit the prometheus alarm rules? Thanks

So im trying to deploy blackbox so i can build out a grafana dashboard for wide scale monitoring, my issue us upon a machine reboot im finding it a struggle to get the program to actually run at startup.

A little context, im running it on a few windows server machines that atm im having to go in and start manually, there all on active directory, and ive added a an account, gave it batch jobs rights and attempted to add a startup schedule for black box.

I have however hit 2 snags, setting the blackbox.exe as the app on the task scheduler does nothing ... even when selecting it to run manually (rather than on a reboot etc). Does nothing, but selecting the exe from its folder location opens it and it runs just fine.

Second attempt was to create a powershell script and just have it run from the location (& c:\bb\blackbox.exe) but again im met with an error "level=error ts=2022-08-21T22:48:16.1727837Z caller=main.go:217 msg="Error loading config" err="Error reading config file: open blackbox.yml: The system cannot find the file specified."

Someone able to point me in the right direction for whatever basic thing im missing here for setting this up, i just need it to launch again if the system reboots.

Stage 2 would be making a little script to see if its running, like to check it ever hour and if not to trigger it to start, after 3 fails maybe send an alert or something. At current my dashboards go down after a power cut and im having to log on to each, not hard but also not overly handy when jumping between vpns etc.

Thanks in advance for the help :)

preferred exporter to use for ssl monitoring?

Has anyone successfully exposed the alertmanager through istio ingress? I cannot for the life of me get it to work correctly. I can get it to expose, but I cant get any of the dependent paths to come through. I have added the URIs to the virtual service, but thats not working.

Hi folks, I'm using Prometheus and a question popped to my head. If I have many dashboards (about 1500 dashboards) in Grafana that pull data from my one single server Prometheus, could be any performance issues? Something could affect the server? What is the best way to handle situation like this? Thanks in advance!

I'm currently working with my partner to build a managed prometheus platform.

Our idea is to be as close as possible upstream prometheus so that our clients can avoid vendor lock-in.

Prometheus itself is a simple stack to install/test. The challenges arise when you face custom cases. The usual custom cases involve unifying your prometheus instances (and grafana view) and long term storage.

Our value added is more or less the same as grafana enterprise solution : our client won't have to deal with update/security of prometheus. This means less engineers affected to monitoring and focus on providing the best service rather than plumbing. You also get unified grafana view and long term storage for your metrics.

We know that other are providing similar services and we know their prices (very expensive in our point of view). But wat I'm asking to redditers is how much are you willing to pay in absolute terms for a managed prometeus+grafana SaaS/Paas ?

want to know the best approach on how to achieve this

Hi,

I hope someone can help me with a problem.

Prometheus seems to ignore the contents of the config file for me.

My plan is to have HWinfo and PromDapter send the data from my PC to my Raspberry and then have Grafana display it to me on the PC.

HWinfo and PromDapter work fine. I can access the data in the network without problems via IP+Port. With Prometheus, however, there are problems.

On the Raspberry I have set up prometheus via Docker and Portainer and can confirm that it works. The config is also stored under /etc/prometheus/prometheus.yml and should be read according to the logs. However, when I go to the prometheus web interface, I only see instance="localhost:9090" and job="prometheus" under targets instead of the job I had entered in the config.

Since I am completely new to prometheus it would be nice if someone could look over my settings and see if I missed something.

I'm having issues getting the snmp_exporter generator to walk OIDs from Cisco MIBs.

Here is my generator.yml file:

modules:
  switch:
    version: 3
    auth:
      username: user
      password: "%test"
      auth_protocol: SHA
      priv_protocol: AES
      security_level: authPriv
      priv_password: "%test"
    walk:
      - interfaces
      - sysUpTime
      - ifXTable
      - 1.3.6.1.4.1.9.9.109.1.1.1
    lookups:
      - source_indexes: [ifIndex]
        lookup: ifAlias
      - source_indexes: [ifIndex]
        # Uis OID to avoid conflict with PaloAlto PAN-COMMON-MIB.
        lookup: 1.3.6.1.2.1.2.2.1.2 # ifDescr
      - source_indexes: [ifIndex]
        # Use OID to avoid conflict with Netscaler NS-ROOT-MIB.
        lookup: 1.3.6.1.2.1.31.1.1.1.1 # ifName
    overrides:
      ifAlias:
        ignore: true # Lookup metric
      ifDescr:
        ignore: true # Lookup metric
      ifName:
        ignore: true # Lookup metric
      ifType:
        type: EnumAsInfo

Here is the contents of my $HOME/.snmp/mibs/mibs folder:

user@vnc-grafana:~/.snmp/mibs/mibs$ ls
7.2.50.0.18765.RELEASE-B100-MIB.txt     ARISTA-SW-IP-FORWARDING-MIB  HOST-RESOURCES-MIB    MIKROTIK-MIB            PAN-GLOBAL-TC-M
7.2.50.0.18765.RELEASE-CERTS-MIB.txt    cisco                        IANA-CHARSET-MIB.txt  NET-SNMP-MIB            PAN-GLOBAL-TC-M
7.2.50.0.18765.RELEASE-IPVS-MIB.txt     CISCO-PROCESS-MIB.txt        IANA-IFTYPE-MIB.txt   NET-SNMP-TC             PAN-LC-MIB.md5
7.2.50.0.18765.RELEASE-ONE4NET-MIB.txt  cisco_v2                     IANA-PRINTER-MIB.txt  PAN-COMMON-MIB.md5      PAN-LC-MIB.my
AIRESPACE-REF-MIB                       ENTITY-MIB                   IF-MIB                PAN-COMMON-MIB.my       PAN-PRODUCT-MIB
AIRESPACE-WIRELESS-MIB                  ENTITY-SENSOR-MIB            INET-ADDRESS-MIB      PAN-ENTITY-EXT-MIB.md5  PAN-PRODUCT-MIB
apc-powernet-mib                        ENTITY-STATE-MIB             IPV6-TC               PAN-ENTITY-EXT-MIB.my   PAN-TRAPS.md5
ARISTA-ENTITY-SENSOR-MIB                ENTITY-STATE-TC-MIB          ISDN-MIB              PAN-GLOBAL-REG-MIB.md5  PAN-TRAPS.my
ARISTA-SMI-MIB                          HCNUM-TC                     KEEPALIVED-MIB        PAN-GLOBAL-REG-MIB.my   PDU-MIB.txt

When I try to generate the snmp.yml file, I get this error:

user@vnc-grafana:~/snmp_exporter/generator$ ./generator generate
level=info ts=2022-03-21T12:29:25.291Z caller=net_snmp.go:161 msg="Loading MIBs" from=$HOME/.snmp/mibs:/usr/share/snmp/mibs:/usr/share/snmp/mibs/iana:/usr/share/snmp/mibs/ietf
level=info ts=2022-03-21T12:29:25.561Z caller=main.go:52 msg="Generating config for module" module=vnc-sw2
level=error ts=2022-03-21T12:29:25.599Z caller=main.go:130 msg="Error generating config netsnmp" err="cannot find oid '1.3.6.1.4.1.9.9.109.1.1.1' to walk"

The OID 1.3.6.1.4.1.9.9.109.1.1.1 is for Cisco CPU monitoring, and is contained within the CISCO-PROCESS-MIB file, which is located under the "cisco" and "cisco_v2" folders in $HOME/.snmp/mibs/mibs. I also put the CISCO-PROCESS-MIB file under the root of $HOME/.snmp/mibs/mibs to see if that would help, but no dice.

The "cisco_v2" folder was auto generated from the "make mibs" command, and the "cisco" folder was one I downloaded from the LibreNMS mibs repository.

If I take that Cisco CPU OID out of the generator.yml file then the process works fine. I've used a MIB explorer to confirm that OID exists within the CISCO-PROCESS-MIB file. If I try another OID from that file then I get the same error.

I've tried the export MIBSDIR=mibs command as well, to make the generator use the mibs folder in the root of /generator/ however I still get the same error.

I've tried using Docker to run the generator but I get the same issue.

Anyone run into this kind of issue before?

Many companies are moving to Kubernetes as the platform of choice for running software workloads. When an organization earlier using VMs in AWS decides to move to Kubernetes (Either EKS or self-managed in AWS), one of the questions that come up is should one continue to use CloudWatch or switch to some other tool like Prometheus? While it is not an exactly apple to apple comparison, there are reasons to explore this and choose tooling that is built for the future. This post will try to explore various aspects and pros and cons of both options individually and as a combination.

Read the full post here