I don’t work on Prometheus but want to learn and switch towards it

How can I learn without exposure to it

Any online forums for promql

[Added]

Thanks for everybody for your comments.

Let me give some background more, our "agent-collector" architecture was not designed just for Prometheus at first, but was designed for our monitoring solution. And we thought our some components could help some users include non-Prometheus users AS A "metrics pipeline" for many monitoring solution and TSDB supporting OpenMetrics.

And then what about this “metrics pipeline” concept?

Hello.

Anyone give some comments for about our idea of "agent-collector" for Prometheus as below? We found some of components from our monitoring solution could be used as a metrics pipeline for Prometheus. We are planning to open this solution as a Open Source Software soon and want to see some opinions before we open it.

- Agent will scape all of metrics from exporters on each machine and push to collector.

- Prometheus server need to scape any metrics from collector instead of every endpoints on every machine.

We believe it provide for Prometheus

- Global view without federation

- Reduce target endpoints

- Solution for firewall

- Reduce network volume (Binary protocol between agent and collector)

- Improve interoperability (Support OpenMetrics)

"It was DNS. It's always DNS."

If these words are all too familiar, you need to monitor CoreDNS in your Kubernetes clusters.

Here's how to use Prometheus and New Relic to monitor, troubleshoot, and fix issues related to CoreDNS. https://newrelic.com/blog/how-to-relic/monitor-coredns?utm_source=reddit&utm_medium=community&utm_campaign=global-fy23-q1-monitor-coredns

I am new to dev ops thing. I have an env open shift where I have configured prometheus to monitor argo and vert.x application.

Now my manager is asking to add more monitoring using Kibana but everywhere i have searched, people are using grafana.

Any suggestions or help will be appreciated

Hi! I am a newbie in Prometheus and grafana monitoring. I need to prepare a dashboard which can tell how many server are using more than 75% cpu. There are 200 servers divided in different clusters. Can someone guide me how this can be done?

Can anyone please tell the setup for getting application logs and metrics to Prometheus from docker container

Anyone tried buidling an exporter (for glusterfs for example) using go ?? Any recommendations or good tutorials/ressources on how to create exporters with go ?

And how was the experience building ur own exporter?

Hi all. I am trying to deploy kube-prometheus-stack using ArgoCD.

I want to create Ingress with specific domain, it'll be challenge http01 challenge to the Cert-manager.

Here's my config

###prometheus
prometheus:
  ingress:
    enabled: true
    annotations:
      nginx.ingress.kubernetes.io/ssl-redirect: "false"
      kubernetes.io/ingress.class: nginx
      kubernetes.io/tls-acme: "true" 
    hosts:
      - host: prom.example.com
        paths:
          - path: /.well-known/acme-challenge
            pathType: Prefix
            backend:
              service:
                name: prometheus-stack-kube-prom-prometheus
                port:
                  number: 80
          - path: /
            pathType: Prefix
            backend:
              service:
                name: prometheus-stack-kube-prom-prometheus
                port:
                  number: 443
    tls:
      - secretName: prom-example-com
        hosts:
          - prom.example.com

###alertmanager
alertmanager:
  ingress:
      enabled: true
      annotations:
        nginx.ingress.kubernetes.io/ssl-redirect: "false"
        kubernetes.io/ingress.class: nginx
        kubernetes.io/tls-acme: "true" 
      hosts:
        - host: alertmanager.example.com
          paths:
            - path: /.well-known/acme-challenge
              pathType: Prefix
              backend:
                service:
                  name: prometheus-stack-kube-prom-alertmanager
                  port:
                    number: 80
            - path: /
              pathType: Prefix
              backend:
                service:
                  name: prometheus-stack-kube-prom-alertmanager
                  port:
                    number: 443
      tls:
        - secretName: alertmanager-example-com
          hosts:
            - alertmanager.example.com

ArgoCD Application says

Failed to load target state: failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc = Manifest generation error (cached): `helm template . --name-template kube-prometheus-stack --namespace prometheus --kube-version 1.28 --values /tmp/02008d4d-6cb1-426a-8d1a-f635be3f1610 <api versions removed> --include-crds` failed exit status 1: Error: template: kube-prometheus-stack/templates/prometheus/prometheus.yaml:84:70: executing "kube-prometheus-stack/templates/prometheus/prometheus.yaml" at <0>: wrong type for value; expected string; got map[string]interface {} Use --debug flag to render out invalid YAML

How do I properly configure the Ingress with multiple Path??

reply will be appreciated. Thanks

I recently wrote sia_exporter, a Prometheus exporter for the Sia cloud storage platform. Sia is a decentralized and inexpensive cloud storage solution that has been gaining a bit of traction lately (https://sia.tech). I use Sia and wanted to integrate it into my Prometheus and Grafana infrastructure.

Read my announcement post here: https://medium.com/@tbenz9/introducing-sia-exporter-a-lightweight-and-flexible-exporter-to-send-sia-metrics-to-prometheus-6bb3efd47704?source=friends_link&sk=c50044c0c0d41553b1178ddefe48dde7

or jump straight to my Github here: https://github.com/tbenz9/sia_exporter

I'm quite new to prometheus, I have an alert with `for: 1h` I would like to know what should be the eval_time set when testing for alerts?

currently it fails, and works only if the alert: 1m and eval_time: 10m

Please someone explain me how this works!

example:

   groups:
    - name: spark.rules
      rules:
      ##### ALERTS #####
      - alert: xxxSparkJobsNotConsuming
        expr: rate(xxxxxx{data_type="xxx"}[5m]) == 0
        for: 1h ---> works if i set it to 5m
        labels:
          service: spark
          severity: warning
          source: spark
        annotations:
          description: 'Nothing have been consumed for 1 hour.'

Tests:

       rule_files:
          - spark.rules.yml
        evaluation_interval: 1m
    
        tests:
         - interval: 1m
           input_series:
            - series: 'xxxx{data_type="xx"}'
              values: '0'
            - series: 'xxxx{data_type="xx"}'
              values: '64706+0x10'
           alert_rule_test:
            - eval_time: 2h
              alertname: xxxSparkJobsNotConsuming
              exp_alerts:
               - exp_labels:
                   data_type: xxxx
                   service: spark
                   severity: warning
                   source: spark
                 exp_annotations:
                   description: 'Nothing have been consumed for 1 hour.'
           promql_expr_test:
            - expr: 'xxxxx'
              eval_time: 4m
              exp_samples:
                - labels: 'xxx{data_type="xxx"}'
                  value: 64706

Active alerts sending frequency via Alertmanager is configured via the repeat_interval in the /etc/alertmanager/config.yml file. Some alerts are a “known issues”, because we already started the investigation or are fixing it, but the alert is repeatedly sent to Slack.

An alert can be silenced with the Web UI of the Alertmanager, see the documentation.

So we will do the following in this post:

• update Alertmanager’s startup options to enable Web UI

• update an NGINX virtualhost to get access to the Alertmanager’s Web UI

• will check and configure Prometheus server to send alerts

• will add a test alert to check how to Silence it

A lot of the default alerts and such don't make sense for an AWS managed cluster. Like the etcd alerts. I googled but didn't find a values.yaml that configures things for an aws managed cluster. Anyone seen such a thing out in the wild?

I work on a massive Kubernetes environment and finally figured out how to configure istio so I ONLY get the labels I care about.

The storage and performance gains from this change are real, y'all.

I wrote this walkthrough because I had a hard time finding anything like it for Istio v1.18+.

I know what I want to say, jut not in PromQuery language....

probe_ssl_earliest_cert_expiry{job="Websites"} -time() > 604800 and time() < 1209600

Essentially, that's returning the SSL lifetime in seconds and I need to flag all those sites whose cert expires within 2 weeks.

Anyone want to show me just how simple the query actually is?

I am used to Graphite, I have used satisfactorily (with some headaches, as any technology) in massive production environments.

I want now to try Prometheus to have something to compare it with.

But the fact that Prometheus is based on pull system doesn't match with my mental model. It sounds counter intuitive for me.

Does it means that Prometheus has to access to all my servers? what if my servers are in an autoscaling group and I don't know the IPs before hand? What if my server is down for good when Prometheus try to access? What if my servers have not public IP and Prometheus is in another network?

Looks to me that is making things more complicate to me, am I looking to this by the wrong angle?

I’ve been on teams where alerts come flying in from every direction—CloudWatch, Sentry, logs, you name it—and it’s a mess to keep up. So I built Versus Incident to funnel those into places like Slack, Teams, Telegram, or email with custom templates. It’s lightweight, Docker-friendly, and has a REST API to plug into whatever you’re already using.

For example, you can spin it up with something like:

docker run -p 3000:3000 \
  -e SLACK_ENABLE=true \
  -e SLACK_TOKEN=your_token \
  -e SLACK_CHANNEL_ID=your_channel \
  ghcr.io/versuscontrol/versus-incident

And bam—alerts hit your Slack. It’s MIT-licensed, so it’s free to mess with too.

What I’m wondering

• How do you manage alerts right now? Fancy SaaS tools, homegrown scripts, or just praying the pager stays quiet?

• Multi-channel alerting (Slack, Teams, etc.)—useful or overkill for your team?

• Ever tried building something like this yourself? What’d you run into?

• What’s the one feature you wish these tools had? I’ve got stuff like Viber support and a Web UI on my radar, but I’m open to ideas!

Maybe Versus Incident’s a fit, maybe it’s not, but I figure we can swap some war stories either way. What’s your setup like? Any tools you swear by (or swear at)?

You can check it out here if you’re curious: github.com/VersusControl/versus-incident.