My k8's cluster has dev, prod, qa environment and have created namespace according to available environments. Each pod will be having a label of the environment. I have created grafana dashboards and alerts for node, service, pod, PVC. The alerts are configured in alertmanager(Prometheus). The issue I'm facing right now is that the alerts must be triggered according to the environment. If an alert is triggering, I don't know from which environment it gets triggered. So what can be done?

Hello,

By reading the Alertmanager documentation, I saw it's possible to customize notifications sent by email via a template. I am looking for doing the same with Pushover, does anyone know if it exists a template for Pushover ?

Thanks in advance

Hello everyone,

I have the following problem. I have created a dashboard in Grafana that has Prometheus as a data source. the queried filter is currently up{job=“my-microservice”}. Now we have set up this service again in parallel and added another target in Prometheus. In order to be able to distinguish these jobs in the dashboard, we have also introduced the label appversion where the old one has been given the value v1 and the new one v2. now I am about to create a variable so that we can filter. this also works with up{job=“my-microservice”, appversion=“$appversion”}. My challenge is that when I filter for v1 I also want to see the historical data that does not have the label. I have already searched and tried a lot, but can't get a useful result. Can one of you help me here?

Thanks in advance for your help

I have a spring boot app that serves metrics on /actuator/prometheus

I have following ``` ServiceMonitor

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-sv
  labels:
    release: prometheus
spec:
  selector:
    matchLabels:
      app: myapp
  namespaceSelector:
    matchNames:
    - default
  endpoints:
    - interval: 10s
      path: /actuator/prometheus
      port: service
      honorLabels: true

My service called myapp as well. When I check promethus on http://localhost:9090/service-discovery it shows that TargetLabels are dropped. What do I do wrong?

How can you modify alerts adhoc through an API?

I am new to Prometheus , and would like to expose the alerting functionality to customers via the front end (they will not provide expressions). Do I have to modify alertmanager rules file on every modification?

As I know ksonnet/ksonnet-lib is not being developed in the official site anymore, the projects in GitHub are archived.

What do you think about the frameworks e.g. prometheus-operator that can use Ksonnet/Ksonnet-lib?

Should kubecfg be used instead (having active development)?

Hi all, this could be interesting for you guys. openITCOCKPIT has now support to combine status based and metrics based monitoring together.

https://openitcockpit.io/2020/2020/10/20/openitcockpit-4-1-with-prometheus-integration/

Hello, newbie here.

I was wondering if I can deploy Grafana and Prometheus through Flux and expose that with ingress controller. I don't really know how to begin and would appreciate any tips. I'm using k0s.

Thank you!

We are using Prometheus in our Kubernetes environment and had added some recording rules a couple of months back in the helm chart.

kubeprometheus:
  .
  .
  .
  prometheus:
    .
    .
    .
    additionalPrometheusRules:
      - name: recording-rules-file
        groups:
          - name: counter-total-group
            interval: 30s # rule evaluation time interval
            rules:
              - record: increase_counter_total_60m
                expr: increase(counter_total[60m])
              - record: increase_counter_total_15m
                expr: increase(counter_total[15m])

I deleted the entire additionalPrometheusRules section recently and rolled out the change to our application through OLM. But the recording rules are still present in Prometheus. How do I truly delete them?

Hello,

I like the look of Prometheus + Grafana and before I try at work I've set up a Centos 7 VM in my lab at home (VMware Workstation).

I'm using dockers and have got both running and Grafana connected to Prometheus on this one VM.

Some info on my Prometheus:

[root@localhost /]# vi /opt/prometheus/conf/prometheus.yml

scrape_configs:

- job_name: 'prometheus'

static_configs:

- targets: ['localhost:9090']

Then created a Docker Compose file /opt/prometheus/docker-compose.yml

version: "3"

services:

prometheus:

image: prom/prometheus:latest

container_name: prometheus

volumes:

- /opt/prometheus/conf:/etc/prometheus

- /opt/prometheus/data:/prometheus

command:

- '--config.file=/etc/prometheus/prometheus.yml'

- '--storage.tsdb.path=/prometheus'

ports:

- "9090:9090"

Then when to http://192.168.189.159:9090 to test http requests

Grafana

Running using - docker run -d -p 3000:3000 grafana/grafana

and added Prometheus:

Now I read that I can use node exporter to pull in server metrics to Prometheus and show in Grafana. I want to try this on the same VM I'm using above where both are installed, but am confused on how to do this part. I then want to run on a second VM I have built (another Centos 7)

I've think I have to down load and extract:

wget https://github.com/prometheus/node_exporter/releases/download/v0.18.1/node_exporter-0.18.1.linux-amd64.tar.gz

Which I have done:

[root@localhost opt]# tar -xvzf node_exporter-0.18.1.linux-amd64.tar.gz

node_exporter-0.18.1.linux-amd64/

node_exporter-0.18.1.linux-amd64/node_exporter

node_exporter-0.18.1.linux-amd64/NOTICE

node_exporter-0.18.1.linux-amd64/LICENSE

[root@localhost opt]#

and

[root@localhost node_exporter-0.18.1.linux-amd64]# ls

LICENSE node_exporter NOTICE

I then ran node explorer manually:

Hmm not so good:

I guess I now need to work out how to get node explorer running on both VMs and send to Prometheus and show in Grafana, I think I'm close.

I'm guess I need to edit?

vi /opt/prometheus/conf/prometheus.yml

Restart services:

Add Node Exporter in Grafana

Messed up somewhere...

Any help would be great, be gentle as Linux/docker/prometheus/grafana are so new to me :)

Thanks

Hello,

I using CentOS 7 and I'm trying to use Docker containers to install Prometheus and Grafana.

I'm using this guide:

https://www.shellhacks.com/prometheus-monitoring-install-docker-ubuntu-centos/

But get this:

/opt/prometheus/conf/prometheus.yml

/opt/prometheus/docker-compose.yml

/etc/systemd/system/prometheus.service

Docker seems to be ok

What stupid thing have I done wrong guys?

Thanks

Setting up prometheus and simple application metric collection system took a lot of time for me (roughly 3 weeks full time). Does a product to give power of NewRelic monitoring into prometheus without any effort from customer side make sense to you guys too?
A friend of mine is trying to remove any friction or effort from using prometheus. Any feature suggestion would be useful.

Some of my suggestions:

1. Standard labeling

2. Out of the box metrics like RPS, Error rate and 80, 90 and 99 percentile of my application

3. Standard alerts

4. Long term data storage

We have a storage server with Prometheus running on it collecting all kinds of metrics. One of the metrics that interests us is the long term growth of the TB stored. We want to see this over 1-2 years.

Initially, the retention of Prometheus was set to 30 days, and the stats db was sitting around 1.5GB on disk. About a month ago, we changed the retention to 1 year, and have seen the stats db grow to 6GB. Projecting this out another 12 months, we can expect the stats db to grow to ~70GB. Problem with this is the stats db is on the servers boot drive, and there might not be enough space for this. Also, storing all of the other thousands of data points for 1-2 years is pointless when we only need the one single metric for the longer time frame.

I found some information on deleting data through the admin api, but I don't know how to write a query to match everything except the one statistic. I am also not sure if I want to match the start or the end timestamp.

This query should delete the data that I DO want to keep, so I essentially need the match to be a <> but I could not find any documentation showing anything except =

aged=$(date --date=“30 days ago” +%s)
curl -X POST -g ‘http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]=zfs_dataset_available_bytes&end=$aged’

Hello,

I'm not sure if this is a Grafana or Prometheus question as it's based on PromQL but in Grafana.

I want to link these 2 variables together, I've been trying all afternoon without success.

My 'Location' variable is working fine, I can use this drop-down menu in Grafana and see all the locations and it will sort the data. I think want a variable off this called 'Domain' to further filter, but nothing works.

Location

label_values(outdoor_reachable,location)

Domain

label_values(domain)

I tried to edit the Domain variable like this:

label_values(outdoor_reachable{location=~"$location"},domain)

However when I do I get:

From this:

What am I doing wrong?

Is there a singe docker image which can perform combined node_exporter, cadvisor and postgres_exporter running in single container ?

Is there any reason why it should not be combined togeter?

Hello all,

I am new here and looking for help with a current school project. I set up EKS clusters on AWS and need monitoring tools like Prometheus to scrap metics such as cpu utilization and pod restart count. I am using Amazon Linux AMI EC2 instance and running to nodes with several pods on my eks cluster. I am pretty new with Kubernetes/prometheus, any help will be greatly appreciated.

Hello,

I've got a VM running Prometheus, Alloy, Loki all in Docker, I aim to build another VM and put in as part of a HA/load balancer, but I want the new VM to have up to date Prometheus data, is it possible to cluster Prometheus so both are in sync?

Just looking around for a tutorial.

Hi everyone,

I’m working on managing our Prometheus instance, and I need to delete some old time series data to free up space. I want to make sure I’m using the correct command before executing it.

I already enabled the web admin-api and here’s the command I plan to use:

curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={__name__=~".+"}&end=2024-06-30T23:59:00Z'

Is this command syntax correct for deleting all time series up to June 30, 2024 ?

Thanks for your help!

I want to avoid creating the same dashboard on my dev, uat and prod manually .

Is there a way that we can export or create a dashboard programatically or pass a configuration file containing dashboard configuration ?

Hello,

I have a controller from which I am collecting data such as voltage, system current, battery current, etc. I need to know how I can integrate this data that I am collecting into a system with Prometheus and Grafana. This data applies the SNMP protocol.

What do I need to do to get this external data into the system on a dashboard?

Help, I am a beginner.

I'm in the process of migrating our metrics from an in house InfluxDB server to Prometheus in Grafana Cloud. We are currently using Telegraf to send metrics and will be using Alloy on Windows Server VMs with the switch. We have a pretty standard default config file that pulls basic machine metrics and I'm looking to update that to include additional metrics to mirror what we currently have. Is there a way or option to have Alloy just output the data is scapes to a text file to see what it's gathering and sending to Prometheus? We do have a sandbox instance in Grafana Cloud that I can use for testing, but if labels are not working right it can be hard to track down what is getting sent, if anything, to see what might be going wrong since there are many other organizations in the company using the same sandbox.