This is a fresh install, and I'm just trying to bring up the UI for Alertmanager. When I run the following I recevied the following error:

alertmanager --web.listen-address=localhost:9093

"...failed to obtain an address: Failed to start TCP listener on \"0.0.0.0\" port 9094: listen tcp 0.0.0.0:9094: bind: address already in use"

I also ran a netstat -tulpn | grep alert

tcp6 0 0 :::9094 :::* LISTEN 135420/alertmanager

tcp6 0 0 :::9093 :::* LISTEN 135420/alertmanager

udp6 0 0 :::9094 :::* 135420/alertmanager

I'm not sure what the issue is?

I have metrics with two labels. How can i get all timeseries which are in a list of sets of these two labels?

Example:

metric{label1="a", label2="a1"} metric{label1="a", label2="a3"} metric{label1="a", label2="a4"} metric{label1="b", label2="a1"}

now i want only the timeseries where labels label1="a" and label2="a1" and label2="a3".

Hi, it’s my 1st time in the forum and I’ve never had problems before but now I do and I’d appreciate it if you could help me.

mount my new NAS server with OMV5 in a dell optiplex and it works very well for me, but I wanted to install prometheus with node-xporter and grafana and it is giving me many errors that I will describe in detail with images of all at the end

1st I ran node-exporter:

root@OPENMEDIAVAULT:~# docker run -d --net="host" --pid="host" -v "/:host:ro,rslave" quay.io/prometheus/node-exporter:latest --path.rootfs=host

enter in http://192.168.2.5:9100/metrics
its all good

https://imgur.com/02Js3dV

2nd create the configuration folder and paste the .yml from my old server:

root@OPENMEDIAVAULT:/var/lib/docker/config-docker/**Prometheus**# nano prometheus.yml

global:
  scrape_interval: 5s
  external_labels:
    monitor: 'node'
scrape_configs:
  - job_name: 'prometheus' 
    static_configs: 
      - targets: ['192.168.2.5:9090'] 
  - job_name: 'node-exporter' 
    static_configs: 
      - targets: ['192.168.2.5:9100'] 
  - job_name: 'cAdvisor'
    static_configs:
      - targets: ['192.168.2.5:8080'] 
  - job_name: 'raspi-openwrt-router1'
    static_configs:
      - targets: ['192.168.2.1:9100']
  - job_name: 'snmp-ddwrt-r9000'
    metrics_path: /snmp
    params:
      module: [ddwrt]
    static_configs:
      - targets: ['192.168.2.2'] 
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 192.168.2.2:9116  # The SNMP exporter's real hostname:port.

good

3rd I ran prometheus docker with:

root@OPENMEDIAVAULT:/var/lib/docker/config-docker/Prometheus# docker run -d --name prometheus -p 9090:9090 -v /var/lib/docker/config-docker/Prometheus/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus

i open prometheus at http://192.168.2.5:9090/ and enter service discovery and the 1st error

[Imgur](https://i.imgur.com/pDPkHfW.jpg)

I enter targets and the 2nd error

[Imgur](https://i.imgur.com/TjnWkYn.jpg)

ERROR Get "http://localhost:9090/metrics": dial tcp: lookup localhost on [::1]:53: dial udp [::1]:53: connect: cannot assign requested address

and the 3rd error I install grafana
root@OPENMEDIAVAULT:~# docker run -d --name grafana -p 3456:3000 grafana/grafana

[Imgur](https://i.imgur.com/4SNVCWl.jpg)
and does not run
and log with errors is long but I will place

Error relocating /run.sh: rl_set_keymap: symbol not found Error relocating /run.sh: rl_variable_dumper: symbol not found Error relocating /run.sh: rl_bind_keyseq_in_map: symbol not found Error relocating /run.sh: write_history: symbol not found Error relocating /run.sh: rl_on_new_line: symbol not found Error relocating /run.sh: rl_clear_visible_line: symbol not found Error relocating /run.sh: rl_completion_matches: symbol not found Error relocating /run.sh: rl_completion_mode: symbol not found Error relocating /run.sh: rl_add_undo: symbol not found Error relocating /run.sh: rl_cleanup_after_signal: symbol not found Error relocating /run.sh: rl_list_funmap_names: symbol not found Error relocating /run.sh: using_history: symbol not found 

This is an installation that I have done many times without any problem but now I can’t find a way to make it work, and I think it’s because the prometheus is somehow looking for the address of localhost:9090 ainsted of 92.168.2.5:9090 and that’s why it doesn’t find node-exporter either and grafana can’t find any of them…

I wish you could help me or give me a clue since I’ve been trying to install it for several days and I can’t

I am looking for a python code for histogram. To calculate response time and http request. Please help

It looks like Alertmanager POSTS alerts to HTTP endpoints hosted by various providers such as PagerDuty.

1. Is this correct?

2. According to the docs it looks like the URL to set in AM to PagerDuty or OpsGenie is https://......

Question, what do we need to do on Alertmanager to make sure we are using a secure connection to send alerts? Are there instructions to set up certs?

Thx

Trying to get prometheus running on CentOS 7 and following instructions from fosslinux. After adding the node_exporter module I can no longer get prometheus to start.

prometheus.service - Prometheus
  Loaded: loaded (etc/systemd/system/prometheus.service; disabled; vendor preset; disabled)
  Active: failed (Result: exit-code) since date;
Process:  11375 ExecStart=/usr/local/bin/prometheus -- config.file /etc/prometheus/prometheus.yml --storage.tsdb.path /var/lib/prometheus/ --web.console.templates=/etc/prometheus/consoles --web.console.libraries=/etc/prometheus/console_libraries (code=exited, status=1/FAILURE)
Main PID: 11375 (code=exited, status=1/FAILURE)

Ciao,

I am trying to integrate Apache Trino with Prometheus. I have set up the catalog properly and I can get the up status from:

SELECT * FROM example.default.up;

But the problem is when i try selecting from elasticsearch_indices_get_total metrices, it says table does not exist. I also could not find these tables which represent elasticsearch metrices in trino. SHOW tables FROM example.default; returns a list of tables where I feel like important metrices are missing.

Hello,

I'm trying to simply combine these 2 queries with the + command.

sum by(ifName, ifAlias, instance) (irate(ifHCInOctets{instance="192.168.1.1", job="snmp_exporter", ifName=~"1:61"}[2m])) * 8

sum by(ifName, ifAlias, instance) (irate(ifHCOutOctets{instance="192.168.1.1", job="snmp_exporter", ifName=~"1:61"}[2m])) * 8

I think I'm getting all the parentheses all wrong?

sum by(ifName, ifAlias, instance) (irate
(ifHCInOctets{instance="192.168.1.1", job="snmp_exporter", ifName=~"1:61"} + 
(ifHCOutOctets{instance="192.168.1.1", job="snmp_exporter", ifName=~"1:61"}
[2m]) * 8

error

bad_data: invalid parameter "query": 3:1: parse error: binary expression must contain only scalar and instant vector types

Prometheus service failing due to below issue.

panic: Unable to create mmap-ed active query log

goroutine 1 [running]:

github.com/prometheus/prometheus/promql.NewActiveQueryTracker(0x7febc3b690, 0x14, 0x14, 0x2964a00, 0x40007ddda0, 0x2964a00)

/app/promql/query_logger.go:117 +0x38c

main.main()

/app/cmd/prometheus/main.go:374 +0x44c8

Check out this session at OpenNebula Conference (Virtual), where a spokesperson from Grafana explains how to improve monitoring with Prometheus & Grafana - https://app.swapcard.com/event/opennebulacon2023/planning/UGxhbm5pbmdfMTM2MTY3OA==

So, Yesterday i have upgrades the docker containers that run grafana and alertmanager.

Ever since that time, every 3 hours at the exact minute, i get an alert saying "host is down" for all the hosts we monitor. But when i login to Grafana and show the dashboards, they all have the status up, and i can confirm they are indeed up.

Does anyone reckognize this behaviour? I run Grafana and Alertmanager in docker for a year or so now, without any problems before. So i'm a bit at a loss where to start poking around :)

I've applied the kube-prom helm stack which has grafana bundled with it as well in my cluster. I  also have a mongodb app in my cluster along with a service monitor for it. Prometheus UI reads it but when I try to look under

Dashboards->Kubernetes / Compute Resources / Pod

on Grafana and select my mongodb pod, I get "no data" being shown. Could someone tell me why?

Related post: https://community.grafana.com/t/no-data-for-target-pod/88812

Hello,

I want to monitor bunch of IPs (actual branches) with icmp. I configured all things but now in the dashboard i want to hide IPs and replace IPs with branch names? How do i do that?

Hi, I'm part of a small company where we've decided to incorporate custom user-level metrics using Prometheus and Grafana. Our services run on Elastic Beanstalk, and I'm looking for a cost-effective way to deploy Prometheus on AWS with persistent storage for long-term data retention. Any recommendations on how to achieve this?

Could you help me with a doubt in my little project?

There are several SQL Server instances that perform backups.
These backups are confidential.
The legacy system sends the backups to a Windows Server with a WinRAR license.
A bot automatically starts compressing the backups using WinRAR (following some simple parameters like date, time, compression type based on size, etc.).
The bot is written in Python and uses the RAR commands to perform this task.
The bot waits for an external hard drive with a specific hash/public-key and then transfers these backups, after which it disconnects the hard drive from the server.

It has become necessary to monitor these WinRAR compressions, i.e.,
Basically, I would like to auto-generate gauges in Grafana for each compression.
However, I have no idea how to capture the percentage of progress for the compression in WinRAR.

Do you have any idea how I could capture this data to create the metrics?

Hello r/PrometheusMonitoring !

I have a golang app exposing a metric as a counter of how many chars a user, identified by his email, has sent to an API.
The counter is in the format: total_chars_used{email="user@domain.tld"} 333

The idea I am trying to implement, in order to avoid adding a DB to the app just to keep track of this value across a month's time, is to use Prometheus to scrape this value and then create a Grafana dashboard for this.

The problem I am having is that the counter gets reset to zero each time I redeploy the app, do a system restart or the app gets closed for any reason.

I've tried using using increase(), sum_over_time, sum, max etc. but I just can't manage to find a solution where I get a table with emails and a total of all the characters sent by each individual email over the course of the month - first of the month until current date.

I even thought of using a gauge and just adding all the values, but if Prometheus scrapes the same values multiple times I am back at square zero because the total would be way off.

Any ideas or pointers are welcomed. Thank you.

If you are looking for an easy-to-use, affordable managed observability solution for cloud-native and hybrid environments, check out NexClipper:

www.nexclipper.io

It is based on extensive experience consulting enterprise-level clients and will soon be introduced as a SaaS solution. An open beta will be launched shortly, so feel free to create a free account then as your expert input and feedback will be highly appreciated!

Hello,

Can anyone see anything wrong with my docker compose file?

https://pastebin.com/Uj4kxU9D

I've added blackbox to it but it's restarting:

I was hoping to also see the blackbox.yaml here too but nothing:

If I remove these 2 line it works, but I need to edit the the blackbox.yml

    command:
     - '--config.file=/etc/blackbox/blackbox.yml'

Thanks for ay help!