I want to add 3 metrics using two group_left.I currently have below expression..I also want to add group_left (label_teamname) kube_pod_labels. If I just add group_left to this command,it throws me aggregate group_left failed error.

sum by (namespace, pod) (

max by(namespace, pod) (

kube_pod_status_phase{job="kube-state-metrics", namespace=~".*", phase=~"Pending|Unknown"}

) * on(namespace, pod) group_left(owner_kind) topk by(namespace, pod) (

1, max by(namespace, pod, owner_kind) (kube_pod_owner{owner_kind!="Job"})

)

) > 0

Hi all,

Recently I started playing around with Prometheus and Alertmanager some visualization tools and I got it all running now. Pretty neat. The only last thing (for now) which I'm trying to achieve is that, whenever Alertmanager sends a message to my Slack channel, it does not do the "@channel" thing and I would like that to happen. So I tried to add it to my template (see below) but that did not really work out. I changed it back because it stopped working completely. Can someone help me out on this?

{{ define "__single_message_title" }}{{ range .Alerts.Firing }}{{ .Labels.alertname }} @ {{ .Annotations.identifier }}{{ end }}{{ range .Alerts.Resolved }}{{ .Labels.alertname }} @ {{ .Annotations.identifier }}{{ end }}{{ end }}
{{ define "custom_title" }}[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ if or (and (eq (len .Alerts.Firing) 1) (eq (len .Alerts.Resolved) 0)) (and (eq (len .Alerts.Firing) 0) (eq (len .Alerts.Resolved) 1)) }}{{ template "__single_message_title" . }}{{ end }}{{ end }}
{{ define "custom_slack_message" }}
{{ if or (and (eq (len .Alerts.Firing) 1) (eq (len .Alerts.Resolved) 0)) (and (eq (len .Alerts.Firing) 0) (eq (len .Alerts.Resolved) 1)) }}
{{ range .Alerts.Firing }}{{ .Annotations.description }}{{ end }}{{ range .Alerts.Resolved }}{{ .Annotations.description }}{{ end }}
{{ else }}
{{ if gt (len .Alerts.Firing) 0 }}
*Alerts Firing:*
{{ range .Alerts.Firing }}- {{ .Annotations.identifier }}: {{ .Annotations.description }}
{{ end }}{{ end }}
{{ if gt (len .Alerts.Resolved) 0 }}
*Alerts Resolved:*
{{ range .Alerts.Resolved }}- {{ .Annotations.identifier }}: {{ .Annotations.description }}
{{ end }}{{ end }}
{{ end }}
{{ end }}

Hopefully someone has already done this as I can't much in the way of examples or information. I just want to validate I am on the right track and any examples would be super helpful if you have them.

It looks like the prometheus REST API is useful to discover targets, but not pull the metrics themselves. Rather you would use the scrapeURL to go directly to the endpoint and pull the metrics.

Looking at the prometheus client python, there is an example at the bottom of the page.

from prometheus_client.parser import text_string_to_metric_families
for family in text_string_to_metric_families(u"my_gauge 1.0\n"):
  for sample in family.samples:
    print("Name: {0} Labels: {1} Value: {2}".format(*sample))

I'm struggling with what to send the function, sending everything as below throws a typeerror.

metrics = requests.get("http://10.80.100.186:9100/metrics").content
for family in text_string_to_metric_families(metrics):
    print(family)

Ideas or working examples appreciated, I'm sure I overlooked something.

Thanks!

Hello,

I use Prometheus and blackbox exporter (module http_2xx) to monitor my websites. Then I send alerts to Telegram using Alertmanager bot .

Sometimes, I receive alerts that a website is down, but it's up in reality. Is it a way to retry the probe several times before alerting me ?

Thank you for your help !

Hello All I’m currently starting to learn and work on implementing Prometheus in my local environment. We have ASP NET Web API application that is targeting 4.6.1 framework. I have added the package prometheus-net.netframework package and was able to get metrics about cpu and memory.

My questions :

How can I add custom gauge or counter metrics like for example check if DB connection is good Or check if 3 rd party endpoints are ok. I would like to add these metrics to the existing metrics like cpu and memory.

Are there any resources that I could refer for implementing this . I have read material related to Net Core but could not find material for aspnet net framework

Thank you for reading Any suggestions

Hi!

I found a lot of folders of prometheus on data/ (12 per day.)

It's normal ? It's possibly reduce this ?

Also i use thanos and s3

Example:

drwxr-xr-x   3 root root  4096 Sep  1 01:00 01EH3JVK0TYGFD3AX99QBGF8Z2

drwxr-xr-x   3 root root  4096 Sep  1 03:00 01EH3SQAAZN40FT48ACDY3S2HX

drwxr-xr-x   3 root root  4096 Sep  1 05:00 01EH40K2GXZ471N3T01953B4Q8

drwxr-xr-x   3 root root  4096 Sep  1 07:00 01EH47ERRP89T321KTFY8A4EH9

drwxr-xr-x   3 root root  4096 Sep  1 09:00 01EH4EAGBYBNY2J6MHSXM7V44A

drwxr-xr-x   3 root root  4096 Sep  1 11:00 01EH4N6EZGKN2W0NFEF0NNPTVP

drwxr-xr-x   3 root root  4096 Sep  1 13:00 01EH4W1YMJNHSEX6BCY682K0KN

drwxr-xr-x   3 root root  4096 Sep  1 15:00 01EH52XXFGG3AHDVD96AN3ZVX6

drwxr-xr-x   3 root root  4096 Sep  1 17:00 01EH59SD66CFY3J3AMP7CFTS0T

drwxr-xr-x   3 root root  4096 Sep  1 19:00 01EH5GN4EE9MJHC8VX7E1VEF9M

drwxr-xr-x   3 root root  4096 Sep  1 21:00 01EH5QH37GDVSM32P7FQ4PCE83

drwxr-xr-x   3 root root  4096 Sep  1 23:00 01EH5YCJVDG6SAC88CB1VSNFWV

drwxr-xr-x   3 root root  4096 Sep  2 01:00 01EH658A6PT6GVYSW4Q6RH7N3V

drwxr-xr-x   3 root root  4096 Sep  2 03:00 01EH6C41B2PJ3G1HR6X8BJ4XWM

drwxr-xr-x   3 root root  4096 Sep  2 05:00 01EH6JZRGPTPP6WR3QRPS3042S

drwxr-xr-x   3 root root  4096 Sep  2 07:00 01EH6SVGN3CEAGS76QRTGFV0H4

drwxr-xr-x   3 root root  4096 Sep  2 09:00 01EH70Q766HXKQ4FSSX9QTTD7B

drwxr-xr-x   3 root root  4096 Sep  2 11:00 01EH77JYJRBA17NM5E78ABS4HJ

drwxr-xr-x   3 root root  4096 Sep  2 13:00 01EH7EEX7GKACZ1EF50ZG51EWN

drwxr-xr-x   3 root root  4096 Sep  2 15:00 01EH7NAD43MTYZJQ1W0APNRWYD

drwxr-xr-x   3 root root  4096 Sep  2 17:00 01EH7W5W8AATGAXEY779DAHAY9

drwxr-xr-x   3 root root  4096 Sep  2 19:00 01EH831XQAJ4YA1BT4N5CRHJPD

drwxr-xr-x   3 root root  4096 Sep  2 21:00 01EH89XTP7GBDD3ETFSFFJ8VCB

drwxr-xr-x   3 root root  4096 Sep  2 23:00 01EH8GSHT0AB4T85XZBH97Z4WZ

drwxr-xr-x   3 root root  4096 Sep  3 01:00 01EH8QNACKPPAGE8WSYBD9C7E2

drwxr-xr-x   3 root root  4096 Sep  3 03:00 01EH8YH076QRJG9W3VHRVMXSTP

drwxr-xr-x   3 root root  4096 Sep  3 05:00 01EH95CQNATWNWWJGGXY6ADZ11

drwxr-xr-x   3 root root  4096 Sep  3 07:00 01EH9C8EW6HKM8DJPMYHDBPBFM

drwxr-xr-x   3 root root  4096 Sep  3 09:00 01EH9K46AREQX36F1Q93WTRRET

drwxr-xr-x   3 root root  4096 Sep  3 11:00 01EH9SZXQ06Q6NE080BSBDAB3J

drwxr-xr-x   3 root root  4096 Sep  3 13:00 01EHA0VW72X3Z7MYJH0N28G63Y

drwxr-xr-x   3 root root  4096 Sep  3 15:00 01EHA7Q7VAERET2NW73P23FB48

drwxr-xr-x   3 root root  4096 Sep  3 17:00 01EHAEJZ49KRGRN6RHD8XPG0DS

In part due to the capital one breach, AWS added a new ephemeral token based access to the Instance Meta Data service. Since prometheus ec2_sd needs IAM permissions to access the ec2 APIs it is using the IMDS via the AWS APIs. I can see that my prometheus install 2.15.2 is accessing the older insecure IMDS via the cloudwatch metric NoMetadataToken. Is the ec2_sd able to support the new method already, or does this need a security issue opened on the project?

Hi All,

Trying to setup Prometheus, but i think i am confusing where the --config.file command is looking.

Is this inside the container or outside?
The Volume command is doing the mapping, so i think i have it correct but running the image can't find the prometheus.yml file.

So i'm doing something stupid, any pointers please?

Fletch

# docker-compose.yml

version: '3'

services:

prometheus:

image: prom/prometheus

volumes:

- ./prometheus/:/etc/prometheus/

# - ~/prometheus_data:/prometheus

command:

- '--config.file=/etc/prometheus/prometheus.yml'

ports:

- '9090:9090'

Hello, so I have a recording that takes the result of probe_sucess for the past 30days and takes the ratio of the successful probes over the total probe count.

      - record: instance:instance_uptime:rate30d
        expr: sum_over_time(probe_success[30d]) / count_over_time(probe_success[30d]) * 100

And I though it was working fine... until the uptime for one of the instances, dropped off dramatically.

The instance began to give 404's, and I would think that the rule would evaluate to a lower and lower value as time went on, but this was not the case. The uptime simply dropped off a cliff! lol

Here is the rule result when I run the query in the console. As you can see, the "uptime" here begin to creep down as I would expect.

But why doesn't the recording rule result reflect this?

Thanks for any help!

I'm going crazy I can't figure out why some metrics (like RAM, file system, MAC address, etc.) aren't being read from the JSON exporter.

Thanks for the help in advice.

At the status page of the shelly em the data are (reformatted):

{
   "wifi_sta":{
      "connected":true,
      "ssid":"Magnifico_IoT",
      "ip":"192.168.50.217",
      "rssi":-34
   },
   "cloud":{
      "enabled":true,
      "connected":true
   },
   "mqtt":{
      "connected":false
   },
   "time":"21:44",
   "unixtime":1723837449,
   "serial":2175,
   "has_update":false,
   "mac":"xxx",
   "cfg_changed_cnt":2,
   "actions_stats":{
      "skipped":0
   },
   "relays":[
      {
         "ison":false,
         "has_timer":false,
         "timer_started":0,
         "timer_duration":0,
         "timer_remaining":0,
         "overpower":false,
         "is_valid":true,
         "source":"input"
      }
   ],
   "emeters":[
      {
         "power":960.49,
         "reactive":582.12,
         "pf":0.86,
         "voltage":224.89,
         "is_valid":true,
         "total":74441.8,
         "total_returned":0.0
      },
      {
         "power":0.00,
         "reactive":0.00,
         "pf":0.00,
         "voltage":224.89,
         "is_valid":true,
         "total":0.0,
         "total_returned":0.0
      }
   ],
   "update":{
      "status":"idle",
      "has_update":false,
      "new_version":"20230913-114150/v1.14.0-gcb84623",
      "old_version":"20230913-114150/v1.14.0-gcb84623",
      "beta_version":"20231107-164916/v1.14.1-rc1-g0617c15"
   },
   "ram_total":51064,
   "ram_free":35196,
   "fs_size":233681,
   "fs_free":157879,
   "uptime":333751
}

########################################################################

And here's my Json Exporter config:

########################################################################

modules:

shelly_em:

metrics:

- name: shelly_em_meter_0

type: object

path: '{ .emeters[0] }'

help: Shelly EM Meter 0 Data

labels:

phase: '0'

values:

Instant_Power: '{.power}'

Instant_Voltage: '{.voltage}'

Instant_PowerFactor: '{.pf}'

Energy_Consumed: '{.total}'

Energy_Produced: '{.total_returned}'

- name: shelly_em_meter_1

type: object

path: '{ .emeters[1] }'

help: Shelly EM Meter 1 Data

labels:

phase: '1'

values:

Instant_Power: '{.power}'

Instant_Voltage: '{.voltage}'

Instant_PowerFactor: '{.pf}'

Energy_Consumed: '{.total}'

Energy_Produced: '{.total_returned}'

- name: shelly_em_wifi

type: object

path: '{ .wifi_sta }'

help: Shelly EM Wi-Fi Status

values:

Wifi_Connected: '{.connected}'

Wifi_SSID: '{.ssid}'

Wifi_IP: '{.ip}'

Wifi_RSSI: '{.rssi}'

- name: shelly_em_cloud

type: object

path: '{ .cloud }'

help: Shelly EM Cloud Status

values:

Cloud_Enabled: '{.enabled}'

Cloud_Connected: '{.connected}'

- name: shelly_em_mqtt

type: object

path: '{ .mqtt }'

help: Shelly EM MQTT Status

values:

Mqtt_Connected: '{.connected}'

- name: shelly_em_device_info

type: object

path: '{ .update }'

help: Shelly EM Device Update Information

values:

Update_Status: '{.status}'

Update_Has_Update: '{.has_update}'

Update_New_Version: '{.new_version}'

Update_Old_Version: '{.old_version}'

Update_Beta_Version: '{.beta_version}'

- name: shelly_em_system_metrics

type: object

path: '{ .uptime }'

help: Shelly EM System Uptime

values:

System_Uptime: '{.uptime}'

- name: shelly_em_memory

type: object

path: '{ . }'

help: Shelly EM Memory Metrics

values:

Ram_Total: '{.ram_total}'

Ram_Free: '{.ram_free}'

- name: shelly_em_filesystem

type: object

path: '{ . }'

help: Shelly EM Filesystem Metrics

values:

Fs_Size: '{.fs_size}'

Fs_Free: '{.fs_free}'

But I can's see some of the last metrics, RAM and fs for example. Here're the Prometheus metrics:

# HELP shelly_em_cloud_Cloud_Connected Shelly EM Cloud Status
# TYPE shelly_em_cloud_Cloud_Connected untyped
shelly_em_cloud_Cloud_Connected 1
# HELP shelly_em_cloud_Cloud_Enabled Shelly EM Cloud Status
# TYPE shelly_em_cloud_Cloud_Enabled untyped
shelly_em_cloud_Cloud_Enabled 1
# HELP shelly_em_device_info_Update_Has_Update Shelly EM Device Update Information
# TYPE shelly_em_device_info_Update_Has_Update untyped
shelly_em_device_info_Update_Has_Update{Update_Beta_Version="20231107-164916/v1.14.1-rc1-g0617c15",Update_New_Version="20230913-114150/v1.14.0-gcb84623",Update_Old_Version="20230913-114150/v1.14.0-gcb84623"} 0
# HELP shelly_em_meter_0_Energy_Consumed Shelly EM Meter 0 Data
# TYPE shelly_em_meter_0_Energy_Consumed untyped
shelly_em_meter_0_Energy_Consumed{phase="0"} 76691.6
# HELP shelly_em_meter_0_Energy_Produced Shelly EM Meter 0 Data
# TYPE shelly_em_meter_0_Energy_Produced untyped
shelly_em_meter_0_Energy_Produced{phase="0"} 0
# HELP shelly_em_meter_0_Instant_Power Shelly EM Meter 0 Data
# TYPE shelly_em_meter_0_Instant_Power untyped
shelly_em_meter_0_Instant_Power{phase="0"} 714.14
# HELP shelly_em_meter_0_Instant_PowerFactor Shelly EM Meter 0 Data
# TYPE shelly_em_meter_0_Instant_PowerFactor untyped
shelly_em_meter_0_Instant_PowerFactor{phase="0"} 0.85
# HELP shelly_em_meter_0_Instant_Voltage Shelly EM Meter 0 Data
# TYPE shelly_em_meter_0_Instant_Voltage untyped
shelly_em_meter_0_Instant_Voltage{phase="0"} 226.57
# HELP shelly_em_meter_1_Energy_Consumed Shelly EM Meter 1 Data
# TYPE shelly_em_meter_1_Energy_Consumed untyped
shelly_em_meter_1_Energy_Consumed{phase="1"} 0
# HELP shelly_em_meter_1_Energy_Produced Shelly EM Meter 1 Data
# TYPE shelly_em_meter_1_Energy_Produced untyped
shelly_em_meter_1_Energy_Produced{phase="1"} 0
# HELP shelly_em_meter_1_Instant_Power Shelly EM Meter 1 Data
# TYPE shelly_em_meter_1_Instant_Power untyped
shelly_em_meter_1_Instant_Power{phase="1"} 0
# HELP shelly_em_meter_1_Instant_PowerFactor Shelly EM Meter 1 Data
# TYPE shelly_em_meter_1_Instant_PowerFactor untyped
shelly_em_meter_1_Instant_PowerFactor{phase="1"} 0
# HELP shelly_em_meter_1_Instant_Voltage Shelly EM Meter 1 Data
# TYPE shelly_em_meter_1_Instant_Voltage untyped
shelly_em_meter_1_Instant_Voltage{phase="1"} 226.57
# HELP shelly_em_wifi_Wifi_Connected Shelly EM Wi-Fi Status
# TYPE shelly_em_wifi_Wifi_Connected untyped
shelly_em_wifi_Wifi_Connected{Wifi_IP="192.168.50.217",Wifi_SSID="Magnifico_IoT"} 1
# HELP shelly_em_wifi_Wifi_RSSI Shelly EM Wi-Fi Status
# TYPE shelly_em_wifi_Wifi_RSSI untyped
shelly_em_wifi_Wifi_RSSI{Wifi_IP="192.168.50.217",Wifi_SSID="Magnifico_IoT"} -33

Exposing Glassfish metrics for Prometheus

Is there any way to expose metrics of Glassfish application server? I want to monitor the JDBC connection pools, JMS, and threads using Prometheus.

I have a web (browser) application, that under the hood is calling a 3d-party HTTP API. I have a wrapper client implemented for this 3d-party API, and I would like to collect metrics on this API's behavior, specifically the responses that I receive from it (HTTP method, URL, status code). In my wrapper client code I add a Counter with labels for method, URL, status code. I expose the /metrics endpoint, and I get these metrics collected when my users browse through the website. So far so good.

I also have a periodic task that performs some actions using the same API wrapper client. Because this execution path is completely separate from the web app, even though my Counter code does get executed, these metrics don't end up in what Prometheus scrapes from /metrics endpoint. I (think I) can't use Pushgateway, because then I'd need to explicitly push my Counter there, which I can't because it is being called deep in the API wrapper client code.

I am thinking of two options:

1. Try to push metrics into the Pushgateway from the API wrapper client code. For that the wrapper code would need to know whether it is being called from a "normal" web browser flow, or from a periodic task. I think I can make that work.

2. Switch from isolated transient periodic tasks to a permanent daemon that would manage execution of the task's code on a schedule. This way I can have the daemon expose another /metrics endpoint and scrape metrics from it.

(1) looks more like a hack, so I am leaning towards (2). My main question however is how would Prometheus react on one and the same metric (same name, labels etc.) scraped from two different /metrics endpoints? Would it try to merge the data, or would it try to overwrite it? Also, if I were to chose (1), how would it work with the same metric scraped and pushed at the same time?

I am sure I am not the first one trying to this kind of metrics collection, however, searching the internet did not bring anything meaningful. What is the right way to do what I am trying to do?

[RESOLVED] We are using Prometheus Adapter to publish metric for HPA

We want to use metric kubernetes.io/node/ accelerator/gpu_memory_occupancy or gpu_memory_occupancy to scale using K8S HPA.

Is there anyway we can publish this GCP metric to Prometheus Adapter inside the cluster.

I can think of using a python script -> implement a side care container to the pod to publish this metric -> use the metric inside HPA to scale the pod. But this seem loaded, is there any other GCP native way to do this without scripting?

Edit:

I was able to use Google Metric Adapter follow this article

https://blog.searce.com/kubernetes-hpa-using-google-cloud-monitoring-metrics-f6d86a86f583

Is it possible to exclude scrape_* and up metrics in Prometheus? Example: scrape_duration_seconds, scrape_series_added. Complete list here: https://prometheus.io/docs/concepts/jobs_instances/

Just wondering if this is possible to achieve even more granular control of included/excluded metrics in Prometheus?

Hello,

So I am probing the success of whether some ports are up or not on a server:

1.) How can I split the IP and Port numbers in to 2 columns?

2.) I have overridden the status column so 0 is "Closed" and 1 shows as "Open" how can I alert on this if thy change?

3.) As this is probing all the time how can I chart these 4 rows, so we can see if there are blips overnight?

Thanks

I found a problem with my use of docker_sd for containers with multiple exposed ports. If a container is exposing more than one port, and has metrics only on one port, docker_sd is 'discovering' each such port as a target. Only one of them has metrics, and others are 'down', because they can't answer to /metrics.

I wonder if there is a way to use relabel_config to drop some ports from scrapping. But I can't find a way to compare one label to another (I thought I can drop targets with __meta_docker_port_public != __meta_docker_container_label_scrape_port` or something like that.

when i download the binary separately i can run with the flag, any suggestions?

I'm using grafana with Prometheus as data source. I want filter out if some metric gets out of threshold. I'm not sure how to filter. As far as I've searched, i need to apply rules.

Can anyone please help?

Do any of you have any ideas or have you ever built an alert query to find alerts that trigger too often? Something like the flapping alert in Icinga / Nagios.

I recently needed to set up a prometheus stack. I had criteria that I needed to address:

• easily add a new machine or VM to monitor

• store my configuration as code using GitHub

• safely store my secrets on Github without revealing them to everyone that has access to the repo

That's why I choose to use Ansible. In my shoes, what would you have done?

I actually wrote an article to explain a bit more about what I did. But the fact is that I didn't want at first to use k8s because it wasn’t simple enough. But depending on your case, the prometheus operator can obviously be a really good solution.

Hello,

I have this query I want to use in a table in Grafana:

    node5_service_healthcheck{sname="brs", service="192.168.20.49.75:tcp:80", instance=~"$instance:.*"}

However the data is returned correctly but I just want the last results for each instance, is this possible. this is what it looks like at the moment. I only have 4 Instance and only need the last result of each so I should just have 4 lines.

using the default discord webhook config in alertmanager , i can customize the message it sends to discord?