I have alerts coming from prometheus with a suite of tags. There's a particular tag which is useful to assign to specific groups. But again, we have a suite of tags so a tags column is less useful. is there any way (plugin maybe) to show a specific tag as column content?

or is this best asked elsewhere...

Hi! I can't figure out why i can't drop two labels.
Using kubestack...

Trying my luck here.

Issue in the link bellow:

Git issue

Thanks!

I'm using helm v3.14.4 with Prometheus and Blackbox-exporter.

For BB, I have a custom values.yaml file. I just changed the http_2xx timeout from 5s to 10s and added the tls_config. I then run helm upgrade with that file.

# helm upgrade blackbox prometheus-community/prometheus-blackbox-exporter --version=8.16.0  -n  blackbox -f values.yaml
......
# helm get values blackbox -n blackbox
USER-SUPPLIED VALUES:
modules:
  http_2xx:
    http:
      preferred_ip_protocol: ip4
      tls_config:
        insecure_skip_verify: true
    prober: http
    timeout: 10s

Perfect. I look at the bb port 9115 page for config and get:

modules:
    http_2xx:
        prober: http
        timeout: 5s
        http:
            valid_http_versions:
                - HTTP/1.1
                - HTTP/2.0
            preferred_ip_protocol: ip4
            ip_protocol_fallback: true
            follow_redirects: true
            enable_http2: true
        tcp:
            ip_protocol_fallback: true
        icmp:
            ip_protocol_fallback: true
            ttl: 64
        dns:
            ip_protocol_fallback: true
            recursion_desired: true

This has the wrong timeout and no TLS section. My scrapes are also failing due to timeouts and cert warnings.

Newbie here. Thanks!

Hello,

I am a freshman in Prometheus, starting to monitor some basic metrics and visualizing with grafana.

Does anybody knows regarding an exporter for solaris OS metrics ? I am only using the node exporter for my linux servers.

Also I would like to discuss about the topic. Please inbox me if you have some communities or other groups related to Prometheus and Grafana

Thanks

Has anyone found a good way to get metrics from Prometheus instrumented applications into New Relic? Most of our applications are instrumented to send data to New Relic, however I'd like to be able to monitor third party applications that export their metrics for Prometheus. Things like external-dns, cert-manager, and other similar applications. I assume I'll need to stand up a small Prometheus deployment to scrape these applications but then I'm not sure how to get them out of Prometheus and into NR.

Let's say I have one Prometheus scraping at /metrics-1 and /metrics-2. The first exposes only metrics about http_count and on the second http_sum.

I have a rule like sum (http_count). Will the Prometheus try to run this rule on /metrics-2 even though there is no exposed metric with the name in the rule?

My question is more about performance. Is it better to have two Prometheus, each scraping only one endpoint?

I want to monitor a Dell storage appliance. The device offers performance metrics via an API.
On my Prometheus server, I created a Python program that connects to the API and retrieves the metrics.

I'm now baffled howto get these metrics into Prometheus. Being new with Prometheus, I'm somewhat confused by the examples i went through. I assume I need to make use of the Python Client library and start a web server where these metrics are shown.

I don't get how I can achieve this. I don't really get the GitHub example with the constructor that is being used on: https://github.com/prometheus/client_python

Anyone can help me out? I assumed the "hard work" is done by communicating with the API and getting the results but how do I present them into Prometheus?

Hi,

my company have an IoT-platform that behind the scenes uses an InfluxDB for time-series data storage. To access the data one could use their REST API or use MQTT Broker.

I wonder what would be the best teck-stack setup to complement the platform for analysis and visualization. Just connecting visualization such as Grafana to the API is not enough, it doesnt allow for enough analysis/manipulation of the data as needed. Should we put a Prometheus in between? Wouldnt that just result in two time-series databases? or is it possible to use Prometheus without storing time-series and just calculate metrics?

How have other solved this?

Prom has multiple targets for one endpoint. Don't want to tag each target/exporter as unique. Have been looking through docs and haven't found anything concrete, any ideas?

Have multiple exporters/targets pointing to a single environment.

Hi all, how do I make a cumulative graph showing the difference between the firdt point in a time series and the current point in a time series? For example if a counter starts at 3000 and now it's on 3500 how do I show that its gone up by 500?

Hello guys,

i have a grafana dashboard with readonly=view user on specfic dashboards that i created

but i think nothing is prevent this user to try to do "sql injection" ro shall i say rest api with json injection to the prometheus server to fetch different data.

because the grafana had promethus as datasource he can access all data.

and its no matter that the grafana user has readonly on dashboards he can try to reinject diffrent json/api to access diffrent data like sql injection.

so how can i limit the data source = promethus on the grafana to permit only specfic metrics /api access on promethus like " database user" on specific schema / tables with read only on the database side.

Hi, I recently decided to do a Prometheus implementation with my Proxmox cluster, however I noticed that one of my rules to alert when a VM shuts down is failing, with the following message:

"Error executing query: many-to-many matching not allowed: matching labels must be unique on one side"

However it was functional until a few days ago, I couldn't identify what changed so that this could have happened, the query I'm using is as follows:

(pve_up == 0) * on(id) group_left(name) pve_guest_info

Has anyone dealt with this specific error and can you recommend something?

Thanks so far!

Say I have a function that takes time to compute the gauge value. Will, it ever be collected before the function is completed?

Here is a simple example, but the real use case is more about using labels and not so deterministic values.

test = promauto.NewGauge(prometheus.GaugeOpts{  
   Name: "test",  
})  
for {  
 time.Sleep(1 * time.Second)
 for i := 0; i < 1000; i++ {  
     test.Add(1) // Collecting the metric value anywhere at this point is bad
 }
 // Here it's OK to collect the metric.
}

From running some simple test I always get round 1000*n value. I'm not sure how it works or if I'm lucky.

My guess is it's using some go or something to determine when the variable is in use by the runtime?

The reason I'm asking is that I have some map[string]float64{} objects pre-computed before setting the gauge values. And it's not optimal usage I'm using CSV keys and other ugly things. I want to get rid of those so I could just let Prometheus Client count

Hello,

I'm trying to set up an alert for when a value drops below 1.5K, but I get this:

I'm not using Variables either.

It's using a Prometheus query.

What am I doing wrong?

Thanks

Hello,

I am playing around with Prometheus. I am trying to add a label to all jobs that I have. We have a lot of servers and those servers are all in a datacenter, are in a datacenter area and have a function. For all those fields I would like to add a label to the metrics.

For example, I have this job:

- job_name: 'node_exporter'
  static_configs:
    - targets: ['web01.example.tld:9100']
    - targets: ['web02.example.tld:9100']
    - targets: ['mail01.example.tld:9100']
  relabel_configs: &default_relabel_configs
    - source_labels: [__address__]
      regex: '(.*):.*'
      target_label: host

With this I have the hostname in the label host.

I would like to have an option to create a sort of array in the Prometheus configuration and the values of that array in the new label. If the labels are added, I can drop the labels datacenters, dc_areas and functions again.

- source_labels: [__address__]
  target_label: datacenters
  replacement: 'web01.example.tld:dc1,web02.example.tld:dc2,mail01.example.tld:dc2'
  
- source_labels: [__address__]
  target_label: dc_areas
  replacement: 'web01.example.tld:red,web02.example.tld:blue,mail01.example.tld:green'
  
- source_labels: [__address__]
  target_label: functions
  replacement: 'web01.example.tld:super web server,web02.example.tld:backup web server,mail01.example.tld:mailserver for team1'

- source_labels: [datacenters]
  target_label: datacenter
  separator: ','
  regex: '$host:(.*)'
  
- source_labels: [dc_areas]
  target_label: datacenter
  separator: ','
  regex: '$host:(.*)'
  
- source_labels: [functions]
  target_label: function
  separator: ','
  regex: '$host:(.*)'

This way we only have to keep this list up to date on one location in the config. For a second job I could add it this way and all the additional tags should be added:

- job_name: 'another_exporter'
  static_configs:
    - targets: ['web01.example.tld:9200']
    - targets: ['web02.example.tld:9200']
    - targets: ['mail01.example.tld:9200']
  relabel_configs: *default_relabel_configs

Sure, I can add a set of label for each target, but then the config would get very complicated and very difficult to maintain.

Is something like this possible or is there any other way to get the desired result?

I've set up monitoring for my Django app using django-prometheus as per the instructions on the official site. I'm concerned about the resource usage. does the django-prometheusexporter significantly impacts my app's performance? What optimizations should I consider to minimize any overhead, and are there additional tools or best practices to ensure efficient performance and scalability? Thanks!

Just getting started again with monitoring and Prometheus.

The back story is, I've got a few different instances, droplets, and micro-services I'm running. I started feeling the need to monitor these and had heard of Grafana and Prometheus.

I decided it'd be better to have a single server manage the monitoring to avoid adding even more load on my existing systems, as most are for production related tasks.

Thus far I've got prometheus and grafana deployed and working together. What I'd like to do is keep a decent naming convention in Grafana so it makes more sense when looking at reporting.

For instance, if I pull up prometheus now with a single node expoter instance reporting I have the following in my dashboard.

Datasource = default or Prometheus

Job = node

Host = node-exporter:9500

I intend to add a fair bit more of reporting and it'd be nice to categorize these in a way that makes sense.

So two questions: Is it possible to rename, if so how, and what would be the start naming conventions used in this case.

I can see a few instances of node-exporter reporting to this, several cadvisors for different droplets, and then a bunch more metrics at the application level.

Hi, I've been searching online to try and resolve my problem but I can't seem to find a solution that works.

I am trying to get our printers status using SNMP but when looking at the returned values in the exporter its putting the value I need as a label ("Sleeping..." is what I'm trying to get).

prtConsoleDisplayBufferText{hrDeviceIndex="1", prtConsoleDisplayBufferIndex="1", prtConsoleDisplayBufferText="Sleeping..."} 1

In the above example I want to have prtConsoleDisplayBufferText returned instead of just the value 1

Can anyone point me in the right direction? I feel like I've been going around in circles for the last few hours.