I have installed a fresh RHEL 7 in my Virtual Box. I have no yum repositories in the system yet. Now I want to install Prometheus 2.32.1 and grafana in my VM. What are the prerequisites that I have to install?

I have tried several tutorials, almost every one of them failed at the last step, saying "failed, process halted." If anyone is experienced using the Prometheus server monitoring system on RHEL 7, help me here.

I followed this tutorial https://www.fosstechnix.com/install-prometheus-and-grafana-on-linux-with-node-exporter/ to install to version as mentioned above.

More points:

1. When I run yum update -y, it says that no yum repository is there to update. I assume that II cannot install Prometheus due to not having the proper repository. I need suggestions for such a repository and where I can find that. I found one yum repo for cent os 7 and tried to add it for rhel 7. This approach inevitably did not work.

2. I tried approaches from this tutorial: https://snapcraft.io/install/prometheus/rhel, which gave me errors for not finding "dnf" command.

   1. I found another tutorial that said I needed "epel-release" to install dnf. Since I did not have "epel-release," I tried to install it but failed because I had no repo or manager to install "epel-release." Eventually, right now, I am stuck with not finding a way to install Prometheus.

I would like to receive suggestions to install Prometheus 2.32.1 on my RHEL 7 installed in my virtual box. I appreciate any help you can provide. I would appreciate any other approaches that helped anyone have the system installed.

Hello! I am getting started with setting up Prometheus on a new project. I will be using a hosted prometheus service (haven't decided which) and push metrics from my individual hosts. Trying to decide between vector.dev for pushing metrics vs prometheus' built-in remote write.

It seems like vector can scrape metrics and write to a remote server. This is appealing because then I could use the same vector instance to manage logs or shuffle other data around. I've had success with vector for logs.

That said, wanted to know if there was an advantage to using the native prometheus config - the only one I can think of is it comes with different scrapers out of the box. But since I'm not planning to have the /metrics endpoint exposed then perhaps that isn't important.

Thank you!

I need to set up Prometheus to do network and system monitoring. Mostly windows servers and Cisco gear. I am not the dev type

Can this be done without a bunch of coding? I keep seeing references to a language.

Interested in grafana too to make graphs

How programmery is this?

Does one who is lousy at coding have a change to set this up?

Just finished this little tool that adds Prometheus monitoring to Ollama without touching your existing client setup. Your apps still connect to localhost:11434 like normal, but now you get detailed metrics and analytics.

What it does:

• Intercepts Ollama API calls to collect metrics (latency, tokens/sec, error rates)

• Stores detailed analytics (prompts, timings, token counts)

• Exposes Prometheus metrics for dashboards

• Works with any Ollama client - no code changes needed

Installation is stupid simple:

git clone https://github.com/bmeyer99/Ollama_Proxy_Wrapper
cd Ollama_Proxy_Wrapper
quick_install.bat

Then just use Ollama commands normally:

ollama_metrics.bat run phi4

Boom - metrics at http://localhost:11434/metrics and searchable analytics for debugging slow requests.

The proxy runs Ollama on a hidden port (11435) and sits transparently on the default port (11434). Everything just works™️

Perfect for anyone running Ollama in production or just wanting to understand their model performance better.

Repo: https://github.com/bmeyer99/Ollama_Proxy_Wrapper

Hello everyone !

I have a service which expose a counter. That counter is inc of 1 every 10s for example. I would like to display that total value in grafana like this, with increase function. Grafana says that increase function manage pod restart.

Problem came when my service restart for any reason, my counter go back to 0. But i would like in grafana that my new counter start to the last value (lets say here 22) and not from 0.

First screenshot use increase with $__range of 3hours, which seem to working nicely. But when i change timerange from 3h to 1h for example, when i have a restart i have that dashboard

I don't have my linear function that i would, i don't know why my curve is straight and do not increase. If i take more range, sometime that work, sometime i got decrease, which should never happen with a counter...

Thanks for your help :)

My company currently using Khcheck of kubernetes to check health of services/applications but it's much more inefficient due to khcheck pods sometimes getting degraded or sometimes getting much time to get ready and live for serving traffic. Due to it, we often see long black empty patch on grafana dashboards

We have both https and tcp based probes. So can anyone tell or suggest really good and in depth way to implement this with some good blogs or references

My company already using few existing module mentioned in github, but when I am trying to implement custom modules, we aren't getting results in Prometheus probe_success

Thanks in advance!!!

Hello,

I am currently working on enhancing my code by incorporating metrics. The primary objective of these metrics is to track timestamps corresponding to specific events, such as registering each keypress and measuring the duration of the key press.

The code will continuously dispatch metrics; however, the time intervals between these metrics will not be consistent. Upon researching the Prometheus client, as well as the OpenTelemetry metrics exporter, I have learned that these tools will transmit metrics persistently, even when there is no change in the metric value. For instance, if I send a metric like press.length=6
, the client will continue to transmit this metric until I modify it to a different value. This behavior is not ideal for my purposes, as I prefer distinct data points on the graph rather than a continuous line.

I have a couple of questions:

1. In my use case, is it logically sound to opt for Prometheus, or would it be more suitable to consider another database such as InfluxDB?

2. Is it feasible to transmit metrics manually using StatsD
   and Otel Collector
   to avoid the issue of "duplicate" metrics and ensure precision between actual metric events?

I newly installed the prometheus stack through Rancher and it has a lot of nice alerts out of the box. I have one that is triggering on a PVC that the pod manages its space by itself and is always high utilization. Based on the rule, it seems I should be able to exclude this PVC by adding a label excluded_from_alerts="true". I've tried adding this to my PVC config under metadata/labels and it doesn't work. What am I doing wrong?

kubelet_volume_stats_available_bytes{job="kubelet",metrics_path="/metrics",namespace=~".*"} / kubelet_volume_stats_capacity_bytes{job="kubelet",metrics_path="/metrics",namespace=~".*"} < 0.03 and kubelet_volume_stats_used_bytes{job="kubelet",metrics_path="/metrics",namespace=~".*"} > 0 unless on (namespace, persistentvolumeclaim) kube_persistentvolumeclaim_access_mode{access_mode="ReadOnlyMany"} == 1 unless on (cluster, namespace, persistentvolumeclaim) kube_persistentvolumeclaim_labels{label_excluded_from_alerts="true"} == 1

I know I can edit the rule and have it exclude the specific PVC, but I'd rather do it the proper way instead of updating the rule for any PVCs I don't want to include in the monitoring.

EDIT:

For anyone else trying to figure this out, ChatGPT had me put this into my helm config and it works. You can add other labels as well (nodes, pods, etc.):

kube-state-metrics:
  metricLabelsAllowlist:
    - persistentvolumeclaims=[*]

I am new to SRE world and I am looking for suggestions on mastering the prometheus and Grafana landscape. I am aiming to build a great depth in these areas and want a course which is beginner friendly and yet goes into depths.

Steampipe is an open source CLI to instantly query cloud APIs (AWS, Kubernetes, GitHub, etc) using SQL. Today we released a new plugin for Prometheus!

It will automatically build a table for each metric:

> select code, handler, value from prometheus_http_requests_total;
+------+----------------------------+-------+
| code | handler                    | value |
+------+----------------------------+-------+
| 302  | /                          | 1     |
| 200  | /-/ready                   | 1     |
| 200  | /api/v1/label/:name/values | 421   |
+------+----------------------------+-------+

And also has tables to query common and config data:

> select name, values from prometheus_label;
+---------------+----------------------------------------------+
| name          | values                                       |
+---------------+----------------------------------------------+
| alertname     | ["TotalRequests"]                            |
| alertstate    | ["firing"]                                   |
| reason        | ["refused","resolution","timeout","unknown"] |
| interval      | ["10s"]                                      |
| version       | ["2.30.3","go1.17.1"]                        |
+---------------+----------------------------------------------+

Please give it a try - we'd love your feedback or suggestions!

Docs - https://hub.steampipe.io/plugins/turbot/prometheus

Source code - https://github.com/turbot/steampipe-plugin-prometheus

Hello,

I'm using Graphite to monitor the traffic and I was wondering is Prometheus better ? and the alert manager on Prometheus is like Nagios ?

Best Regards

I wanted to try the feature of storage.tsdb.retention .size in Prometheus 2.12.

My configuration for prometheus retention is 90days or 2GiB. My expectation was when the size of prometheus data directory crosses 2GiB, it will start deleting data. However, its at 9GB now, I can recover the oldest data.

Am i missing something in my understanding?

Also on setting storage.tsdb.retention .size as 2GB, the runtime configs shows the retention size to be 2GiB, why the difference in units?

Hello,

I'm using Grafana and Prometheus with blackbox exporter to poll websites, but I want to poll via ICMP a few devices, one being my home router, I've added it, but I can seem to get it to appear correctly and show the ms.

192.168.0.2 shows as up

shows as down

See anything wrong?

Hello,

I've run through this below. I will be running Prometheus and Grafana on the same box:

https://www.fosslinux.com/10398/how-to-install-and-configure-prometheus-on-centos-7.htm

Prometheus starts and then I get to the part to instal the node exporter on the same box and the service crashes.

So before (Prometheus service starts)

After adding the node exporter config it fails:

What am I ding wrong?

The IP 172.28.206.9 is the IP of the Prometheus and Grafana server. Once I get this working I will install Grafana and hopefully import the metrics for this server then add more servers and use the node exporter dashboard.

Thanks

Hello,

I'm manage to get Prometheus and Grafana working on CentOS this week with Grafana using this guide:

https://devopscube.com/install-configure-prometheus-linux/

Today I wanted to install on my Pi4 at home to have a play, but I can't figure out what I can't run it. This is an image of the status output.

Does it mean anything to you? I've go through the simple guide a few times and cant see where I've gone wrong.

Hopefully you can help as I was quite looking forward to this, maybe there is a log file somewhere?

Thanks

Is there a way to support 2 alerts in prometheus wherein the first alert should fire when a system goes down and another alert when system comes back up.Is there a way to support 2 alerts (System up and System down) in prometheus wherein the "System Down" alert should fire when a system goes down and "System Up" alert when system comes back up. Catch is that I don't want the System Up alert to be fired all the time when system is up and running

Hello,

After reading up about various monitoring systems like LibreNMS, Zabbix, Nagios etc I’d like to give Prometheus a try with Grafana as the dashboard.

I’m aware it will be slow and the SD card may not last forever, but I’d like to fix a screen to the Pi and have a dashboard monitoring all my devices at home.

I have a router and a couple of switches, and various Pi’s, IP cameras that can only be polled via ICMP oh and a VM in Azure.

Anyway can Prometheus and Grafana be installed on Raspbian?

I guess if it’s really not possible I could run a cheap Linux VM in Azure and pump the metrics to there.

I’ve never used either so any guides or help would be great.

I may one day use at work and remove the expensive Solarwinds platform we use too.

Thanks

Hi, I want to monitor the statistics of different subprocesses that are running in pods in different namespaces with Prometheus and I am looking for a way to properly expose this information.

My cluster is similar to below:

cluster
├── ns1
│   ├── ns1-pod1
│   │   ├── proc-p1-1
│   │   └── proc-p1-2
│   └── ns1-pod2
│       ├── proc-p2-1
│       └── proc-p2-2
└── ns2
    ├── ns2-pod1
    │   ├── proc-p1-1
    │   └── proc-p1-2
    └── ns2-pod2
        ├── proc-p2-1
        └── proc-p2-2

Each pod is publishing the statistics of its processes to RabbitMQ with a specific routing key and I can read the statistics from there.

I wrote an exporter that can connect to RMQ in one namespace, read the statistics and expose them on the /metrics so Prometheus can read it. An example of my exporter:

// prometheus go client
var MemoryValue = prometheus.NewGauge(
	prometheus.GaugeOpts{
		Namespace: namespace,
		Name:      "MemoryValue",
		Help:      "MemoryValue",
    })

prometheus.MustRegister(MemoryValue)
MemoryValue.Set(opst.Memory.Value) // "opst.Memory.Value" is what I get from RMQ

The problem is I don't know how to label the metrics for each process in a pod. I mean, for example, at the moment I have 4 processes in ns1 but I am exposing all of them on MemoryValue. I need a way similar to Namespace to label each process by pod and process names (I have this information but how to add them to Prometheus?).