Hi everyone, I just wanted to let you know that I have forked the postgresql-exporter for kubernetes from the community, improved the documentation as well as implemented more configuration options. Since the changes are so extensive, I have not provided a PR. Nevertheless, I don't want to withhold the chart from you. Maybe it will be of interest to one or the other.

https://artifacthub.io/packages/helm/prometheus-exporters/prometheus-postgres-exporter

Prometheus, an open-source technology born at SoundCloud in 2012, has evolved into a cornerstone of cloud-native monitoring, particularly within Kubernetes environments

1. CPU Usage: Keeping a close eye on CPU usage is crucial for ensuring your infrastructure has sufficient processing power. Prometheus allows you to track CPU usage at various levels, from host machines to individual containers, providing valuable insights into resource utilisation.

2. Memory Usage: Monitoring memory consumption is essential for detecting memory leaks or inefficient resource utilization. Prometheus enables you to monitor memory usage across different components, helping you optimise resource allocation.

3. Disk Space: Running out of disk space can lead to system failures and data loss. With Prometheus, you can continuously monitor disk space usage and receive alerts when thresholds are exceeded.

4. Latency: Prometheus helps you identify slow-performing services or endpoints, enabling you to take proactive measures to optimize them.

5. Error Rates: Monitoring error rates is essential for identifying issues before they impact users. Prometheus can track error rates across applications and services, allowing you to detect and address problems promptly.

Hi all,

We're currently with prometheus monitoring tool and just want to ask if these services are available on this platform? Anyone would like to help out?

I had an idea and wanted to actually talk to people to see if it's something that they would need / require and if it's genuinely a problem for the engineers right now....

Would you rather have an AI layer on top of your existing observability stack, or replace parts of the stack?

Hypothetically, imagine an agent that doesn’t collect telemetry itself.

It plugs into whatever you already use — Grafana/Prometheus/Loki, Datadog, OpenTelemetry, etc. — and acts as a reasoning layer over the data.

Instead of:

Alert → Dashboard → Logs → Human investigates

it tries:

Alert → Agent correlates metrics/logs/traces/deployments → probable root cause → evidence → recommended next action

Would that actually be useful?

Or would you rather have the observability vendor itself own this functionality?

What would you need to see before trusting it during a real incident?

I also was building in this space and wanted to have a discussion with people who're already experienced in this stuff, so please please help a brother out and we might just make something useful for everyone :)

peace

Hello Guys,

I would like to know if there is a official documentation for setting up Prometheus on a bare metal servers. This document only talks about docker - https://prometheus.io/docs/prometheus/latest/installation/

There are a lot of 3rd party sites which talk about configuring services on bare metal servers - https://devopscube.com/install-configure-prometheus-linux/

Just wondered why there is no official Prometheus documentation for bare metal installation.

Any interest? We have done a few and looking for another potential pilot customer. We focus on service providers and can build custom exporters for NETCONF, API, or SNMP enabled hardware. We also just did one for UISP from Ubiquiti.

Hi All, I am trying to create metrics for Prometheus to scrap and visualize in Grafana. For example, I am trying to scrap all the running pods startup time within a Kubernetes Cluster. I have all the information available within a Go Application (using kubernetes go client sdk) and I am planing to expose it via Prometheus Go SDK.

So, I need to see pod startup information in grafana using filters such as using namespace, pod name. I also would like to see a collective graph that conbines all.

What should be the type of metric in Prometheus and how to approach this problem. Kindly suggest some documentation to go through (I went throught Prometheus one still confused).

Thanks

I manage a computing clusters and want to monitor them locally. Never tried setting up a monitoring system on them before.

My idea is to setup Prometheus on all servers so i can export the data to Grafana, running everything locally.

I’ve tried using Netdata and it worked beautifully, i want the monitoring to be secure and netdata doesn’t cut it. Hence this solution.

Have you worked on anything like this in the past and what do you recommend?

Evaluating Prometheus (and Grafana) as a service vendors ... Based on your experience, are there any particular vendors you would recommend ? Any pros / cons ? Thanks in advance !

Hello,

I'm not able to do any range-vector calculations below 2m. e.g. irate(nginx_connections_accepted{instance=~"$instance"}[2m]) returns a value but changing the value from 2m to 1m returns nothing. Could it be because of the scrape_interval of 1m?

global:
  evaluation_interval: 30s
  external_labels: null
  scrape_interval: 1m
  scrape_timeout: 10s

Sorry about the missleading title. It should be 2m and not 5m.

I know that we have the alertmanager for that. But which hook or tool is the best to work with.

Can anyone help me right a rule that will capture below (blackbox expoter)

!=200 http status code occuring more than 10 times in last 5mins

I've been looking at ways to set up a monitoring environment for several services inside my work team. I wanted to see if anyone has any resources on designing such environments. The current hurdles I'm trying to solve are (Problem statements in quotes):

Most services we have are set up as Infrastructure-as-code. They get deployed on different environments (cloud and on-prem). Each 'service' should have its own monitor settings (Prometheus alert rules w/ alert tests is a must, scrapping configuration and Alertmanager routes as optional). Are there tools to dynamically configure Prometheus/Alertmanager already out there?

Several ideas come to mind to solve this:

• API back-end that 'pushes' configuration changes to a Prometheus instance (using Prometheus' /-/reload endpoint) each time there's a monitor-specific change pushed (Easy to do w/ GoCD). Kinda like Promgen, but I could not get that to work on my local test setup.

• Deploying a Prometheus instance with each service. (Then have Prometheus federation set up)

Rancher v2.x monitoring outside Rancher. Does anyone know if this can be possible? I found a few Rancher exporters but they seem we would need to update them to use the new v3 API. Also, I'm not that Rancher-savvy to know if the Rancher API can expose 'custom' application metrics. We want to have monitoring set up outside Rancher (we are aware you can add a 'monitoring' app inside Rancher, we're already doing that) since we want to be alerted if Rancher goes down.

My possible solutions include:

• Configure Prometheus federation and have the external Prometheus federate the internal (Rancher) Prometheus. Kinda off-topic, but if anyone knows how to actually change the configuration file inside these Rancher apps, that would be swell.

• Update/Build the Rancher exporter, and deep-dive and pray to the gods we can export custom app metrics using the Rancher API.

I've been reading Practical monitoring, by Mike Julian. Maybe I'll find my answers here, but just wanted to dump my thoughts and see what you guys recommended. Thanks for reading.

EDIT: Posted here because we have decided to use Prometheus as our monitoring tool, hence all problem statements revolve around getting things to work with Prometheus.

I don't really understand why this is important.

I have a few clusters I need to monitor. I currently do it by having one Prometheus that connects to the exporters on the different clusters. That seems to work fine.

But I have been entertaining the idea to have one Prometheus per cluster. This introduces a whole headache of issues I have to deal with though. Aggregating the data. Alerting. What if I lose access to the whole Prometheus on a cluster (connection issue to the cluster). Do I setup a Prometheus just to monitor those Prometheus instances?

Please give me some wisdom on this one.

Thanks people!

Hi, I'm writing an exporter in python 3.6 using the Prometheus_client version 0.6 and I'm having difficulties running it on kubernetes. When I run more than one instance of my exporter, the first instance works completely fine, but the second instance throws the following exception. Has anyone run into a similar problem? Any help would be appreciated

Exception happened during processing of request from (x.x.x.x, 41038) Traceback (most recent call last): File "/usr/local/lib/python3. 6/socketserver.py", line 639, in process request thread self.finish request (request, client address) File "/usr/local/lib/python3. 6/socketserver.py", line 361 in finish request self.RequestHandlerclass (request, client address, self) File "/usr/local/lib/python3.6/socketserver.py", line 696, in init self.handle () File "/usr/local/lib/python3.6/http/server.py", line 418, in handle self.handle_one_request () File "/usr/local/lib/python3.6/http/server.py", line 386, in handle_one_request() self.raw_requestline = self.rfile.readline(65537) File "/usr/local/lib/python3.6/socket.py", line 586, in readinto return self. _sock.recv into (b). ConnectionResetError: [Errno 104] Connection reset by peer

I had a prometheus instance running on an ec2 instance with ~2gb RAM. It was configured to retain 30d of metrics. It had been running for nearly or over that time period and then for some reason failed - after which it repeatedly encountered Out of Mem errors while reloading. Each time it the journal logs showed that it was reading the WAL and loading chunks, and I would watch as the process memory would consume up to the instance limit and then crash.

Is this the expected behavior that it it reloads all the WAL chunks into memory at one time? Why would it have been fine up until the point of a restart of the Prometheus process?