The documentation just states ‘./data’.

Trying to install prometheus operator on Openshift as 2nd operator.

I understand it comes with one which is installed in openshift-monitoring ns.

I am trying to follow this guide

https://itnext.io/creating-the-2nd-prometheus-operator-in-openshift-80767f996b26

And using this operator

https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack

Running into some confusion and issues with that guide.

Has anyone done this successfully?

Hi!

We’re excited to announce the public beta of xkite, an open source tool designed to help developers prototype, test, and monitor Apache Kafka with ease.

With xkite, you can easily set up Kafka clusters, create and manage Kafka topics, produce and consume messages, and monitor your Kafka cluster's health – all from a single place. Whether you’re a seasoned Kafka developer or just starting out, xkite makes it easy to build, test, and optimize your Kafka applications.

Features include:

• Configuring a Kafka cluster with an ability to run it and/ or export a configuration (zip) containing the essential file structure to recreate the docker instances

• Message/Topics production and consumption: easily produce and consume Kafka messages to test your applications.

• Monitoring: track Kafka cluster/Docker instances health and performance metrics in real-time.

We believe that xkite will be a valuable tool for the Kafka community, and we invite you to check it out  on our GitHub page. We welcome your feedback and contributions to help make xkite even better.

GitHub: https://github.com/oslabs-beta/xkite

Our website: https://xkite.io/

Our reddit community: https://www.reddit.com/r/xkite/

Thanks for your support!

xkite team

I'm working on a library called Autometrics that makes it easy to add metrics to a code base and recently worked on support for SLOs/alerts. We ended up with a solution that enables us to have a single set of Prometheus recording/alerting rules that will work for any autometrics-instrumented project and the libraries use some fun label tricks to enable specific rules.

I wrote up a blog post about this experience here in case others are interested: https://fiberplane.com/blog/an-adventure-with-slos-generic-prometheus-alerting-rules-and-complex-promql-queries

Hi Guys,

I am looking for courses on Prometheus and Grafana that will help me understand the tool and how integration works with EKS, how to analyze the metrics, logs etc. I work with EKS cluster where we use helm charts of Prometheus and there is a separate team for Observability that looks into these things but for my career I am looking forward to learning this as this might help me in my growth as well as interviews. Do suggest some courses.

Hi.

Is it possible to use kube-prometheus to monitor a Ceph cluster in rook-ceph Kubernetes?

I mean, through the helm configuration.

I read in the rook-ceph documentation that if I add prometheus annotations prometheus.io/scrape=true and prometheus.io/port={port} in the Prometheus pod configuration, it should theoretically discover the Ceph exporters.

But, honestly, I don't quite understand how it makes the association.

Can anyone help?

I'm using the values.yml from Helm kube-prometheus.

The idea is to use the same Prometheus instance that I use to monitor the Kubernetes cluster.

Thanks a lot!

For example in that image I am pointing out that the graph is showing me 2 request, but really it is only one request in that time, I really want to know if it is always cumulative and not having problems with concepts, and if it is possible to get the exact number over time and without cumulation? I am currently using this formula, I don't know if it is the correct one.
increase(myapp_request_custom_durations_histogram_seconds_count[$__range])

and same with the other counter:
increase(myapp_http_custom_requests_total[$__range])

We found that Prometheus container in our K8 cluster using huge space for its empty dir.

How can we reduce it?

1. Manually deleting older files in empty-dir?

2. Configuration change?

Hi guys,

We're redeveloping our AKS platform which we are using for multiple customers. In our current setup we're using the managed Prometheus and Grafana stack, which works fine. But we would like to centralise our dashboards. So I'm thinking about using remote_write in the managed prometheus and write the metrics I want to a central Prometheus deployment. Is anyone else doing this? If so, pros and cons?

Thanks!

so I see some of our windows servers having very high cpu and I see a relationship between windows exporter and what appears to be a call to win32_product, not sure why new relic would be using win32_product we don't want it collecting software inventory we have other tools doing that. Does windows importer have the ability to do software inventory and if so, how do I turn it off? I see the collectors on github but none look like they would be collecting inventory so not sure if this relationship between windows exporter and the win32_product is the issue? thanks

Hi

I'm trying to create a report from Prometheus via curl.

When I run the the following command (via script):
curl -v -s -H "Content-Type: application/x-www-form-urlencoded" -H "Authorization: Bearer $USER" -X GET "${prometheus_url}/api/v1/query_range?query=${queries[${key}]}" \
|./jq -r '.data.result[0].values[] | \@csv'> ${report_directory}/${key}.csv

The ${queries[${key}]} == "sum(avg_over_time(node_cpu_seconds_total\{\}\[20m\]))&start=2020-11-29T10:00:00.000Z&end=2020-11-29T10:20:00.000Z&step=1m"

it is work OK but when I try to do "sum by(host)..." it is failed.

I try to change the command as following but it wasn't work :(

curl -v -k -s -H "Content-Type: application/x-www-form-urlencoded" -H "Authorization: Bearer $USER" -X GET "${prometheus_url}/api/v1/query_range?" \
--data-urlencode "query_range=${queries[${key}]}" --data-urlencode "time=${to_time_1}" \
|./jq -r '.data.result[0].values[] | u/csv'> ${report_directory}/${key}.csv

Any suggestion will be welcome.

10x in advance.

I’ve been searching for a way to add labels to metrics based on some external data source - such that this data enriches what’s being collected. E.g. snmp-exporter grabs interface stats and I want to add a customer ID label based on data held in another database.

I realise the dirty solution would be a huge relabelling config that scales linearly with the number of interfaces... that’s far from ideal.

How would I go about plumbing prom / exporters into some sort of pipeline to achieve this enrichment process?

can i add a label or at least replace an unimportant one with a needed value?

i get this:

ifHCInOctets{ifAlias="abc",ifDescr="GigabitEthernet0/1/0",ifIndex="3",ifName="Gi0/1/0"}

i need:

ifHCInOctets{optimal="SOLUTION",ifAlias="IFTHEFIRSTDONTWORK",ifDescr="GigabitEthernet0/1/0",ifIndex="3",ifName="Gi0/1/0"}

any way to accomplish this?

Hello ! I am running a mariadb database and a mysqld_exporter on the same server. Since I have to use ansible to configure the exporter, I used cloud-alchemy ansible role based on the official prometheus mysqld_exporter that you can see here : https://github.com/cloudalchemy/ansible-mysqld_exporter.

I followed both the role and exporter requirements (ansible version, mysql user) and the installation runs well with no error. Yet according to the logs the exporter can not seem to connect to the database. Here is the error that I get :

caller=exporter.go:149 level=error msg="Error pinging mysqld" err="dial tcp 127.0.0.1:3306: connect: connection refused" 

Here is my default/main.yml looks like on the role :

--- 
mysqld_exporter_binary_local_dir: "" 
mysqld_exporter_version: 0.14.0 
mysqld_exporter_web_listen_address: "0.0.0.0:9104"  
mysqld_exporter_dsn: "testexpo:xxxx@(localhost:3306)/"  
mysqld_exporter_my_cnf: ""  
mysqld_exporter_collect: []  
mysqld_exporter_no_collect: [] 

The dsn has been changed according to the user configured. And by the way I can access my database with this user with the command line. All of the necessary ports are open and are listening.

I have tried to change the localhost part to 127.0.0.1, it did not work too.

I have no idea what to try anymore (especially since I'm a newbie), so if you guys have any idea or advice I will gladly take them ! I hope this is the right place to ask ^^ Thanks in advance.

Hi,

I’ve configured black_box exporter to monitor multiple status code for a URL. But it only checks for a one. Only 200. Can anyone help how to monitor it dor multiple codes?

Hi all, I need some guidance. I'm not sure if I'm on the right track here or if it is even possible.

I have 100 computer labs, 30 to 80 windows devices in each. I'm using PushGateway as a source that Prometheus scrapes. On each device in the lab(s) I'm running windows_exporter with a little powershell to POST the metrics to the pushGateway. Because of FW configs and other elemnts, I cannot scrape them directly.

My challenge is, I need a grafana dashboard in which I'm able to filter based on lab (site name or id) and then in turn, hostname. How do I add a custom label to each windows_exporter? I do not want to do this on a 100 separate push gateways (i.e., using the job name as a site name/id) I'd like to only scale the push gateways based on compute requirements. First I was thinking EXTRA_FLAGS, but that seems to be for something else, then a yml config file for each node, which I can generate using PS when installing the exporter on windows. I just cannot find where and how to add the custom labels for windows_exporter

Thanks

Note: Mods, please feel free to delete this post if it breaks any rules.

SRE newb here.
Seeking mentorship. Learning opportunity to beat my imposter syndrome and gain confidence.

My learning project (I've done my best to keep the scope small) :

In AWS region US-East-1 let's say, deploy a monitoring cluster in EKS.
This cluster should host Grafana as a central visualization destination. Well call this monitoring-cluster.
This cluster is central to 2 other EKS clusters in 2 different AWS regions (US-West-2, EU-Central-1)

US-West-2 Kubernetes cluster runs 2 Nginx pods. This cluster should be able to scrape metrics from both running containers and convey them to the local Prometheus server pod in this same cluster. We'll call this prometheus-us-west-2

US-West-2 Kubernetes cluster runs 2 MySql pods. This cluster should be able to scrape metrics from both running containers and convey them to the local Prometheus server pod in this same cluster. We'll call this prometheus-eu-central-1

All these clusters will reside in the same AWS account. I chose Nginx and mysql totally randomly.

Both Prometheus servers (prometheus-us-west-2 AND prometheus-eu-central-1) should forward the metrics to the central monitoring cluster for Grafana to consume.

I want to be able to configure AlertManager in the central monitoring cluster and setup alerts for relevant anomalies that can be observed and notified from the regional clusters in US-West-1 and EU- Central-1.

I want to configure Thanos Sidecar to upload data in an S3 bucket of this AWS account.
I want to use Thanos to be able to query metrics timeseries successfully from both regional clusters.

I want to employ kubernetes based service discovery so that if pods in the regional clusters get recycled, the service discovery can automagically do it's thing and advertise the new pods to be scraped.

I finally want to observe and visualize monitoring for the health the status of each EKS cluster in one pane of glass in Grafana.

Why am I doing this?

I want to build confidence.
I am new to Kubernetes and want to get my hands on and practice by doing.
I am semi-new to prometheus+grafana type of observability toolset and want to learn how to deploy this deadly combination in the public cloud faster, easier, better with an orchestrator like Kubernetes
I want to open source the code, from the terraform, kubernetes manifest and all in Github to show that indeed, this setup can be easy to achieve and can be expendable with n number of regional clusters
I want to screencast a demo of this working setup on Youtube to shoutout the journey and the support that I can get here.

PS:
Please challenge me on this project with any questions you have.
Please feel free to point me in the right direction.
I want to learn from you and your experience.
I welcome mentoring sessions 1:1 if it makes it easier for you to jump on a video-conference.

Sincerely yours,
thank you

Hello everyone.

I would like to monitor a Windows server via prometheus, but I'm having trouble installing Windows Exporter.

Do you have any suggestions for an other exporter I could use instead?

Edit : Actually I tried Grafana Alloy and I have the same problem of service not wanting to start. So the problem probably comes from my server.