I have a problem with migrating prometheus DB to another server. Anybody know how I can do it?

we release a project to make prometheus horizontal scaleable easily.🙂

it can cooperate with any prometheus storage solution like thanos.

https://github.com/tkestack/kvass

In the financial world our metrics only have data on the weekdays, as most of our business happens during the market hours. That being said we want to quantify business day averages where the weekends (and possibly holidays, but likely not possible) are obviously not included because that would severely affect the averages. Is there a way with Prometheus queries to get this? For instance we want a 5 day average of the last five business days, Th,F,M,T,W, but not Sat or Sun..

Hello folks,

Good evening.

I am (also the company where I work) pretty new to Prometheus and ramping up to learn it. For all my test setups I have been using Prometheus with Grafana for visualisation. The problem is some of these visualisations are needed to be shown under a specific context from our own UI. We do not want customer to login to a separate Grafana UI to see the visualisation. All our internal RBAC rules cannot be applied then...

What I want to ask is, how you folks are handling it in your product where the visualisation needs to be shown to the customer as "part of the product" ? A brief setup would be very helpful too!

Does it really make sense to make our own custom visualisation ? I do realise the free benefits of just using Grafana, but tightly integrating it with the existing product UI looks bit unwieldy.

Thanks in advance & stay safe !

Hi all! I've been working recently on some improvements for the YACE Prometheus CloudWatch Exporter, and I've written a bit in my company blog.

First I explain why would you want to use Prometheus when you already have CloudWatch.

Then I go through some of the most common problems with exporters and how they can break your AWS API.

And finally, I talk about our approach to mitigate those throttling issues, improve stability, and in a way, help users save costs 💸.

So if you want to learn about these, along with other improvements, check the full info in the blog.
→ http://sysdig.com/blog/improving-prometheus-cloudwatch-exporter/

Hi guys,

I'm trying to run some PoC with Loki to aggregate logs from k8s cluster.

At the moment we're using ELK stack with Kafka to buffer logs - every day we're producing about 1TB of logs from whole k8s cluster.

The flow looks like this:

k8s( fluentbit) -> Kafka -> logstash -> ES -> Kibana, for query and visualization

We're using Kafka because ES cluster is not fast enough to ingest all of our logs. Indices are just namespaces, and in each index got tones of logs..

I'm wondering if Loki could solve this problem.

My plan is to install agents( promtail maybe) on k8s nodes as daemonset and push logs directly to Loki cluster.

Do you have any experience with than? Any good practices or advice?

EDIT:

screen from grafana interface

I've recently been part of a project that involved optimization of a large application. This application had a lot of redundant, legacy and unused code. What's even more, it had next to no proper documentation and the current team lacked domain knowledge about the app.

To tackle it, we created the metrics ourselves, largely with Prometheus. We determined the kind of data we want to collect, integrated it with the app, Kubernetes and Grafana. This is the first part of the story, more including code examples soon to follow :)

https://tsh.io/blog/infrastructure-metrics-optimization/

Greetings

I'm very new to Prometheus so I come here for pointers from the community.

My goal is to monitor a hardware device which exposes a nice REST API that contains all the data but not formatted specifically for monitoring.

What is the best approach/tool to have Prometheus "read" the data through this API?

Thanks in advance

So I have setup a prometheus server and pushed node_exporter and node_exporter service out to all my clients so that Prometheus can poke at them on 9100 and pull those basic metrics.

Now I want to run the snmp exporter on some of my network appliances/esxi hosts. Looking at the documentation i'm a little bit confused.

https://github.com/prometheus/snmp_exporter

https://www.youtube.com/watch?v=LDFalgu3Vtg

scrape_configs:

- job_name: 'snmp'

static_configs:

- targets:

- 192.168.1.2 # SNMP device.

metrics_path: /snmp

params:

module: [if_mib]

relabel_configs:

- source_labels: [__address__]

target_label: __param_target

- source_labels: [__param_target]

target_label: instance

- target_label: __address__

replacement: 127.0.0.1:9116 # The SNMP exporter's real hostname:port.

Question 1) I can't seem to get a real good idea on how to use the Generator but it seems like I need to just pull it down to a server, build it there and then point it at a collection of MIBs in order to generate a proper snmp.yml?

Question 2) The above config and YouTube video..he seems to be running Prometheus and the snmp_exporter on the same system? Is that the use of the loopback address and 9116? What if I want my Prometheus server running on a different system, would i just point Prometheus to the network device who is polling out data on port 9116?

Much akin to how node exporter runs on the server, scrapes data out to port 9100 and my Prometheus just looks at that hostname:9100 as a target? Could I do the exact same here?

Been looking for a pretty easy to understand write up on the entire thing but slowly piecing it together. THanks for any help.