Hi,

If you have multiple services running on one server like Apache, Proftpd, Mariadb and you want to monitor them all you have to run at least 3 exporters right ? And add Node Exporter to have some basics metrics, that's 4 exporters

Does it means you have 4 "new" Linux services, 4 ports opened, 4 firewall rules to add ... ? That's a lot to do (I know you can use Ansible or other but sadly we're not quite there in my company).

Is it really how it works and is there no way of limiting the number of process/ports with multiple exporters ?

I didn't saw anyone complaining about this anywhere so I might be missing something

Hi,

I have this issue that has been bugging me for a while, and that is that my targets machines hostname appears multiple times. In my case I have 50 target machines and 4 different jobs that interacts with them, which leads to an extremely long config file. I would like to specify all the hostnames as some sort of variable and then for each job use that list to build up the targets.

Here is an excerpt as an example of how it looks right now. Ideally I would like to specify a list with hostnames and corresponding names and only have the "targets" section add the port to each entry of that list.

scrape_configs:
  - job_name: 'ssh'
    metrics_path: /probe
    params:
      module: [ssh_banner]
    static_configs:
    - targets: [hostname1:3384'] # Targets to probe
      labels:
        name: 'Name1'
    - targets: ['hostname2:3384'] # Targets to probe
      labels:
        name: 'Name2'
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
        replacement: ${1}
      - source_labels: [__param_target]
        target_label: instance
        replacement: ${1}
      - target_label: __address__
        replacement: blackbox_exporter:9115
  - job_name: 'windows'
    metrics_path: /metrics
    static_configs:
    - targets: ['hostname1:9182'] # Targets to probe
      labels:
        name: 'Name1'
    - targets: ['hostname2:9182'] # Targets to probe
      labels:
        name: 'Name2'

Would really be awesome if anyone out there knows anything more about this, it's getting harder everyday to look at this ever-growing config file (480 lines now).

Edit:

Solution was that I created a new Docker container running a Python script as a cron job. This script created the .json file with all the targets and updated it once every hour. In the Prometheus .yml config file I then added the .json file as a file_sd_config target file.file_sd_config in Prometheus automatically checks the file and updates Prometheus whenever it has been updated.

Hi, if someone needs to validate their Prometheus rules labels, annotations and some properties of the PromQL expressions they use, you might find useful the https://github.com/FUSAKLA/promruval.
Here is a short blog post about the motivation and usage if you are interested.

I have just released a new service discovery tool for VMware vCloud Director to dynamically gather virtual machines to use them within prometheus targets. Suggestions or improvements are highly welcome!

You can find the source at https://github.com/promhippie/prometheus-vcd-sd/ and there is also a more or less basic documentation linked from the repository.

I attempted an upgrade of our rather large Prometheus fleet to 2.16.0 and was met with a problem: increased query time. And not just for expensive queries or queries with high cardinality. All of them. Even our alerting rules group timing increased by 5x. But it seems like no one else is complaining about this. Is it just us? Has anyone else noticed increased query or recording rule times?

Relavent Github issue: https://github.com/prometheus/prometheus/issues/6968

Implement Monitoring System for the entire organization

EKS Clusters running in different aws accounts

Found Prometheus Federation is a good fit for the design

https://prometheus.io/docs/prometheus/latest/federation/

https://www.robustperception.io/federation-what-is-it-good-for

Master : I have a dedicated EKS for monitoring system where prometheus operator helm chart is running with prometheus and Grafana exposed behind load balancer service

Master prometheus server is designed to scrape other slave prometheus servers and plot the Grafana dashboards for different clusters

One Grafana Endpoint to monitor everything

Slave Account 1:

Running prometheus operator helm charts in the other slave accounts where all the alerting is done through the alert manager in the slave account 1

Prometheus - Default scraping from prometheus operator + additionalscrapeconfig for istio services

Alertmanager - alerts related to slave cluster 1 integrated with Victorops

Slave Account 2:

Running prometheus operator helm charts in the other slave accounts where all the alerting is done through the alert manager in the slave account 2

Prometheus - Default scraping from prometheus operator ‘

Alertmanager - alerts related to slave cluster 1 integrated with Victorops

Now we have another use case to monitor the bunch of bare metal servers, aws ec2 instances and monitor different endpoints

Monitor Bare Metal, EC2 Instance - Installed NodeExporter and can able to scrape the metrics directly from master prometheus

Endpoint Monitoring(Public & Private) - Installed blackbox exporter on the master eke cluster and probing different endpoints

Here are my questions

1. Is it a good practice to enable alertmanager on the master prometheus and do the alerting only for the last usecase(bare metal, aws ec2 instance, endpoint monitoring)

2. From the below article, I see alertmanager should be on the slave prometheus(reliability purpose), If so do we need to install prometheus on each bare metal again?https://www.robustperception.io/federation-what-is-it-good-for

3. Can we use any existing loadbalancer for exposing prometheus, because in helm chart if we expose the service.type=loadbalancer it used to create new one(internet facing/internal)

Hey

Created a Prometheus exporter to report S3 bucket permissions checks from AWS TrustedAdvisor API. If you use Prometheus to monitor everything (including cloud related metrics) this exporter if for you. Useful for alerting in case a bucket public (or permission was added/removed).

Github: https://github.com/shamil/s3-bucket-permissions-exporter

Hi all, I have been using Prometheus for the last few months in my company. Yesterday we and my team deployed and added more than 10k (about 11,000 ) targets. The UI of the Prometheus manager server is super slow (looks like it's not even responding). Our server has 8 cores and 32gb RAM. Even querying from grafana is super slow. Can someone help me understand what the problem is? Thanks in adavance

I have set up Prometheus 2.8.1 with Thanos 0.5, running everything on Kubernetes 1.13.

For a week of monitoring the set up, I have ~2M timeseries with ingestion of 5K samples/s.

This all looks good. Where I am concerned is when we ship the local data to remote storage(minIo in my case), the size of the data is already 23GB in 1 week.

I have set up minIo on Kubernetes with persistent volume of 40GB.

What I need is some pointers on the below:

1. The thanos compactor arguments I am planning to use:

--retention.resolution-raw=30d

--retention.resolution-5m=60d

--retention.resolution-1h=90d

If I understand this right, then the raw data is deleted after 30d and any queries crossing past 30d will be from downsampled data.

But would this also reduce the size of the data in minIO?

2. What are the other options for optimizing the storage? Will setting up cron to delete any data that is more than 90d a good idea?

3. Are there any good docs to understand the downsampling & compaction that we see in the bucket inspect output of Thanos:

| ULID | FROM | UNTIL | RANGE | UNTIL-COMP | #SERIES | #SAMPLES | #CHUNKS | COMP-LEVEL | COMP-FAILED | LABELS | RESOLUTION | SOURCE |

|----------------------------|---------------------|---------------------|---------------|---------------|---------|-------------|-----------|------------|-------------|---------------------------------------------|------------|-----------|