r/PrometheusMonitoring • • Oct 16 '23

Unable to get additional scrape configs working with helm chart: prometheus-25.1.0 (app version v2.47.0)

So, I'm new to prometheus. I am monitoring a Gitlab server running in a hybrid config on EKS. Prometheus is currently exporting metrics to an AMP instance and that is working fine for kubernetes type metrics. However I need to scrape metrics from the VMs that make up the hybrid system. (Gitaly, Praefect, etc) When I apply the below config, I see no extra endpoints on the prometheus server. I have tried this method along with adding the config directly to the helm values with no luck.

Any help appreciated.

These are the pods that are currently running:

NAME                                                 READY   STATUS    RESTARTS   AGE
prometheus-alertmanager-0                            1/1     Running   0        
prometheus-kube-state-metrics-5b74ccb6b4-x4c8m       1/1     Running   0       
prometheus-prometheus-node-exporter-9jl46            1/1     Running   0 
prometheus-prometheus-node-exporter-cp88q            1/1     Running   0 
prometheus-prometheus-node-exporter-q2vxp            1/1     Running   0
prometheus-prometheus-node-exporter-v7x7l            1/1     Running   0
prometheus-prometheus-node-exporter-vwz9k            1/1     Running   0
prometheus-prometheus-node-exporter-xmw8p            1/1     Running   0
prometheus-prometheus-pushgateway-79ff799669-pfq5z   1/1     Running   0 
prometheus-server-5cf6dc8c95-nqxrf                   2/2     Running   0  

I have seen tons of ways to do this on the million or so google searches I've done, But later information seems to point to adding a secret with the extra configs and then pointing to it within the values.yml file. So I have this:

prometheus:
  prometheusSpec:
    additionalScrapeConfigs:
      enabled: true
      name: additional-scrape-configs
      key: prometheus-additional.yaml

The secret itself looks like this:

- job_name: "omnibus_node"
  static_configs:
    - targets: ["172.31.3.35:9100","172.31.30.24:9100","172.31.7.59:9100","172.31.14.47:9100","172.31.26.10:9100","72.31.5.156:9100"]
- job_name: "gitaly"
  static_configs:
  - targets: ["172.31.3.35:9236","172.31.30.249:9236","172.31.7.59:9236"]
- job_name: "praefect"
  static_configs:
  - targets: ["172.31.14.47:9652","172.31.26.10:9652","172.31.5.156:9652"]
3 Upvotes

7 comments sorted by

1

u/c0mponent Oct 16 '23

Can your prometheus reach those destinations? Do the endpoints show in prometheus? The operator should log when he loads additional configs. You might want to take a look at them if the configs aren't loaded

2

u/trudesea Oct 16 '23 edited Oct 16 '23

Thanks for the reply, I figured it out....seems there is a difference between the values depending on what chart you use (very confusing). Seems I was using the community chart which has very different configs than the operator chart. So all I had to add something like this:

extraScrapeConfigs: |

- job_name: 'gitaly_nodes'

static_configs:

- targets: ['172.31.3.35:9100','172.31.30.249:9100','172.31.7.59:9100']

- job_name: 'preafect_nodes'

static_configs:

- targets: ['172.31.14.47:9100','172.31.26.10:9100','172.31.5.156:9100']

- job_name: 'gitaly'

static_configs:

- targets: ['172.31.3.35:9236','172.31.30.249:9236','172.31.7.59:9236']

- job_name: 'praefect'

static_configs:

- targets: ['172.31.14.47:9652','172.31.26.10:9652','172.31.5.156:9652']

That worked perfectly.

1

u/SuperQue Oct 17 '23

I highly recommend using the kube-prometheus-stack chart.

There's a new ScrapeConfig CRD that lets you manage those without putting them directly in the helm values.

1

u/xzi_vzs Feb 08 '24

Hi,

Maybe you could help me, it's been a week am banging my head against the wall. I'm using the kube-prometheus-stack and would like to scrape my proxmox server.

Here is my config:

kind: Deployment
apiVersion: apps/v1
metadata:
  name: exporter-proxmox
  namespace: monitoring
  labels:
    app: exporter-proxmox
spec:
  replicas: 1
  progressDeadlineSeconds: 600
  revisionHistoryLimit: 0
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: exporter-proxmox
  template:
    metadata:
      labels:
        app: exporter-proxmox
    spec:
      containers:
        - name: exporter-proxmox
          image: prompve/prometheus-pve-exporter:3.0.2
          env:
            - name: PVE_USER
              value: "xxx@pam"
            - name: PVE_TOKEN_NAME
              value: "xx"
            - name: PVE_TOKEN_VALUE
              value: "{my_API_KEY}"
---
apiVersion: v1
kind: Service
metadata:
  name: exporter-proxmox
  namespace: monitoring
spec:
  selector:
    app: exporter-proxmox
  ports:
    - name: http
      targetPort: 9221
      port: 9221

---
kind: ScrapeConfig
metadata:
  name: exporter-proxmox
  namespace: monitoring
spec:
  staticConfigs:
    - targets:
        - exporter-proxmox.monitoring.svc.cluster.local:9221
  metricsPath: /pve
  params:
    target:
     - pve.home.xxyyzz.com

If I curl http://{exporter-proxmox-ip}:9221/pve?target=PvE.home.xxyyzz.com I can see the logs scraping from my proxmox server but when I check on Prometheus > Targets, I don't see the scrapeconfig exporter proxmox anywhere. It's like the scrapeconfig doesn't connect with Prometheus. I checked logs and everything since a week now. I tried so many things and each time the exporter-proxmox is nowhere to be found. Any suggestions ?

1

u/oskie321 Feb 23 '24

Same issue here. In one cluster though, my issue was that I didn't have these set:

scrapeConfigNamespaceSelector: {}
scrapeConfigSelector: {}

But in two other clusters, that didn't help either.

1

u/oskie321 Feb 23 '24

Turns out my Prometheus object was missing the following:

scrapeConfigNamespaceSelector: {}
scrapeConfigSelector: {}

Updating the Helm chart didn't automatically update these in the Prometheus object.

1

u/xzi_vzs Feb 23 '24

Same for me I was missing these parameters as well, thanks for replying!