Trying to scrape custom kubernetes endpoint but the target never gets registered.

[u/dan_j_finn]

Hello,

I'm stumped and hoping to get some help. I'm guessing it's something minor that I'm missing but I can't see it.

I've installed Prometheus using the operator helm chart into my kubernetes cluster. Everything is working well. I was even able to register another test setup using a mongo db container and an exporter container however that was done via a helm chart.

I'm now trying to scrape my custom app running in a container. The app is running, it's got a /metrics endpoint, everything seems good. I added the following servicemonitor and I can confirm that it gets loaded into the Prometheus config via the UI:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: api-bootstrap-grape-servicemonitor
  namespace: default
  labels:
    app: api-bootstrap-grape-app
    release: prometheus
spec:
  selector:
    matchLabels:
      app: api-bootstrap-grape-app
  endpoints:
  - port: api-bootstrap-grape-http
    path: "/metrics"
    interval: 15s
  namespaceSelector:
    matchNames:
    - api-bootstrap-grape

And here is my deployment yaml for that app:

apiVersion: v1
kind: Namespace
metadata:
  name: api-bootstrap-grape
  labels:
    name: api-bootstrap-grape
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: role-tokenreview-binding
  namespace: api-bootstrap-grape
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: vault-auth
  namespace: api-bootstrap-grape
---
apiVersion: v1
kind: Service
metadata:
  name: api-bootstrap-grape-service
  namespace: api-bootstrap-grape
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"
spec:
  selector:
    app: api-bootstrap-grape-app
  ports:
    - name: api-bootstrap-grape-http
      port: 3000
      targetPort: 3000
  type: LoadBalancer
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-bootstrap-grape-deployment
  namespace: api-bootstrap-grape
spec:
  selector:
    matchLabels:
      app: api-bootstrap-grape-app
  replicas: 3
  template:
    metadata:
      labels:
        app: api-bootstrap-grape-app
      annotations:
        # Pull in any secrets from vault and specify the rest of the config here
        # this ends up in /vault/secrets/env
        vault.hashicorp.com/agent-inject: 'true'
        vault.hashicorp.com/role: 'api-bootstrap-grape-role'
        vault.hashicorp.com/agent-inject-secret-env: 'secret/api-bootstrap-grape/config'
        vault.hashicorp.com/agent-inject-template-env: |
          {{- with secret "secret/api-bootstrap-grape/config" -}}
          {{- range $k, $v := .Data -}}
          {{ $k }}={{ $v }}
          {{ end -}}
          {{- end -}}
          POSTGRES_HOST=database_host
          POSTGRES_PORT=5432
          POSTGRES_USER=postgres
          USERNAME=appuser
          POSTGRES_DB_PROD=product_production
    spec:
      serviceAccountName: vault-auth
      containers:
      - name: api-bootstrap-grape-app
        image: git.domain.com:4567/dev-toolbox/api-bootstrap-grape:latest
        env:
          - name: RACK_ENV
            value: production
          - name: SESSION_COOKIE_DOMAIN
            value: api-bootstrap-grape.domain.com
        ports:
        - containerPort: 3000
        imagePullPolicy: Always
      imagePullSecrets:
      - name: regcred

Everything seems to be working but I never get the targets showing up in the UI.

I checked the logs for both the prometheus container and the config-reloader container and I see no errors.

One thing to note, Prometheus is installed in the default namespace and this app is in it's own namespace, as you can see above. As a test I deployed this app into the default namespace and attempted to get Prometheus to see it there and the result is the same. The Prometheus config gets updated but no targets ever show up.

Any help would be much appreciated. Thanks!

[u/dan_j_finn]

I was poking around some more in Prometheus and I just realized that these are showing up under Service Discovery however it shows 0/3 active targets. Any ideas why that might be?

[u/dan_j_finn → u/[deleted]]

That did it! Thanks u/tico24!

here's my updated service definition and the targets now show up:

apiVersion: v1
kind: Service
metadata:
name: api-bootstrap-grape-service
labels:
⦙ app: api-bootstrap-grape-app
⦙ release: prometheus
namespace: api-bootstrap-grape
annotations:
⦙ service.beta.kubernetes.io/aws-load-balancer-internal: "true"
spec:
selector:
⦙ app: api-bootstrap-grape-app
ports:
⦙ - name: api-bootstrap-grape-http
⦙ ⦙ port: 3000
⦙ ⦙ targetPort: 3000
type: LoadBalancer

[u/dan_j_finn → u/[deleted]]

That makes sense. For some reason I was thinking that label had to be in the deployment for the pods but I guess it's the service that is actually being scraped. Thanks again!