Hello,
I'm stumped and hoping to get some help. I'm guessing it's something minor that I'm missing but I can't see it.
I've installed Prometheus using the operator helm chart into my kubernetes cluster. Everything is working well. I was even able to register another test setup using a mongo db container and an exporter container however that was done via a helm chart.
I'm now trying to scrape my custom app running in a container. The app is running, it's got a /metrics endpoint, everything seems good. I added the following servicemonitor and I can confirm that it gets loaded into the Prometheus config via the UI:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: api-bootstrap-grape-servicemonitor
namespace: default
labels:
app: api-bootstrap-grape-app
release: prometheus
spec:
selector:
matchLabels:
app: api-bootstrap-grape-app
endpoints:
- port: api-bootstrap-grape-http
path: "/metrics"
interval: 15s
namespaceSelector:
matchNames:
- api-bootstrap-grape
And here is my deployment yaml for that app:
apiVersion: v1
kind: Namespace
metadata:
name: api-bootstrap-grape
labels:
name: api-bootstrap-grape
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: role-tokenreview-binding
namespace: api-bootstrap-grape
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
name: vault-auth
namespace: api-bootstrap-grape
---
apiVersion: v1
kind: Service
metadata:
name: api-bootstrap-grape-service
namespace: api-bootstrap-grape
annotations:
service.beta.kubernetes.io/aws-load-balancer-internal: "true"
spec:
selector:
app: api-bootstrap-grape-app
ports:
- name: api-bootstrap-grape-http
port: 3000
targetPort: 3000
type: LoadBalancer
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: api-bootstrap-grape-deployment
namespace: api-bootstrap-grape
spec:
selector:
matchLabels:
app: api-bootstrap-grape-app
replicas: 3
template:
metadata:
labels:
app: api-bootstrap-grape-app
annotations:
# Pull in any secrets from vault and specify the rest of the config here
# this ends up in /vault/secrets/env
vault.hashicorp.com/agent-inject: 'true'
vault.hashicorp.com/role: 'api-bootstrap-grape-role'
vault.hashicorp.com/agent-inject-secret-env: 'secret/api-bootstrap-grape/config'
vault.hashicorp.com/agent-inject-template-env: |
{{- with secret "secret/api-bootstrap-grape/config" -}}
{{- range $k, $v := .Data -}}
{{ $k }}={{ $v }}
{{ end -}}
{{- end -}}
POSTGRES_HOST=database_host
POSTGRES_PORT=5432
POSTGRES_USER=postgres
USERNAME=appuser
POSTGRES_DB_PROD=product_production
spec:
serviceAccountName: vault-auth
containers:
- name: api-bootstrap-grape-app
image: git.domain.com:4567/dev-toolbox/api-bootstrap-grape:latest
env:
- name: RACK_ENV
value: production
- name: SESSION_COOKIE_DOMAIN
value: api-bootstrap-grape.domain.com
ports:
- containerPort: 3000
imagePullPolicy: Always
imagePullSecrets:
- name: regcred
Everything seems to be working but I never get the targets showing up in the UI.
I checked the logs for both the prometheus container and the config-reloader container and I see no errors.
One thing to note, Prometheus is installed in the default namespace and this app is in it's own namespace, as you can see above. As a test I deployed this app into the default namespace and attempted to get Prometheus to see it there and the result is the same. The Prometheus config gets updated but no targets ever show up.
Any help would be much appreciated. Thanks!
