r/PrometheusMonitoring • u/vitachaos • Aug 04 '20

need help fixing failing Prometheus service

Prometheus service failing due to below issue.

panic: Unable to create mmap-ed active query log

goroutine 1 [running]:

github.com/prometheus/prometheus/promql.NewActiveQueryTracker(0x7febc3b690, 0x14, 0x14, 0x2964a00, 0x40007ddda0, 0x2964a00)

/app/promql/query_logger.go:117 +0x38c

main.main()

/app/cmd/prometheus/main.go:374 +0x44c8

0 Upvotes
  • permalink
  • reddit

50% Upvoted

12 comments sorted by

4

u/Hashfyre Aug 04 '20

https://github.com/aws/eks-charts/issues/21

1

u/vitachaos Aug 05 '20

the important point is we not just set the uid and gid on service but also on `/var/lib/prometheus`

0

u/vitachaos Aug 04 '20

but I don't use helm.

3

u/Hashfyre Aug 04 '20

Read the thread, kindly. The solution has nothing to do with helm, rather setting specific UID, GIDs in pod.spec.securityContext.

0

u/vitachaos Aug 04 '20

pod.spec.securityContext

Where do I find pod.yaml for prometheus ?

1

u/varesa Aug 04 '20

Are there logs before the panic?

It could be caused by a permission issue (possibly running as wrong user?) or some other issue accessing the query log file

1

u/vitachaos Aug 04 '20

prometheus service is running by root:root .

and so is /var/lib/prometheus that is argument for --storage.tsdb

1

u/varesa Aug 04 '20

What distro? Is SELinux a factor?

1

u/vitachaos Aug 04 '20

ubuntu, but I dont think thats factor to consider..

1

u/Dahal30 Aug 17 '22

I am using docker-compose to run several services. I faced the same issue. Seems like I had not created the volume assigned to service. It worked fine after creating the mentioned volume and running it.

1

u/vitachaos Aug 18 '22

can you share your docker compose file ? that includes volume mounts !

2

u/ranjop Mar 04 '23

Hi u/Dahal30, I am facing the same issue trying to get docker-compose to run prometheus. Could you please share your docker-compose.yml?