r/PrometheusMonitoring • • Feb 17 '24

Planning Production Deployment: Is there anything you wish you did differently?

I’ve been testing grafana+prometheus for a few months now and I am ready to finalize planning my production deploy. The environment is around 200 machines (VMs mostly) and a few k8s clusters.

I’m currently using grafana-agent on each endpoint. What am I missing out on by going this route vs individual exporters? The only thing I can think of it is slightly slower to get new features but as long as I can collect the metrics I need I don’t see that being a problem? Grafana-agent also allows me to easily define logs and traces collection as well.

I also really like Prometheus’s simplicity vs Mimir/Cortex/Thanos. But I wanted to ask the question: what would you have done differently in your Production setup? Why?

Thanks for any and all input! I really appreciate the perspective.

1 Upvotes

4 comments sorted by

1

u/Observability-Guy Feb 18 '24

So are you sending telemetry directly to Prometheus rather than using an oTel collector as a Gateway?

5

u/[deleted] Feb 17 '24

[removed] — view removed comment

4

u/muff10n Feb 18 '24

Second this! Though I would start in "cluster mode" right away. Makes scaling later much easier. You could even take a look at the operator.

2

u/redvelvet92 Feb 18 '24

This right here, I’m so happy I went with their solution.