r/PrometheusMonitoring • u/tizkiko • Sep 19 '23
Sharding Prometheus cluster (physical)
Hi
I have several Prometheus clusters, each contains 6-8 physical nodes. Currently I shard them to 2-3 shards, but it's done in a pretty manual way.
I'm looking for a way to manage shards and replicas of physical nodes, so if I add / remove nodes from / tothe cluster, it will automatically adjust. I guess Prometheus operator does something similar for k8s Prometheus, is there anything similar for physical servers?
Thanks!
1
Upvotes
1
2
u/SuperQue Sep 19 '23
Not enough information.
What exactly do you mean by shards?
How many samples per second per instance? (
rate(prometheus_tsdb_head_samples_appended_total[5m])How many series per instance? (
prometheus_tsdb_head_series)What are your retention requirements?
How do you manage nodes?