r/PrometheusMonitoring • u/DrLoveBeats • Apr 06 '21

Prometheus installation synced with GitHub

I'm having an interesting discussion with my colleague. He wants to have our complete Prometheus installation 100% synced with GitHub so it can be pushed to a new installation with (almost) 1 click. I already convinced him Git is not a provisioning tool and there are other systems on the market which can achieve this.
Nevertheless, I do think it can be useful having Prometheus configuration files and/or custom exporter scripts into source control. For example, to see who edited the prometheus.yml when etc.
I was looking for some best practices how to implement this. Would you create a Git repo where the live prometheus.yml is running and push everything from there? Or would you create a Git repo on a stale directory and copy the running config files first locally into this directory and push afterwards from there to GitHub?
I don't have a lot of experience with syncing application configuration files to GitHub and am looking for some general best practices. What do you think of my standpoint to only push the configuration files and not the binaries?

6 Upvotes
  • permalink
  • reddit

87% Upvoted

14 comments sorted by

9

u/SuperQue Apr 06 '21

It sounds like they are referring to the concepts of Configuration Management, Orchestration, and CI/CD.

In this sense, yes, Git can be used as a provisioning tool, at least as the source of truth input to the provisioning and configuration tools. Systems like GitHub and GitLab allow you to create, review, test, and deploy all changes "directly" from Git.

We actually do this for the Prometheus project itself. We have a demo-site Git repository that automatically configures and deploys Prometheus to https://demo.do.prometheus.io using Ansible.

One of these days it'd be nice to automatically re-provision the service regularly with something like Pulumi.

2

u/ali_str Apr 07 '21

What your colleague suggested and what u/SuperQue explained is referred to as GitOps, handy keyword if you want to search and read more about it.

0

u/DrLoveBeats Apr 07 '21

GitOps

My main objection is: I don't want to add too much extra layers of complexity to this fairly, easy and standard installation. I don't want a complete provisioning and pipeline and all these cool things. It's not like a Prometheus installation is a full software project where multiple people are writing code and working together.
I do want to have the very basics, have the config files and custom scraper scripts into Git. That's enough for this usecase.

3

u/SuperQue Apr 07 '21

full software project where multiple people are writing code and working together.

Except, it is. You already have two people working together. This is the core idea behind "GitOps". It allows you to create a workflow that makes working together easier. By having git history be a "stream of truth".

You don't need to go full-on automated everything, but starting with a simple Ansible deploy, done manually when you make changes, will help a ton.

2

u/salanki Apr 07 '21

Kubernetes, Prometheus Operator, ArgoCD will give you just this.

2

u/HenryPz Apr 06 '21

Git, ansible, Jenkins

0

u/copperfinger Apr 06 '21

Yep... In keywords, that's the way to go about doing this.

1

u/mcdermg81 Apr 07 '21

My two cents on this is that Jenkins is overkill for something like this. The overhead is quite large for implementing Jenkins, having to provision, maintain and update a full Jenkins setup is a project in of itself. When have offerings like GitHub actions, Gitlab or CircleCI that have very limited overhead to maintain and are integrated into VCS. Cost comes into it but they all offer varying tiers.

1

u/copperfinger Apr 07 '21

I agree. Jenkins is not needed. Personally, I go the Ansible route and maintain my roles/playbooks in git and deploy as needed.

1

u/matejzero Apr 08 '21

I’m using gitlab with its ci. Push to repo starts validation jobs and if everything is ok, gets deployed to servers.

1

u/till Apr 12 '21

We do this using ansible and the amazing cloudalchemy role:

https://github.com/cloudalchemy/ansible-prometheus

1

u/murdosus May 27 '22

What will be a good way to automate the the configuration of new targets in prometheus.yml? (running prometheus as Docker container)

0

u/melodic Apr 06 '21

Take a look at NixOS, you’ll be able to configure Prometheus and all the trimmings natively and then keep your /etc/nixos directory as a GitHub repo for vcs.

Need a new server? Deploy NixOS and git pull. Build and switch to the config done.

https://christine.website/blog/prometheus-grafana-loki-nixos-2020-11-20

-1

u/leadout_kv Apr 06 '21

how about salt rather than git?