Provisioning Dashboards with Grafana
Background Grafana has become the de-facto visualization tool for Prometheus. While it is cool to run a central Grafana hooked up to an RDS database, I think it is even better if you can make Grafana completely configurable via git and thus have stateless Grafana instances which you can scale horizontally. Based on this philosophy, I have been running a Grafana setup at Red Hat, here’s some key points: Grafana runs as pods on a Kubernetes (OpenShift) cluster Each dashboard is mounted into the pod via ConfigMap Our GitOps pipeline takes care of adding the dashboard configmaps into the namespace, so all dashboards and their changes ultimately must end up in Git One of the best benefits of this approach is that you never have to worry about Grafana upgrades/downgrades. Because the pods are stateless, you can simply roll out a new version as long as the dashboard schema stays consistent. ...