AUTOMATE GRAFANA DASHBOARD IMPORT PROCESS

In a recent project I did for one of my customers, we wanted to gain insight on our application. We are running a Kubernetes cluster, so we decided to deploy Prometheus & Grafana as our monitoring solution. Once deployed, we wanted to automate the process of importing various Grafana dashboards into the application.

While you have the option to manually import a dashboard once Grafana is up and running, we wanted to automate the process so we can keep Grafana as stateless as possible and ease the administration overhead when we upgrade or deploy it across different environments.

Before we dive into the “how-to”, I want to take a minute to describe the environment we’re going to work on, since it’s crucial to our automation process.

Environment Overview

Grafana is an “open platform for beautiful analytics and monitoring”. It will visualize your collected metrics and present them in a graphical way (line charts, histograms, gauges, tables, etc.) As an open source application, you can deploy it in many ways (you can read about some of the options here)

We are working in a Kubernetes (k8s) environment, then we proceeded to choose a monitoring solution. We decided to deploy Prometheus Operator and Kube-Prometheus using helm. Since each of these 3 can be developed into a blog post (or series of…) on its own, I won’t go into too much details about them.

I will, however, point out that:

  • Helm is package manager for k8s allowing you to deploy applications to a k8s cluster (think of yum or apt for linux)
  • Prometheus Operator allows you to deploy Prometheus on top of k8s, “in a kubernetes-native” way
  • Kube-Prometheus is a deployment of Prometheus that is using the Prometheus Operator to provide a running Prometheus instance together with various exporters and Grafana

Our Architecture will look like this:

1

What do you need in order to play along?

If you want to see it all in action, here’s what you need:

  • A running k8s cluster with version 1.9 and up with helm installed
  • Prometheus Operator & Kube-Prometheus deployed in your cluster (If this is your first time doing this, I highly suggest you read the documentation on github)

Run the below commands against your k8s cluster to install Prometheus operator on it:

kubectl create ns monitoring
git clone https://github.com/coreos/prometheus-operator.git
cd prometheus-operator/helm/kube-prometheus
helm repo add coreos https://s3-eu-west-1.amazonaws.com/coreos-charts/stable/
helm dependency build
cd charts
tar -zxvf grafana-0.0.37.tgz
rm grafana-0.0.37.tgz
cd ../../..
helm install --name prometheus-operator --namespace monitoring helm/prometheus-operator/
helm install --name kube-prometheus --namespace monitoring helm/kube-prometheus/

Problem Statement

So, if we’re using Kube-Prometheus to deploy Grafana, and we want to automatically import a custom dashboard to it, how do we go about it?

  1. Prepare your dashboard for importing (or “template” it)
  2. Modify the Grafana deployment to include the modified dashboard
  3. Upgrade your helm release and watch the magic happen

Prepare your Dashboard for import

Start by creating your dashboard. It can be a fresh one created from scratch or one you took from the Grafana Dashboards page. For the sake of learning, we will use Kubernetes cluster monitoring (via Prometheus) to play with.

Download the dashboard .json file and edit it with your text editor, turning it from this:

To this:

Note how we wrapped the start and end of our original json to make it a template

This process is rather easy if you’re using an existing template. However, if you’re exporting a live dashboard from Grafana, there are a few “gotcha’s” that you should pay attention to:

  • If the original datasource isn’t called Prometheus in Grafana (let’s say, it’s called dev-Prometheus) you need to find all the references for it in the .json and modify them. For example:

Note that you will likely have multiple references to the datasource  parameter in your dashboard.json file. You should modify all of them.

Modify the deployment to include the modified dashboard

Now that we have our dashboard ready, we need to add it to the Grafana helm chart inside kube-prometheus for Grafana to pick it up.

Reading through the Grafana helm chart documentation, this is what we’re looking for:

Adding Grafana Dashboards

You can either add new dashboards via serverDashboardConfigmaps in values.yaml. These can then be picked up by Grafana Watcher.

serverDashboardConfigmaps:
  - example-dashboards
apiVersion: v1
kind: ConfigMap
metadata:
  name: example-dashboards
data:
{{ (.Files.Glob "custom-dashboards/*.json").AsConfig | indent 2 }}

# Or
#
# data:
#   custom-dashboard.json: |-
# {{ (.Files.Get "custom.json") | indent 4 }}
#
# The filename (and consequently the key under data) must be in the format `xxx-dashboard.json` or `xxx-datasource.json`
# for them to be picked up.

We will add a folder called custom-dashboards into the Grafana chart folder (located inside kube-prometheus/charts/grafanaand copy our .json there.

Note the comment at the bottom and name your file accordingly:

The filename (and consequently the key under data) must be in the format `xxx-dashboard.json` or `xxx-datasource.json` for them to be picked up.

We will add a configMap and call it “custom-dashboards”, directing it to load any json file in our “custom-dashboards” directory inside the Grafana chart:

Lastly, we will modify the main values.yaml located inside the Grafana chart directory, directing Grafana to load a configMap called custom-dashboards:

And we will also direct the kube-prometheus chart to pick up our new, modified Grafana chart. Open up the requirements.yaml file (inside the kube-prometheus folder) and modify it to point to our modified chart:

Let’s recap what we did:

  1. Created a configMap that will load any json files inside a folder called custom-dashboards and named the configMap custom-dashboards
  2. Directed Grafana to load new configMap named custom-dashboards into it’s configuration
  3. Directed kube-prometheus to load our modified Grafana chart instead of the default one

Tip!

By doing it this way, we can add future dashboards simply by adding them to the custom-dashboards directory and refresh the configMap in our deployment

Upgrade your helm release and watch the magic happen

Lastly, let’s update our kube-prometheus deployment with the new Grafana configMap and see if everything works

helm upgrade kube-prometheus helm/kube-prometheus/

Lastly, let’s open Grafana and see our new custom dashboard in action!

$ kubectl get pod --namespace=monitoring
NAME READY STATUS RESTARTS AGE
alertmanager-kube-prometheus-0 2/2 Running 2 14d
kube-prometheus-exporter-kube-state-76f498d465-9rxf5 2/2 Running 2 14d
kube-prometheus-exporter-node-xzp2z 1/1 Running 1 14d
kube-prometheus-grafana-5f8c9cdb94-qf8jh 2/2 Running 0 6m
prometheus-kube-prometheus-0 3/3 Running 3 14d
prometheus-operator-d75587d6-g4rwc 1/1 Running 1 14d

$ kubectl port-forward kube-prometheus-grafana-5f8c9cdb94-qf8jh 5000:3000
Forwarding from 127.0.0.1:5000 -> 3000
Forwarding from [::1]:5000 -> 3000

Open up your browser and point it to http://localhost:5000, then choose our imported dashboard Kubernetes cluster monitoring (via Prometheus) under the home menu:

2

Summary

Prometheus is an amazing monitoring system, and Grafana let’s us visualize our data in a beautiful way

When deploying the monitoring solution with the suggested approach (the “operator” way), we get a full-fledged monitoring solution, and we get a way to customize it to our needs in a kubernetes native way.

Meaning, when we need to modify a component’s configuration (Grafana in our case), we do it by modifying kubernetes components (helm charts, configMaps) instead of modifying software configuration files.

This way, we can maintain our own version of the software (our helm chart) and re-use it with our needed modifications in any environment.


 

So what do you think about Grafana dashboard import process? Do you want to know more? ping me @omerbarel

Oh, and one last thing – stay tuned for my future blog posts as I dive deeper into helm scopes, charts, sub-charts and global variables.

Ciao,

Omer Barel,

DevOps Consultant, CodeValue

Advertisements

One thought on “AUTOMATE GRAFANA DASHBOARD IMPORT PROCESS

Add yours

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Create a free website or blog at WordPress.com.

Up ↑

%d bloggers like this: