Setting up Prometheus Stack
The default Prometheus/Grafana setup will give you everything including the kitchen sink. At this point in time it’s worth keeping all of that around for the sake of learning, but some small modifications are required to make it a bit more secure.
Requirements
A successful deployment depends on just a few tasks.
- Adding the Prometheus Stack repo
- Defining the values for the Helm chart
- Installing the Helm chart with the configuration
- Exposing Grafana to the internet
Let’s take this step by step.
Add the Helm chart repository
This command won’t appear to do much on its own, but it keeps track of charts you want to use. Like adding a dependency, say.
helm repo add prometheus-community https://prometheus-community.github.io/helm-chartsThis will make a few prometheus-related charts available but when it comes to installing Prometheus, only a specific chart will be used.
Configure the helm chart
A helm chart is configured using a single values file, and whatever you define there is passed into the chart before installing it. There is no consistent structure for these files, it really depends on the author of the chart.
The structure for Prometheus Stack is… well… see for yourself 😅. We’re not going to go through roughly 2.5k lines worth of config (no matter how well-documented it is), most of which consists of sensible defaults.
Instead, there is just one thing that is worth tweaking. Save this config into a file:
grafana: admin: existingSecret: grafana-admin userKey: admin-user passwordKey: admin-passwordWhat is it doing? The default Grafana setup hard-codes a generic admin account and password that is common to all default installations. That is no bueno, especially if you decide to expose Grafana to the internet, so this merely replaces the default admin account with one of your own.
Create your login credentials
Now you need to create a secret and that leaves us at an impasse. It’s also not secure to store passwords in plaintext inside your config (which might be a git repo too).
It’s better to use the kubectl CLI for this, which is not perfect but will do the job for now.
kubectl create secret generic grafana-admin \ --from-literal=admin-user=$YOUR_USERNAME \ --from-literal=admin-password=$YOUR_PASSWORD \ --namespace prometheus-stackMake sure to swap $YOUR_USERNAME and $YOUR_PASSWORD with values of your own, of course. Once you run it, it will have stored your new login credentials as a secret in the same namespace that Grafana runs in.
Install the chart
You now have enough to set up a full-blown monitoring solution! Time to make it happen by running this one simple command:
helm install prometheus-stack prometheus-community/kube-prometheus-stack \ --namespace prometheus-stack --create-namespace \ --file helm/prometheus-stack.ymlThis is likely to take a moment to run, which creates a nice opportunity to explain what you just did.
Breaking it down
That’s a lot of repetition in the command! Let’s break it down:
helm install prometheus-stack prometheus-community/kube-prometheus-stackThis installs the chart kube-prometheus-stack from the prometheus-community repo and names the corresponding release prometheus-stack. A release, in Helm terms, is a successful deployment of a chart. Each update creates a new release so you can roll back to previous ones if necessary.
helm install ... --namespace prometheus-stack --create-namespaceIt’s useful to isolate your deployments to their own namespace rather than dumping them all in the default one, and you might as well give it the same name as the release.
helm install ... ... --file helm/prometheus-stack.ymlWhile it’s not necessary to provide your configuration in YAML format (you can do it all via the CLI if you’re desperate), this is used to configure the values the given chart provides.
You can put this file anywhere; personally I keep them in one folder to make it easier to see what I’m using Helm for.
Reveal Grafana to the internet
Nothing you’ve deployed is publicly accessible so far, so assuming you’ve successfully worked through 🌐 Internet Exposure, you should have enough in place to change that.
By default, a service isn’t available to the internet at large unless it has an ingress to make it visible. If you’re familiar with Apache HTTPD or Nginx then an ingress is something like a virtual host with a reverse proxy.
Create the ingress config
The Helm chart already did all of the heavy lifting, so all that is required to make the service available is to create an Ingress manifest.
apiVersion: networking.k8s.io/v1kind: Ingressmetadata: name: grafana namespace: prometheus-stack annotations: kubernetes.io/tls-acme: "true" cert-manager.io/cluster-issuer: letsencrypt-prodspec: ingressClassName: nginx rules: - host: grafana.your-domain.com11 collapsed lines
http: paths: - path: / pathType: Prefix backend: service: name: prometheus-stack-grafana port: number: 80 tls: - hosts: - grafana.your-domain.com secretName: grafana.your-domain.com-tlsApply the ingress config
It’s helpful to save this in a different directory to the Helm chart configuration, because kubectl will complain if you try to apply all of the config in a directory and one of the files is invalid.
Speaking of applying configuration, do that, changing the path to the file if you need to:
kubectl apply -f prometheus-stack/grafana-ingress.ymlAfter a moment, and provided you completed the steps to set up external-dns and cert-manager, you should be able to access Grafana at https://grafana.your-domain.com. If you can’t see it yet, don’t worry! DNS changes can take a minute to propagate.