Grafana Monitoring Stack
Together with alloy, mimir, and loki, we cover gathering metrics and logs as well as creating dashboards with Grafana. We even deploy alert-manager for you to create your own alerts.
You can see an overview of the whole Prometheus Stack Argo CD Application at small-hack/argocd-apps:grafana_stack.
| Application |
Description |
| Alloy |
for collecting metrics and logs |
| Mimir |
for aggregating metrics and storign them in S3 |
| Loki |
for aggregating logs and storing them in S3 |
| Alert Manager |
for sending alerts to matrix |
| Grafana |
for querying metrics/logs and displaying dashboards |
Important note
We haven't generated new screenshots, but we've updated how we now deploy the Prometheus CRDs. They are now deployed separately, so that anything that relies on them that gets deployed earlier on, such as your identity provider, which you may want to secure the prometheus related frontends.
Example configs
Custom Resource Definitions (CRDs)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32 | apps:
prometheus_crds:
description: |
[link=https://prometheus.io/docs/introduction/overview/]Prometheus[/link] CRDs to start with.
You can optionally disable this if you don't want to deploy apps with metrics.
enabled: true
argo:
# secrets keys to make available to Argo CD ApplicationSets
secret_keys: {}
# git repo to install the Argo CD app from
repo: https://github.com/small-hack/argocd-apps
# path in the argo repo to point to. Trailing slash very important!
path: prometheus/crds/
# either the branch or tag to point at in the argo repo above
revision: main
# kubernetes cluster to install the k8s app into, defaults to Argo CD default
cluster: https://kubernetes.default.svc
# namespace to install the k8s app in
namespace: prometheus
# recurse directories in the provided git repo
directory_recursion: false
# source repos for Argo CD App Project (in addition to argo.repo)
project:
name: prometheus
source_repos:
- https://github.com/prometheus-community/helm-charts.git
destination:
# automatically includes the app's namespace and argocd's namespace
namespaces:
- kube-system
- prometheus
|
kube-prometheus-stack
NOTE: You need to also enable the CRD app above for this to work!
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96 | apps:
grafana_stack:
description: |
Full monitoring stack with:
- [link=https://grafana.com.com/oss/mimir]Alloy[/link] - for collecting metrics and logs
- [link=https://grafana.com.com/oss/mimir/]Mimir[/link] - for aggregating metrics and storign them in S3
- [link=https://grafana.com/oss/loki/]Loki[/link] - for aggregating logs and storing them in S3
- [link=https://prometheus.io/docs/alerting/latest/alertmanager/]Alert Manager[/link] - for sending alerts to matrix
- [link=https://grafana.com/oss/grafana/]Grafana[/link] - for querying metrics/logs and displaying dashboards
smol-k8s-lab supports initialization by setting up your ingress hostnames. It will also setup Oauth2 for Grafana directly by creating an app in Zitadel for you.
For Alert Manager, we use vouch-proxy via Ingress resource annotations to forward users to Zitadel for auth, so the frontend is not insecure.
enabled: false
init:
# if init is enabled, we'll set up an app in Zitadel for using Oauth2 with Grafana
enabled: true
restore:
enabled: false
restic_snapshot_ids:
seaweedfs_volume: latest
seaweedfs_filer: latest
backups:
# cronjob syntax schedule to run grafana stack pvc backups
pvc_schedule: 10 0 * * *
s3:
# these are for pushing remote backups of your local s3 storage, for speed and cost optimization
endpoint: ""
bucket: ""
region: ""
secret_access_key:
value_from:
env: GRAFANA_STACK_S3_BACKUP_SECRET_KEY
access_key_id:
value_from:
env: GRAFANA_STACK_S3_BACKUP_ACCESS_ID
restic_repo_password:
value_from:
env: GRAFANA_STACK_RESTIC_REPO_PASSWORD
argo:
# secrets keys to make available to Argo CD ApplicationSets
secret_keys:
## you can delete these if you're not using tolerations/affinity
# toleration_key: ""
# toleration_operator: ""
# toleration_value: ""
# toleration_effect: ""
## these are for node affinity, delete if not in use
# affinity_key: ""
# affinity_value: ""
# FQDN to use for Thanos web interface
thanos_hostname: ""
# FQDN to use for Grafana
grafana_hostname: ""
# FQDN to use for the Loki UI
loki_hostname: ""
# FQDN to use for Alert Manager web interface
alert_manager_hostname: ""
# set the local s3 provider for the loki/mimir S3 backups. can be minio or seaweedfs
s3_provider: seaweedfs
# local s3 endpoint for loki/mimir S3 backups, backed up constantly
s3_endpoint: ""
# capacity for the PVC backing your local s3 instance
s3_pvc_capacity: 100Gi
# git repo to install the Argo CD app from
repo: https://github.com/small-hack/argocd-apps
# path in the argo repo to point to. Trailing slash very important! This
# is an app of apps. Change to "monitoring/kube-prometheus-stack/" to
# only install kube-prometheus-stack (foregoing loki and push gateway)
path: grafana_stack/app_of_apps/
# either the branch or tag to point at in the argo repo above
revision: main
# kubernetes cluster to install the k8s app into, defaults to Argo CD default
cluster: https://kubernetes.default.svc
# namespace to install the k8s app in
namespace: "monitoring"
# recurse directories in the provided git repo
directory_recursion: false
# source repos for Argo CD App Project (in addition to argo.repo)
project:
name: monitoring
source_repos:
- registry-1.docker.io
- https://grafana.github.io/helm-charts
- ghcr.io/grafana/helm-charts
- https://github.com/prometheus-community/helm-charts.git
- https://prometheus-community.github.io/helm-charts
- https://seaweedfs.github.io/seaweedfs/helm
destination:
# automatically includes the app's namespace and argocd's namespace
namespaces:
- kube-system
|