--- title: Setting Up Prometheus Monitoring in Kubernetes description: A comprehensive guide to implementing Prometheus monitoring in your Kubernetes cluster pubDate: 2025-04-19 heroImage: /blog/images/posts/prometheusk8.png category: devops tags: - kubernetes - monitoring - prometheus - grafana - observability readTime: 9 min read --- # Setting Up Prometheus Monitoring in Kubernetes Effective monitoring is crucial for maintaining a healthy Kubernetes environment. Prometheus has become the de facto standard for metrics collection and alerting in cloud-native environments. This guide will walk you through setting up a complete Prometheus monitoring stack in your Kubernetes cluster. ## Why Prometheus? Prometheus offers several advantages for Kubernetes monitoring: - **Pull-based architecture**: Simplifies configuration and security - **Powerful query language (PromQL)**: For flexible data analysis - **Service discovery**: Automatically finds targets in dynamic environments - **Rich ecosystem**: Wide range of exporters and integrations - **CNCF graduated project**: Strong community and vendor support ## Components of the Monitoring Stack We'll set up a complete monitoring stack consisting of: 1. **Prometheus**: Core metrics collection and storage 2. **Alertmanager**: Handles alerts and notifications 3. **Grafana**: Visualization and dashboards 4. **Node Exporter**: Collects host-level metrics 5. **kube-state-metrics**: Collects Kubernetes state metrics 6. **Prometheus Operator**: Simplifies Prometheus management in Kubernetes ## Prerequisites - A running Kubernetes cluster (K3s, EKS, GKE, etc.) - kubectl configured to access your cluster - Helm 3 installed ## Installation Using Helm The easiest way to deploy Prometheus is using the kube-prometheus-stack Helm chart, which includes all the components mentioned above. ### 1. Add the Prometheus Community Helm Repository ```bash helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo update ``` ### 2. Create a Namespace for Monitoring ```bash kubectl create namespace monitoring ``` ### 3. Configure Values Create a `values.yaml` file with your custom configuration: ```yaml prometheus: prometheusSpec: retention: 15d resources: requests: memory: 256Mi cpu: 100m limits: memory: 2Gi cpu: 500m storageSpec: volumeClaimTemplate: spec: storageClassName: standard accessModes: ["ReadWriteOnce"] resources: requests: storage: 20Gi alertmanager: alertmanagerSpec: storage: volumeClaimTemplate: spec: storageClassName: standard accessModes: ["ReadWriteOnce"] resources: requests: storage: 10Gi grafana: persistence: enabled: true storageClassName: standard size: 10Gi adminPassword: "prom-operator" # Change this! nodeExporter: enabled: true kubeStateMetrics: enabled: true ``` ### 4. Install the Helm Chart ```bash helm install prometheus prometheus-community/kube-prometheus-stack \ --namespace monitoring \ --values values.yaml ``` ### 5. Verify the Installation Check that all the pods are running: ```bash kubectl get pods -n monitoring ``` ## Accessing the UIs By default, the components don't have external access. You can use port-forwarding to access them: ### Prometheus UI ```bash kubectl port-forward -n monitoring svc/prometheus-operated 9090:9090 ``` Then access Prometheus at http://localhost:9090 ### Grafana ```bash kubectl port-forward -n monitoring svc/prometheus-grafana 3000:80 ``` Then access Grafana at http://localhost:3000 (default credentials: admin/prom-operator) ### Alertmanager ```bash kubectl port-forward -n monitoring svc/prometheus-alertmanager 9093:9093 ``` Then access Alertmanager at http://localhost:9093 ## For Production: Exposing Services For production environments, you'll want to set up proper ingress. Here's an example using a basic Ingress resource: ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: prometheus-ingress namespace: monitoring annotations: nginx.ingress.kubernetes.io/ssl-redirect: "true" spec: rules: - host: prometheus.example.com http: paths: - path: / pathType: Prefix backend: service: name: prometheus-operated port: number: 9090 - host: grafana.example.com http: paths: - path: / pathType: Prefix backend: service: name: prometheus-grafana port: number: 80 - host: alertmanager.example.com http: paths: - path: / pathType: Prefix backend: service: name: prometheus-alertmanager port: number: 9093 ``` ## Configuring Alerting ### 1. Set Up Alert Rules Alert rules can be created using the PrometheusRule custom resource: ```yaml apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: node-alerts namespace: monitoring labels: release: prometheus spec: groups: - name: node.rules rules: - alert: HighNodeCPU expr: instance:node_cpu_utilisation:rate1m > 0.8 for: 5m labels: severity: warning annotations: summary: "High CPU usage on {{ $labels.instance }}" description: "CPU usage is above 80% for 5 minutes on node {{ $labels.instance }}" ``` ### 2. Configure Alert Receivers Configure Alertmanager to send notifications by creating a Secret with your configuration: ```yaml apiVersion: v1 kind: Secret metadata: name: alertmanager-prometheus-alertmanager namespace: monitoring stringData: alertmanager.yaml: | global: resolve_timeout: 5m slack_api_url: 'https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK' route: group_by: ['job', 'alertname', 'namespace'] group_wait: 30s group_interval: 5m repeat_interval: 12h receiver: 'slack-notifications' routes: - receiver: 'slack-notifications' matchers: - severity =~ "warning|critical" receivers: - name: 'slack-notifications' slack_configs: - channel: '#alerts' send_resolved: true title: '{{ template "slack.default.title" . }}' text: '{{ template "slack.default.text" . }}' type: Opaque ``` ## Custom Dashboards Grafana comes pre-configured with several useful dashboards, but you can import more from [Grafana.com](https://grafana.com/grafana/dashboards/). Some recommended dashboard IDs to import: - 1860: Node Exporter Full - 12740: Kubernetes Monitoring - 13332: Prometheus Stats ## Troubleshooting ### Common Issues 1. **Insufficient Resources**: Prometheus can be resource-intensive. Adjust resource limits if pods are being OOMKilled. 2. **Storage Issues**: Ensure your storage class supports the access modes you've configured. 3. **ServiceMonitor not working**: Check that the label selectors match your services. ## Conclusion You now have a fully functional Prometheus monitoring stack for your Kubernetes cluster. This setup provides comprehensive metrics collection, visualization, and alerting capabilities essential for maintaining a healthy and performant cluster. In future articles, we'll explore advanced topics like custom exporters, recording rules for performance, and integrating with other observability tools like Loki for logs and Tempo for traces.