Monitor Kubernetes Services & Pods with Prometheus and Send Alerts to Slack

🧩 Short Summary

Proactively alert to service endpoint changes and pod/container issues (Pending, Not Ready, Restart spikes) using Prometheus metrics, formatted and sent to Slack.

🗂️ Category

DevOps / Monitoring & Observability

🏷️ Tags

kubernetes, prometheus, slack, alerting, sre, ops, kube-state-metrics

✅ Prerequisites

Prometheus scraping kube-state-metrics v2.x. Slack App or Incoming Webhook (channel access). n8n instance with outbound access to Prometheus & Slack.

🔑 Required Credentials in n8n

Slack: Bot OAuth (chat:write) or Incoming Webhook URL.

(Optional) Prometheus Basic Auth (if your Prometheus needs it).

🧠 What This Template Does

Detects pods stuck in Pending (scheduling problems like taints/affinity/capacity). Detects containers Not Ready (readiness probe failures). Detects container restart spikes over a sliding window (default 5 minutes). Detects service discovery changes (endpoint count diffs current vs previous snapshot). Sends clean, emoji-enhanced Slack alerts with pod/namespace/service context. Outputs a 5-minute summary block to reduce noise.

📣 Slack Message Style (examples)