Monitor AI agent runs with Gemini judge, Google Sheets, Slack and Gmail
Monitor AI agent runs with Gemini judge, Google Sheets, Slack and Gmail
Quick overview
This live watchdog monitors AI agent executions daily, sampling runs for Gemini judge scoring, tracking quality drift, and detecting cost, latency, and silent-failure anomalies. It logs scores and baselines to Google Sheets and alerts operations via Slack, Telegram, and Gmail.
How it works
Runs on a daily schedule (or manually) and loads configuration values such as the monitored n8n URL, API key, workflow IDs, sampling settings, and alert thresholds.
Validates the configuration and builds an n8n executions API request for the lookback window, then fetches recent executions from the monitored n8n instance.
If the executions fetch fails, sends a Gmail alert to the ops email address and stops the run gracefully.
Normalizes execution records, estimates cost from token counts, deduplicates against previously scored execution IDs stored in Google Sheets, and flags failed runs and “silent failures” (for example, empty or error-like output despite a success status).
Always selects failures and silent failures for review and deterministically samples additional runs, then uses Google Gemini to judge the sampled outputs and produce a 0–100 score with a short reason.
Compares current window metrics (mean score, cost, mean/p95 latency, silent-failure rate) against pooled historical baselines from Google Sheets to detect drift and threshold breaches, with an alert cooldown based on recent alert history.
Appends per-run scores and a baseline snapshot to Google Sheets, posts alerts to Slack (and Telegram if enabled), logs alert decisions to Google Sheets, and sends a daily digest email via Gmail.
Setup
Create an n8n API key on the monitored instance and set the correct n8nBaseUrl, n8nApiKey, lookbackHours, and workflow ID filters (or clear targetWorkflowIds to watch all) in the configuration.
Create a Google Sheet with tabs named RunScores, Baselines, and AlertLog, add a Google Sheets credential in n8n, and set sheetId in the configuration.
Add credentials for Gmail and Slack, then set opsEmail and slackChannelId (and enableTelegram plus telegramChatId and Telegram credentials if you want Telegram alerts).
Add a Google Gemini API key, set the geminiModel if needed, and tune sampling/threshold settings (sampleRate, maxJudgePerRun, driftStdDev, score/cost/latency floors, alertCooldownHours, and token prices) before activating the workflow.