Elliott's Second Brain

Kobi

Properties1
tagskobi, watchdog, firebase, monitoring, agent

2 min read

Kobi — Functions Watchdog

Role

Named watchdog agent for the SMK9 Firebase Cloud Functions error pipeline. Kobi watches the api Cloud Run service in real time and barks when something bites — 4xx/5xx HTTP errors, ERROR-severity log entries, or any pattern that signals the live API is broken.

Named by Elliott Soto after seeing repeated watchdog alerts fire from the system and deciding this vigilance deserved a name. Kobi is also Elliott’s dog — the dog behind the original SMK9 logo — making him the right guardian for the business’s live infrastructure.

Responsibilities

  • Query Google Cloud Logging for the api Cloud Run service every 15 minutes
  • Alert via Telegram when new HTTP 4xx/5xx or ERROR-severity entries appear since the last check
  • Stay silent when the API is clean (zero-cost watchdog pattern)
  • Track a state file so the same errors never fire twice

Why Kobi exists

Before Kobi (pre-2026-07-07), errors in Firebase Cloud Functions were invisible unless Elliott manually ran firebase functions:log — and that command doesn’t even work for 2nd-gen Cloud Run functions (returns empty bodies). Errors could sit undetected for hours. Kobi was built to close that gap and was named at the moment Elliott realized the watchdog alerts were doing real work.

Implementation

  • Script: /home/eco/.hermes/scripts/functions_error_monitor.py
  • Cron job ID: f78ddcec5543
  • Schedule: Every 15 minutes, 24/7
  • State file: /home/eco/.hermes/data/functions_error_monitor_state.json
  • Delivery: Telegram → Elliott (silent if clean)
  • Auth: Firebase Admin SDK service account (sound-mind-k9-business-build-firebase-adminsdk)

Alert pattern

🚨 Functions error monitor — N error/4xx/5xx log entries on 'api' since [ts]:
[2026-07-07T20:12:25Z] HTTP 401 — /admin/payment-match
[2026-07-07T20:12:26Z] HTTP 401 — /bookings/pending-holds
...

Known alert history

  • 2026-07-07 ~20:12 UTC: 50× HTTP 401 on staff endpoints — stale adminKey in smk9-staff.html hitting endpoints whose Firebase Secret had rotated. Root cause: admin key gates removed from all staff-internal endpoints same session. Pre-deploy alerts, not post-deploy.

hermes | functions-error-monitor | smk9-intake-form | spider-man

Graph View