# 監控面板紅了四天，壞的從來不是服務

- URL: https://justfly.idv.tw/%e7%9b%a3%e6%8e%a7%e9%9d%a2%e6%9d%bf%e7%b4%85%e4%ba%86%e5%9b%9b%e5%a4%a9%ef%bc%8c%e5%a3%9e%e7%9a%84%e5%be%9e%e4%be%86%e4%b8%8d%e6%98%af%e6%9c%8d%e5%8b%99/
- 日期: 2026-05-21
- 分類: WEB&RIA
- 標籤: Linux, server, 神島

![監控面板紅了四天，壞的從來不是服務]

##### 一萬一千次失敗的告警
健康探針連續回報失敗，FailingStreak 數千次，監控面板整整四天一片紅。直覺反應是服務掛了，但那個服務從頭到尾都在正常處理請求——cron 跑得動，hook 觸發得了，對外 API 沒有任何異常回應。

##### 技術環境

容器化服務環境，以 Docker HEALTHCHECK 指令配置健康探針，定期以 `curl -sf` 打特定管理端點；exit code 0 為健康、非零為不健康。底層為 Node.js 應用，管理端點的存取規則由框架版本控制，服務本身不感知探針的存在，兩者的關聯僅存在於監控系統的解讀層。問題模式與框架無關——任何「探針目標端點」與「框架版本存取規則」不同步更新的部署流程，都會複現相同行為。

這就像 Gogoro app 上的橘色警示連亮四天，但每次換電池、上路都跑得順。最後發現是韌體升級後感測器回傳格式跑掉了，電池本身沒事。巷口的車行師傅接過手機看一眼，抬頭說「沒問題啊，好好的」——問題出在讀取的那一層，不在被讀取的東西。

##### 框架升級後的靜默改動
根因不難找，但要找到需要把直覺壓下去。框架從 4.x 升到 5.x 之後，某個管理端點悄悄加上了認證要求——沒有帶 token，一律回 401。健康探針用的是早在 4.x 時期寫死的 curl 指令，指向那個現在需要認證的路徑。curl 遇到 4xx 回傳非零 exit code，監控系統把這個訊號解讀成「服務不健康」。
Health.Log 每筆都是 ExitCode: 1，Output 是空的。空的 Output 是關鍵：curl -sf 對 4xx 不輸出任何內容，只靜默地退出。監控系統看到的是「失敗了」，卻沒有任何可讀的錯誤訊息告訴你失敗在哪一層。

##### 錯誤傳染鏈（時序）

```
監控系統              curl 探針              應用服務
    |                      |                    |
    |── 觸發健康探測 ──────>|                    |
    |                      |── GET /mgmt/health─>|
    |                      |                    |── auth check
    |                      |                    |   no token ← 框架 5.x 新增規則
    |                      |&1 | grep "HTTP/"
```

##### Side Effects That Should Be Isolated

- **Probe target endpoint versioning**: Explicitly audit whether the probe’s target path still satisfies the new version’s access rules on every major/minor upgrade — don’t rely on release notes being exhaustive.

- **Global auth middleware scope**: When adding a global auth middleware or tightening access rules, audit whether the change reaches healthcheck, readiness, and liveness probe paths.

- **Alert threshold trigger logic**: When FailingStreak exceeds a threshold, trigger a direct service verification step alongside the alert — not just a probe status read — to catch probe-vs-service divergence before escalating.

- **Post-rebuild probe verification**: After a container rebuild, run an automatic probe path reachability check to confirm the new version’s access rules align with the HEALTHCHECK instruction.

- **Silent failures without log output**: `curl -sf` produces no output on 4xx/5xx. Add `--write-out "%{http_code}"` to critical probe paths to generate a readable diagnostic signal on failure.

- **Multi-environment config drift**: Inconsistent HEALTHCHECK configurations across dev/staging/prod mean issues only surface in production at a specific version boundary, compounding debugging cost.

- **CI/CD deployment smoke tests**: Post-deploy smoke tests should include a live HTTP check of the healthcheck endpoint path — not just container start state — to validate probe reachability after every deploy.

- **Downstream health propagation**: If the /health endpoint depends on downstream services (DB, cache), separate “liveness probes” from “dependency health probes” to prevent downstream jitter from misclassifying upstream health.

Rule of thumb: if a configuration can silently diverge across a version boundary such that probe state and service state no longer agree, it belongs on an explicit version-upgrade audit checklist.

— 邱柏宇

##### 延伸閱讀

- [節點重建後，服務端連 decode 都懶得做](https://justfly.idv.tw/%e7%af%80%e9%bb%9e%e9%87%8d%e5%bb%ba%e5%be%8c%ef%bc%8c%e6%9c%8d%e5%8b%99%e7%ab%af%e9%80%a3-decode-%e9%83%bd%e6%87%b6%e5%be%97%e5%81%9a/)

- [一杯手搖飲裡塞了三個台灣的未來](https://justfly.idv.tw/bubble-tea-three-futures-taiwan/)

- [比特幣儲能雙箭齊發 搭上美股輪動順風車](https://justfly.idv.tw/2026-05-18-daily_stock_recommendations-2/)

### Four Days of Red Alerts, Zero Service Failures

##### Eleven Thousand Failures, One Functioning Service
The health probe kept firing failures. FailingStreak climbed into the thousands. The monitoring dashboard stayed red for four straight days. The obvious read: something is broken. The actual situation: the service was handling every request normally — cron jobs ran, hooks fired, external API responses were clean throughout.
Think of a Gogoro app flashing an orange battery warning for four days while the bike runs perfectly every ride. A firmware update had scrambled the sensor’s return format; the battery itself was fine. The mechanic at the corner shop glances at your phone and says, without looking up: “Nothing wrong with it.” The fault is in the reading layer, not what’s being read.

##### Technical Environment

Containerized service deployed via Docker, with a HEALTHCHECK instruction running `curl -sf` against a specific management endpoint on a fixed schedule; exit code 0 signals healthy, any non-zero exit signals unhealthy. The underlying application is Node.js; endpoint access rules are controlled by the framework version. The service has no awareness of the probe — their connection exists only in the monitoring system’s interpretation layer. The failure pattern is framework-agnostic: any deployment workflow where the probe’s target endpoint and the framework’s access rules drift out of sync will reproduce the same behavior.

##### The Silent Change Inside a Version Bump
The root cause wasn’t complicated, but finding it required suppressing the wrong instinct first. When the framework moved from 4.x to 5.x, a management endpoint quietly gained an authentication requirement — no token, always a 401. The healthcheck command had been written back in the 4.x era, hardcoded to hit that exact path. curl encountering a 4xx exits with a non-zero code. The monitoring system reads that signal as “service unhealthy.”
Every entry in Health.Log showed ExitCode: 1, Output: empty. That empty output is the tell. curl -sf produces no output on 4xx — it just exits silently. The monitoring system registered “failed” with nothing to indicate which layer had failed.

##### Error Propagation Sequence

```
Monitoring System      curl Probe          Application Server
      |                     |                      |
      |── trigger probe ───>|                      |
      |                     |── GET /mgmt/health ──>|
      |                     |                      |── auth check
      |                     |                      |   no token ← added in 5.x
      |                     |