靜默失敗不是沒事：pm2 online 也可能是假活 | 監控誤區解析JustFLY~JustBlog~

冰箱壓縮機還在轉、燈也會亮，但溫控壞了食物早就臭掉——光盯指示燈沒用，得開門聞味道。同一週內連查四起事故，全是這種味道。

技術環境

四起事故各自的棧不完全相同，但有一個共同特徵：監控層與業務層之間存在落差。容器管理工具（pm2 / Docker）量的是 process 存活與 port listening，nginx 量的是 reverse proxy 連通，cron 量的是排程執行本身，API 量的是 HTTP status code。每層各回各的綠燈，沒有一層會主動驗證「業務結果是否正確」。

觸發問題的執行模式分三類：背景常駐 process（pm2 / systemd）、定時排程（cron / Celery beat）、API request handler。三類的共同點是「主流程回傳成功訊號後，依賴下游側效應完成業務結果」，而側效應的失敗被吞掉或被視為非關鍵。

資料層涵蓋 MySQL 8、Redis、OAuth provider。三者各自有型別不相容、key 漂移、token 過期的失敗模式，與框架無關——任何「主操作成功 → 同步側效應」的設計都會複現相同行為。

四種綠燈底下的死法

第一起：容器管理工具顯示 online，nginx 全程吐 502，持續超過 22 小時。根因是版本升級後後端改聽新 port，但 nginx 反代與環境變數仍指向舊 port。容器本身健康，業務 API 全掛。管理介面可以正常訪問，這反而讓人更晚起疑。

第二起：OAuth 認證過期，定時任務每分鐘心跳正常，連續 13 天零產出。每次執行都在日誌裡留下「未認證跳過」，但沒有任何告警。從外部看是一個活著的 process，從業務角度看是 13 天的空白。

第三起：內容生產管線執行成功，但因為 Redis key 型別錯誤觸發了降級邏輯，靜默吞掉錯誤，用 fallback 內容填充輸出。execution status 全綠，產出存在，只是不對。沒有特別去比對內容差異的話，根本不會察覺。

第四起：交易列表 API 因參數型別問題回傳 500，App 端靜默吞錯，顯示空列表。使用者看到空畫面，可能以為「還沒有交易記錄」。這個狀態潛伏了將近一個月，直到有人主動投訴才開始排查。

錯誤傳染鏈（時序）

Scenario 1 — port drift（22h outage）
  Docker           nginx            backend (port 8081 → 8082)
    |                 |                      |
    | health ✓        |                      |
    |                 |── proxy :8081 ───────>|  ← 舊 env, 連線失敗
    |                 |<── 502 ───────────────|
    |                 |                      |
  monitoring: container online ✓ / API: 502 ✗ / business: dead

Scenario 2 — OAuth expired（13d silent）
  cron              service          OAuth provider
    |                   |                     |
    |── tick ──────────>|                     |
    |                   |── refresh token ──>|
    |                   |<── 401 ─────────────|
    |                   |  log: skip          |  ← 吞掉,不告警
    |                   |                     |
  monitoring: heartbeat ✓ / output: 0 / business: silent 13d

Scenario 3 — Redis type drift（silent degradation）
  pipeline          handler          Redis
    |                   |                  |
    |── run ───────────>|                  |
    |                   |── GET key ───────>|
    |                   |<── WRONGTYPE ─────|
    |                   |  fallback: dummy  |  ← 靜默降級
    |<── success ──────|                   |
    |                   |                  |
  monitoring: status ✓ / output: exists / business: wrong content

Scenario 4 — MySQL param type（30d hidden）
  App               API              MySQL 8
    |                 |                   |
    |── GET /txns ───>|                   |
    |                 |── SELECT … WHERE id=? →|
    |                 |<── 500 ────────────|
    |  show empty list|                   |  ← 吞錯, 不重試
    |                 |                   |
  monitoring: app ok / API: 500 ✗ / business: users see nothing

四條鏈的共同節點：監控層在「主流程回 success」那一步就點綠燈，下游的失敗如果發生在同步側效應、且未被顯式升級為告警，就會沿著鏈條一路靜默傳到終端使用者。終端層（App / 業務方）收到的要不是空資料要不是錯誤內容，而不是「上游有東西壞了」的訊號。

分界點在哪裡

這四起事故沒有共同的技術根因——port 錯位、OAuth refresh token 競態、Redis key 型別漂移、MySQL 參數型別不相容，各自獨立。共同點很清楚：監控指標正常，業務結果異常。

傳統 uptime 監控的假設是「process alive ≈ 服務可用」，這個假設在這四個案例裡全部失效。process alive ≠ 服務可用。execution success ≠ 有產出。heartbeat ≠ 有認證。status 200 ≠ 資料正確。

部分功能正常製造的假象最危險。綠色 dashboard 一出現就停止追查，等到業務方主動投訴，往往已經是幾天甚至幾週之後。

Code 對照：修法前後

Scenario 2 修法前：cron handler 對 refresh token 失敗一律 catch + log + skip。

// before
async function tick() {
  try {
    await refreshToken();
    await doWork();
  } catch (e) {
    logger.warn("unauthenticated, skipping");  // ← 問題在這裡
  }
}
// monitoring: heartbeat 正常, output 為零, 無告警

Scenario 2 修法後：認證失敗視同當機，emit metric + 觸發告警通道。

// after
async function tick() {
  try {
    await refreshToken();
    await doWork();
  } catch (e) {
    metrics.counter("auth.failure").inc();        // ← Prometheus
    alerter.fire("auth.failure.streak", {          // ← 第 N 次才升級
      streak: state.authFailureCount,
      window: "13d"
    });
    state.authFailureCount++;
    throw e;  // ← 不吞, 讓 process exit / supervisor restart
  }
}

Scenario 4 修法前：App 端 fetch 失敗直接 render empty，不區分 loading / error / empty。

// before
const res = await api.getTransactions();
setTransactions(res.data ?? []);  // ← 500 與空資料同 UI

// after
const res = await api.getTransactions();
if (res.status >= 500) {
  setError({ code: res.status, retry: api.getTransactions });
  return;
}
setTransactions(res.data ?? []);

為什麼這麼難早發現

靜默失敗的設計初衷通常是「優雅降級」——不要因為一個小錯誤炸掉整個服務。這個邏輯在容錯設計上是對的，副作用是讓錯誤從可見變成隱形。降級路徑吞掉的不只是錯誤，還有排查的線索。

OAuth 過期的那起案例，日誌裡每一條「未認證跳過」都是線索，但沒有任何機制把這件事聚合成一個訊號推出來。個別的 skip 看起來無害，連續 13 天的 skip 才是問題——但沒有人每天去數。

App 吞錯顯示空列表，是很常見的前端防禦設計，避免把後端錯誤直接暴露給使用者。代價是：後端 500 變成使用者端的「空資料」，兩個層面都沒有人知道發生了什麼。

該被隔離的側效應類型

event log：寫入失敗不應回頭影響主操作，但應聚合成 metric（寫入量驟降本身就是訊號）。
推播通知：第三方 SMTP / FCM / APNS 失敗 → queue 重試，不可同步阻塞主流程。
webhook：對外 callback 失敗應 outbox pattern 重投，不能讓被呼叫方把整筆交易拖死。
快取失效：cache invalidation 失敗不應讓寫入失敗，但下次讀到 stale 資料要可被觀察。
搜尋索引：index 更新與 DB commit 的先後順序錯了，搜尋結果就會與事實脫節；需要 reconciliation job。
analytics：埋點失敗絕對不能影響業務邏輯，但埋點率本身要監控。
OAuth refresh：token 過期要主動告警，不能只在 log 裡 skip。
Redis fallback：型別錯誤不能無聲降級，要丟 metric + 自動 fail-closed（拒絕服務而非給錯資料）。
port / config drift：版本升級後 listener port 改了，反代沒跟上——需要 deployment 時的 config diff 檢查。

判斷標準：如果這段邏輯失敗不應讓使用者看到「操作失敗」，但又會讓業務結果偏離預期，那它就需要邊界隔離 + 獨立告警通道。降級要吵鬧，不能安靜。

確認方式與留給未來的一件事

這四起事故能更早被發現，靠的是更接近業務的指標：日誌行數驟降、產出數量歸零、API 調用量異常。這些變化在事後看起來都很明顯，但沒有設定基線比對的話，變化本身不會主動出現在任何儀表板上。

端到端煙霧測試也是關鍵缺口——真正調用業務邏輯並驗證結果。port 錯位的事故如果有一個每五分鐘打一次真實 API endpoint 的測試，22 小時會縮短成幾分鐘。

最具體的修法方向：降級必發告警、靜默失敗視同當機、App 錯誤狀態顯性呈現並附重試按鈕。降級要吵鬧，不能安靜。

下次遇到「執行成功但結果不對」，第一個懷疑的對象是監控指標的定義從一開始就量錯了東西。

— 邱柏宇

pm2 Says Online. The Service Is Dead.

The compressor is running. The light is on. But the thermostat broke three days ago and everything inside is rotting. Checking the indicator light tells nothing — open the door and smell.

Four incidents in one week. All four had green dashboards.

Technical Environment

Four incidents, four different stacks — but the same gap: monitoring and business outcomes live in different layers. Process managers (pm2, systemd, Docker) report process liveness. Reverse proxies (nginx) report proxy reachability. Schedulers (cron, Celery beat) report task execution. APIs report HTTP status codes. None of these layers independently verify whether the business result is correct.

Three execution modes triggered the failures: long-running daemons, scheduled jobs, and synchronous request handlers. All three share a pattern: the main flow returns a success signal, then depends on downstream side effects to complete the business outcome — and those side effects, if swallowed or treated as non-critical, vanish silently.

The data layer spans MySQL 8, Redis, and an external OAuth provider. Each has its own failure mode (type incompatibility, key type drift, token expiry). None of them are framework-specific — any "main op success → synchronous side effect" design reproduces the same behavior.

Four Ways to Die with a Green Light

First: the container manager showed online the entire time. nginx was returning 502 for over 22 hours. A version upgrade had changed the backend’s listening port, but the reverse proxy config still pointed to the old one. The container itself was healthy. The business API was completely unreachable. The admin interface still worked, which is exactly why no one looked harder.

Second: OAuth credentials expired. A scheduled job kept sending heartbeats every minute for 13 days straight — and producing nothing. Each run logged “unauthenticated, skipping” and moved on. No alert fired. From a process-monitoring perspective, everything was alive. From a business perspective, 13 days of silence.

Third: a content pipeline reported execution success, but a Redis key type mismatch triggered a silent degradation path. The error was swallowed. Fallback content was used instead. Output existed, just wrong output. Without actively diffing results against a baseline, there was no visible signal anything had changed.

Fourth: a transaction list API started returning 500 due to a parameter type incompatibility. The app silently swallowed the error and displayed an empty list. Users saw nothing. Likely assumption: no transactions yet. This ran for nearly a month before someone complained.

Error Propagation Sequence

Scenario 1 — port drift (22h outage)
  Docker           nginx            backend (port 8081 → 8082)
    |                 |                      |
    | health OK       |                      |
    |                 |── proxy :8081 ───────>|  ← stale env, connection refused
    |                 |<── 502 ───────────────|
    |                 |                      |
  monitoring: container online OK / API: 502 FAIL / business: dead

Scenario 2 — OAuth expired (13d silent)
  cron              service          OAuth provider
    |                   |                     |
    |── tick ──────────>|                     |
    |                   |── refresh token ───>|
    |                   |<── 401 ─────────────|
    |                   |  log: skip          |  ← swallowed, no alert
    |                   |                     |
  monitoring: heartbeat OK / output: 0 / business: silent for 13 days

Scenario 3 — Redis type drift (silent degradation)
  pipeline          handler          Redis
    |                   |                  |
    |── run ───────────>|                  |
    |                   |── GET key ───────>|
    |                   |<── WRONGTYPE ─────|
    |                   |  fallback: dummy  |  ← silent degrade
    |<── success ──────|                   |
    |                   |                  |
  monitoring: status OK / output: exists / business: wrong content

Scenario 4 — MySQL param type (30d hidden)
  App               API              MySQL 8
    |                 |                   |
    |── GET /txns ───>|                   |
    |                 |── SELECT … WHERE id=? →|
    |                 |<── 500 ────────────|
    |  show empty list|                   |  ← swallow error, no retry
    |                 |                   |
  monitoring: app OK / API: 500 FAIL / business: users see nothing

Common node across all four chains: monitoring flips green at "main flow returned success." Downstream failures, when they happen inside synchronous side effects and aren't escalated to alerts, propagate silently down to the end user. The terminal layer (the app, the business owner) receives either empty data or wrong content — not a signal that something upstream broke.

Where the Line Actually Is

These four incidents had four different root causes — port drift after a version upgrade, an OAuth refresh token race condition, Redis key type leakage across agents, MySQL 8 prepared statement parameter handling. Nothing in common technically.

What they share: monitoring was normal, business results were broken.

The standard uptime model assumes process alive ≈ service available. That assumption failed in every single one of these cases. Process alive ≠ service available. Execution success ≠ output produced. Heartbeat ≠ valid authentication. Status 200 ≠ correct data.

Partial functionality creates the most dangerous illusion. A green dashboard stops the investigation before it starts. By the time a user files a complaint, days or weeks have passed.

Code Diff: Before and After

Scenario 2, before: the cron handler catches refresh token failure, logs once, and skips.

// before
async function tick() {
  try {
    await refreshToken();
    await doWork();
  } catch (e) {
    logger.warn("unauthenticated, skipping");  // ← problem is here
  }
}
// monitoring: heartbeat OK, output zero, no alert

Scenario 2, after: auth failure is treated as a crash. Emit a metric and trigger an alert channel.

// after
async function tick() {
  try {
    await refreshToken();
    await doWork();
  } catch (e) {
    metrics.counter("auth.failure").inc();        // ← Prometheus
    alerter.fire("auth.failure.streak", {          // ← escalate after N
      streak: state.authFailureCount,
      window: "13d"
    });
    state.authFailureCount++;
    throw e;  // ← do not swallow; let the process exit / supervisor restart
  }
}

Scenario 4, before: the app's fetch failure renders as empty, with no distinction between loading / error / empty.

// before
const res = await api.getTransactions();
setTransactions(res.data ?? []);  // ← 500 and empty data look identical

// after
const res = await api.getTransactions();
if (res.status >= 500) {
  setError({ code: res.status, retry: api.getTransactions });
  return;
}
setTransactions(res.data ?? []);

Why These Stay Hidden

Silent failure is usually designed in deliberately. Graceful degradation, fallback logic, swallowing non-critical errors — all reasonable patterns for preventing one broken thing from taking down the whole service. The cost is that errors stop being visible. The degradation path swallows not just the error but the diagnostic signal.

In the OAuth case, every single “unauthenticated, skipping” log entry was a clue. But no mechanism aggregated them into a signal. One skip looks fine. Thirteen consecutive days of skips is a crisis — except no one was counting.

An app showing an empty list instead of an error is standard defensive frontend design. The tradeoff: a backend 500 becomes “no data” at the user layer. Neither layer surfaces what actually happened.

Side Effects That Should Be Isolated

Event log: write failure must not roll back the main op, but should aggregate into a metric — a drop in write volume is itself a signal.
Push notifications: third-party SMTP / FCM / APNS failures → queued retry, never synchronously block the main flow.
Webhooks: outbound callback failures should use an outbox pattern for replay. The caller must never be allowed to drag a whole transaction down.
Cache invalidation: invalidation failure must not fail the write, but stale reads must be observable on the next access.
Search index: index updates and DB commits must keep order; otherwise search diverges from fact. A reconciliation job is required.
Analytics: tracking failures must never affect business logic, but the tracking rate itself is a metric to watch.
OAuth refresh: token expiry must alert proactively. Logging "skip" once is not enough.
Redis fallback: type errors cannot degrade silently. Emit a metric and fail closed (refuse the request instead of returning wrong data).
Port / config drift: version upgrades that change listener ports must be caught by a deployment-time config diff check before they reach production.

Rule of thumb: if this logic failing should not show the user "operation failed," but it would still cause business outcomes to diverge from expected, then it needs a boundary plus its own alert channel. Fallbacks must be loud.

What to Watch Instead

All four incidents could have been caught earlier with business-layer metrics: log volume dropping sharply, output count hitting zero, API call volume diverging from baseline. In hindsight these signals look obvious. Without a baseline to compare against, they don’t appear in any dashboard automatically.

End-to-end smoke tests are the other gap — actually invoking business logic and validating the result. If the port-drift incident had a test hitting a real API endpoint every five minutes, the 22-hour outage becomes a five-minute alert.

The concrete fixes: degradation must fire an alert, silent failure should be treated as equivalent to a crash, app error states need to be visible with a retry button. Fallbacks should be loud.

Next time something reports execution success but produces wrong results — the first suspect is whether the monitoring metric was measuring the right thing from the start.