備用鑰匙開門太順，根本不會知道主鑰匙其實斷了。等到備用鑰匙也丟了,才發現兩把都沒了。

連續兩天,一個情報掃描 agent 的 web_search 工具回報「disabled or no provider is available」。每天照常交付:戰略情報 5 則、文化情報 3 則,準時落地,沒有任何察覺異常。工具層在燒,任務層一片平靜。

故障與失敗之間有一道沉默的牆

這道牆叫做 容錯機制。agent 內建多層 fallback:主工具掛掉,自動改抓 RSS feed;RSS 不通,直接 fetch 已知的媒體 URL。只要任一層成功,流程就視為「完成」,錯誤安靜地寫進 session log,不往上浮。

表面看起來是韌性。實際上是一種沉默。

問題不是 fallback 設計錯了。問題是系統設計讓「有產出」與「主路徑健康」這兩件事看起來等價,但根本不是。一個在吃備援管道的 agent,跟一個主工具正常運作的 agent,產出結果可能完全一樣——直到備援也垮掉的那天為止。

容易在哪裡踩進去

第一個誤判點:把「有產出」當成「系統正常」的憑證。這個等式在多數情況下成立,就是危險的原因。例外恰好是最不會去查的那一天。

第二個誤判點:以為工具層的錯誤會噴到最上層,讓察覺。實際上中間層會把錯誤消化掉。log 裡有記錄,但沒有去讀 log,因為任務完成了。

第三個誤判點也是最危險的:fallback 順序設計不當,最後一層也是易失效的服務。某天條件對了,多層同時垮,才會突然全斷。那個「突然」在外部看起來毫無預警,但 log 裡其實已經記了好幾百次降級紀錄。

怎麼確認系統是不是在吃備援

翻 agent session log,搜尋主工具關鍵字,看實際呼叫成功率。接著比對兩個數字:「任務完成數」和「主要工具成功數」。如果後者遠低於前者,系統就在吃 fallback,而且可能已經吃了一段時間。

這兩個數字平時不會出現在任何儀表板上,因為沒有覺得需要。把數字放進監控,加一條規則:主要工具連續 N 次未被成功呼叫,自動發警報。這條規則平時不會響,但響的那天,救的是接下來幾週的技術債。

還有一個不常做但很有效的手段:定期手動把 fallback 關掉跑一輪,驗證主路徑是否仍然健康。這是唯一能確認「主鑰匙還沒斷」的方式。

靜默累積的代價

情報掃描任務容錯空間高,備援資料來源的品質差距不大,這次降級運作兩天沒有明顯損失。但同樣的架構,換成金流 API、換成定價資料同步、換成任何對精確度敏感的系統,兩週後可能才從對帳單發現問題。

技術債的一個典型形態,就是沒有知道在累積。fallback 是為了韌性設計的,但韌性本身可以成為盲點——當主路徑失效率超過某個門檻,卻沒有任何告警觸發,等於系統在用「看起來正常」掩蓋「其實不正常」。

每週自動統計各工具實際使用率,fallback 觸發超過閾值就發警報,這兩件事的工程量都不大。沒有做,不是因為難,是因為在系統「看起來正常」的時候,沒有動機去做。

這就是備用鑰匙問題的核心:讓沒有機會知道主鑰匙斷了。

— 邱柏宇

---

Output Doesn’t Mean Healthy — Your Fallback Is Lying to You

The backup key opens the door so smoothly that nobody notices the main key snapped in half. Discovery comes only when the backup goes missing too — and suddenly nothing works.

For two consecutive days, an intelligence-scanning agent reported its web_search tool as “disabled or no provider is available.” Deliveries continued without interruption: strategic reports, cultural briefs, all on schedule. Nobody flagged anything. The tool layer was on fire; the task layer showed green.

There’s a Silent Wall Between Failure and Fault

That wall is called a fault-tolerance mechanism. The agent ran a multi-layer fallback chain: primary tool down, switch to RSS feeds; RSS unavailable, directly fetch known media URLs. As long as any layer succeeded, the workflow marked itself complete. Errors went quietly into the session log and stayed there — no upstream escalation, no alert, nothing.

It reads like resilience. It functions like silence.

The problem isn’t that fallback exists. The problem is that the system makes “produced output” and “primary path healthy” look equivalent — and they aren’t. An agent running on its third-tier backup produces the same artifact as one whose primary tool is firing cleanly. Until the backup collapses too, no one can tell the difference from the outside.

Three Places Where the Misread Happens

The first: treating output as a health signal. This equation holds often enough that it feels safe. That’s exactly why it’s dangerous — the exception is the day least likely to get checked.

The second: assuming tool-layer errors will bubble up and make noise. They don’t. The middle layer absorbs them. The log has the record; nobody reads the log because the task finished.

The third, and the worst: if the fallback chain is poorly ordered and the last resort is itself an unreliable service, everything holds until the day conditions align and multiple layers fail simultaneously. From outside, it looks like a sudden outage with no warning. Inside the logs, there are hundreds of silent degradation entries going back weeks.

How to Check Whether the System Is Already on Backup

Pull the agent session log. Search for the primary tool name. Measure actual call success rate. Then compare two numbers: tasks completed vs. primary tool calls succeeded. If the second is significantly lower than the first, the system has been running on fallback — and has been for a while.

Neither number typically appears in any dashboard, because until now there was no perceived need. Adding a monitoring rule that fires when the primary tool goes N consecutive rounds without a successful call takes an afternoon. That rule will almost never trigger. The day it does, it stops weeks of silent debt accumulation.

One more practice that rarely gets done but earns its keep: periodically disable fallback entirely and run a cycle. It’s the only real confirmation that the main key still works.

What Silent Accumulation Actually Costs

Intelligence scanning has high fault tolerance. Backup data sources are close enough in quality that two days of degraded operation left no measurable damage. Swap the same architecture into a payment API, a pricing sync, or anything accuracy-sensitive — and the problem surfaces two weeks later in a reconciliation report.

Technical debt accumulates in exactly this mode: invisibly, while the system continues to report normal. Fallback was designed to buy resilience. Resilience itself becomes a blind spot when primary tool failure crosses a meaningful threshold and no alert ever fires.

Weekly automated stats on actual tool usage rates. An alert threshold on fallback trigger counts. Neither is hard to build. The reason they don’t exist isn’t engineering complexity — it’s that when a system looks fine, there’s no pressure to look closer.

That’s the backup-key problem in one sentence: it removes the signal that would tell the main key is gone.

— 邱柏宇

延伸閱讀

• 400 不等於壞掉，只是還沒好
• fetch is not defined，沙箱不是 Node.js
• 週報第一行偶發亂碼：UTF-8 字元被 TCP chunk 切半
• robots.txt 一直都在，只是裡面裝的是 HTML
• 那通超時的請求，對方早就接到了