重試不是韌性：如何避免自動化管線定時製造噪音

火災警報器壞掉一直響，正確做法不是每 20 分鐘去按一次看會不會自己好——而是派人蹲點默默觀察，真的修好了再通知大家。這個比喻聽起來理所當然，但落到自動化管線的設計上，幾乎每個人都按過那顆沒用的按鈕。

連打九次的那個晚上

一個餐廳自動發文管線，上游搜尋 API 服務商發生大規模 outage，從 08-26 起持續卡死。舊設計是失敗就每 20 分鐘重試一次。結果連續 9 次失敗，每次都觸發一則錯誤通知，把通知通道輪番轟炸成垃圾訊息。

表面上看，這個管線有重試機制，有韌性。實際上，retry 的前提是「失敗是短暫的，會自動恢復」。當外部依賴的問題不是瞬時抖動，而是持續數小時到數天的系統性故障，這個前提早就失立。重試繼續跑，只是在定時製造噪音。

通知副作用的三層後果

每次重試失敗都觸發告警，這件事有三層傷害，而且一層比一層隱蔽。

第一層最直接：監控通道被淹沒，失去實際意義。第二層是 oncall 工程師疲勞轟炸，收到第四則通知時已經開始忽略了。第三層最危險：當真正需要人工介入的新故障出現，訊號會被前面累積的噪音掩蓋掉，根本看不到。

還有一個更隱蔽的細節：工作流引擎裡 includeData=true 的設定，看似方便除錯，實際上會讓內部佔位符（像 __n8n_ 這類欄位）也被掃進通知內容，通知格式變得更混亂，訊噪比更低。每次重試都在放大這個問題。

分界點：outage 性質的判斷

傳統指數退避重試在短暫的網路抖動或服務重啟場景下工作得很好。分界點在於：這次的失敗是「瞬時性的」還是「系統性的」？判斷方式不是靠直覺，而是看失敗持續的時間跨度。一旦超過某個閾值，管線應該主動切換策略，而不是繼續重試並繼續通知。

真正該做的是 probe-first：背景靜默探測服務是否恢復，探測失敗完全不發通知，探測成功才觸發真正的補發，補發成功後自動清除探測流程。

實測數字很清楚

改成 probe-first 之後，設定 manual 執行模式（不自動觸發錯誤通知），每 30 分鐘背景呼叫上游 API。隔夜共執行 17 次靜默探測，於 00:03 UTC 探測通過，隨即完整補發成功。全程只發出 2 則有效通知：探測通過、補發成功。零噪音。

對比舊設計的 9 則錯誤告警，這個差距說明了什麼是「有意義的通知」和「噪音」的實際分野。

留給下次的一件事

設計重試機制時，應該同時設計「通知副作用的靜默條件」。外部依賴的 outage 策略需要兩個階段分開處理：第一階段 probe-first，背景靜默探測，失敗不通知；第二階段 conditional-retry，探測通過才真正重試加通知。

另外值得明確標註的是「query-specific 止損點」，也就是這筆任務最晚接受到幾點才有補發的意義。時間窗口關閉之後，補發不一定是正確動作，繼續探測只是繼續消耗資源。本案設定的止損點是隔日 09:00，超過這個時間就應該自動放棄並記錄，而不是無限重試下去。

技術環境

工作流引擎：n8n（self-hosted on n8n-server）
排程：OpenClaw cron（imac）
觸發鏈：OpenClaw cron → n8n webhook → 餐廳內容發布 API → IG / Threads
上游依賴：第三方搜尋 API（生成餐廳內容素材）
通知通道：Slack #openclaw
模式切換：manual 執行（停用自動錯誤通知）

時序

時間 (UTC+9)	事件
08-26 起	上游 API 進入大規模 outage
08-29 晚	舊設計連 9 次 retry 失敗，9 則錯誤通知
08-29 深夜	切換 probe-first 模式，停自動通知
08-29 22:00 起	每 30 分鐘背景靜默探測
08-30 00:03 UTC	探測通過，觸發真實補發成功
08-30 09:00	query-specific 止損點（超過即放棄）

Code 對照

舊設計（retry-with-notify）：

// 失敗就重試 + 通知
if (response.status !== 200) {
  retry();           // 每 20 分鐘
  notify('ERROR');   // 每次都送
}

新設計（probe-first）：

// 探測與執行分開
probe();             // 背景靜默跑，失敗不通知
if (probe.status === 200) {
  execute();         // 探測通過才執行
  notify('SUCCESS'); // 只在成功路徑送
}

側效應清單

監控通道被 9 則同質錯誤淹沒，信號失去意義
oncall 工程師疲勞，第 4 則通知後注意力開始下降
n8n includeData=true 讓內部佔位符（__n8n_ 開頭欄位）�進通知 payload，訊噪比再降一階
後續真正需要人工介入的新故障被前面累積的噪音蓋掉，看不見
無止損點時，probe-first 會變成另一種形式的無限迴圈，只是更安靜

— 邱柏宇

When Your Retry Loop Becomes the Attack

A restaurant’s auto-posting pipeline stalled when its upstream search API provider went down in a large-scale outage starting August 26th. The old design was straightforward: on failure, retry every 20 minutes. Over the course of one night, it retried 9 times. Each retry triggered an error notification. Nine alerts, zero useful information.

The fire alarm was broken and someone kept pressing the button every 20 minutes to see if it would fix itself.

The False Comfort of “Having Retry”

A retry mechanism looks like resilience on paper. And it is — under the right conditions. The assumption baked into every retry loop is that failures are transient: a brief network hiccup, a service restart, something that resolves itself within minutes. When the upstream dependency enters a systemic outage lasting hours or days, that assumption collapses. The retry keeps firing. The notification side-effect keeps firing with it.

The damage compounds in three layers. First, the monitoring channel floods and loses meaning. Second, the oncall engineer starts ignoring alerts by the fourth message. Third — most dangerous — when a genuinely new failure appears and needs human intervention, it’s invisible under the noise.

There’s an additional wrinkle: workflow engines configured with includeData=true for debugging convenience will sweep internal placeholder fields (like __n8n_ prefixed keys) into notification payloads, making each alert harder to parse. Every retry amplifies the mess.

The Threshold That Changes Everything

The distinction isn’t between “good retry” and “bad retry.” It’s about correctly classifying failure type. Transient failures — a fault-tolerant retry with exponential backoff handles these fine. Systemic outages — a completely different protocol is needed, one that separates the detection question from the notification question.

The probe-first pattern addresses this directly. Instead of retrying the actual job and notifying on each failure, a background probe workflow runs silently on a schedule — manual execution mode, no automatic error alerts. It pings the upstream API every 30 minutes. Only when the probe succeeds does it trigger the real retry and send a notification. When the probe fails, nothing is sent.

What the Numbers Look Like

After switching to probe-first, the pipeline ran 17 silent probes overnight. At 00:03 UTC, the probe passed. The actual job ran and completed successfully. Total notifications sent across the entire incident: 2 — one for probe success, one for job completion. Zero noise.

Against the 9 error alerts from the old design, the operational difference is concrete. Nine alerts communicated nothing actionable. Two alerts communicated everything.

One Thing Worth Noting Next Time

Any “query-specific” cutoff should be explicitly set when designing the probe. In this case, the cutoff was 09:00 the following morning — after that point, the delayed post would no longer be relevant, and continuing to probe would just burn resources. Without a defined stop condition, probe-first becomes its own form of infinite loop, just a quieter one.

The design principle: separate the detection phase (silent, no alerts on failure) from the execution phase (only triggered on confirmed recovery, notification on completion). Retry and notification are not the same thing. They shouldn’t share an on/off switch.

Technical Environment

Workflow engine: n8n (self-hosted on n8n-server)
Scheduler: OpenClaw cron (imac)
Trigger chain: OpenClaw cron → n8n webhook → restaurant content publishing API → IG / Threads
Upstream dependency: third-party search API (restaurant content sourcing)
Notification channel: Slack #openclaw
Mode: manual execution (automatic error notification disabled)

Timeline

Time (UTC+9)	Event
08-26 onward	Upstream API enters large-scale outage
08-29 evening	Old design: 9 retries fail, 9 error alerts
08-29 late night	Switch to probe-first, disable auto alerts
08-29 22:00 onward	Background probe every 30 min, silent on failure
08-30 00:03 UTC	Probe succeeds, triggers real recovery
08-30 09:00	Query-specific cutoff (auto-abandon after this)

Code Comparison

Old design (retry-with-notify):

// Retry on failure + notify
if (response.status !== 200) {
  retry();           // every 20 min
  notify('ERROR');   // every failure
}

New design (probe-first):

// Detection and execution are decoupled
probe();             // silent background, no alert on failure
if (probe.status === 200) {
  execute();         // only execute after probe passes
  notify('SUCCESS'); // only on success path
}

Side Effects

Monitoring channel flooded by 9 homogeneous errors; signal becomes meaningless
oncall engineer fatigue; attention drops from the 4th alert onward
n8n includeData=true leaks internal placeholders (fields prefixed __n8n_) into the notification payload, degrading signal-to-noise further
Genuine new faults that need human intervention get buried under accumulated noise
Without an explicit stop condition, probe-first becomes its own form of infinite loop — just a quieter one