觸發記錄全綠，但任務從來沒有真正開始

有點像在台灣用便利商店店到店寄件：追蹤頁面一直顯示「已出庫、運送中」，但包裹在第一個中繼站就靜默卡住，從沒真正派出去。你每天查、查了四天，系統從沒報過一個錯。排程任務也會這樣。先講結論：觸發成功不等於任務開始跑，排程層只管「按下按鈕」，執行環境有沒有真的起來，要另外有人回報。

情境

很常見的排程架構：排程器（cron 或任何定時觸發器）照時間觸發任務，派工層收到觸發就回 202，然後替每一次執行開一個獨立的隔離環境（獨立行程、容器或沙箱），在裡面載入執行環境、工具設定與任務設定，再開始跑任務本身。隔離環境有啟動時限，假設是 60 秒，超過就放棄退出。

這次碰到的是：某個排程任務在一次平台升級之後，連續四天沒有任何輸出。觸發記錄完整，時間準確，日誌欄位沒有空缺，一切看起來都在運作。唯一的異狀是：預期的結果從沒出現。

錯誤傳染鏈（時序）

Scheduler            Dispatcher             Isolated Worker
    |                       |                      |
    |── trigger job ───────>|                      |
    |<── 202 Accepted ──────|                      |
    |                       |── spawn worker ──────>|
    |                       |                      |── boot runtime...
    |                       |                      |── load tools config...
    |                       |                      |── load task config...
    |                       |                      |── [T+60s timeout] ←── 超時點 ✗
    |                       |                      |<── silent exit (no output)
    |                       |   (no failure event propagated back)
    |                       |                      |
    trigger log: ✓ success                     task: never started ✗

排程層收到 202 就認為觸發完成；失敗發生在更底層的啟動階段，那一層沒有義務向排程層回報，這就是全程零告警的原因。

分界點在升級那一刻

升級前任務跑得好好的，升級後開始靜默失敗。這個分界點本身就是第一條線索，但很容易被略過，因為「升級」通常被當成無害的背景事件，不是排查的起點。

往下挖，根因是那次升級改了隔離環境的初始化流程，啟動變慢，還沒準備好就撞到 60 秒時限退出。任務連第一行邏輯都沒碰到。不是跑到一半逾時，是連門都沒進。

為什麼沒有人發現

難點不在技術，在通知架構。排程器負責「觸發」這一層，它確實觸發了，也確實記錄了成功。隔離環境的逾時發生在另一個更底層的地方，那一層沒有義務回報失敗，兩者之間沒有橋。

結果就是全程零告警。排程器覺得自己做完了，隔離環境在角落裡靜靜逾時，沒有人叫。這種跨層靜默比單一層的錯誤更難察覺，因為每一層單獨看都沒有問題。

最關鍵的確認來自對照組：同一台機器上，直接在常駐主程序裡執行、不另開隔離環境的其他排程任務，這四天一直正常。同樣的機器、同樣的時間、同樣的排程器，差別只在執行模式。

修法比預想的簡單

把出問題的任務從「每次另開隔離環境」改成「在已經跑著的常駐主程序裡執行」，繞過那段會逾時的啟動流程。很多排程器改完設定會自動重讀，不用重啟服務。

這是繞過，不是根治。隔離環境為什麼在升級後、在某些機器上啟動超過 60 秒，問題還開著；低規格、長時間沒重啟、初始化某一步卡住都有可能，但沒有一個被完整驗證成唯一原因。

代價也要記得：常駐主程序是共用的，隔離環境提供的狀態隔離就沒了。當下這個取捨合理，但不是零成本。

設定對照：修法前後（示意，欄位名依你的排程器而定）

修法前（每次觸發另開隔離環境，啟動逾時就靜默失敗）

{
  "name": "nightly-report",
  "schedule": "0 13 * * *",
  "runMode": "isolated",        // ← 每次觸發都另開一個隔離環境，啟動時限 60 秒
  "payload": { "task": "產出今日報表" }
}

修法後（在常駐主程序內執行，不經過隔離環境的啟動）

{
  "name": "nightly-report",
  "schedule": "0 13 * * *",
  "runMode": "main",            // ← 直接交給已經在跑的主程序，不用等啟動
  "payload": { "task": "產出今日報表" }
}

不想放棄隔離的話，另一條路是讓任務一開跑就先寫一筆「已開始」，再用一支小看門狗比對「有觸發、沒開始」：

# 看門狗：觸發後 N 分鐘內沒有「已開始」記錄就告警
for job in triggered_jobs(since=now - 2*TIMEOUT):
    if job.triggered_at < now - TIMEOUT and not started(job.run_id):
        alert(f"{job.name} 觸發了但沒有開始執行（run {job.run_id}）")

該被隔離的側效應類型

排程觸發記錄（event log）：寫觸發記錄這件事要跟執行層脫鉤，觸發成功不能拿來當執行成功的代理指標。
執行環境生命週期回報（lifecycle hook）：啟動成功、逾時退出、執行完成三個事件要分別往上報，不能讓排程層只看得到「觸發 202」。
跨層失敗通知（cross-layer failure propagation）：隔離環境啟動逾時，要有一條明確的失敗事件路徑送回排程器，而不是靜默丟掉。
可觀測性輸出（observability sink）：各階段時間戳、退出原因、退出碼寫進查得到的 log，不要只留在那個環境自己的記憶體裡。
健康探針（health probe）：定期量啟動耗時，接近時限就先告警，不要等結果缺席才發現。
重試佇列（retry queue）：啟動逾時多半是暫時性失敗，應該自動進重試，而不是讓那次執行窗口直接作廢。
主機資源警告（host resource alert）：低規格或長時間沒重啟的機器，啟動耗時逼近門檻時就該示警。
升級回歸測試（upgrade regression gate）：會動到初始化流程的升級，先在測試環境對每種執行模式量完整啟動時間，別等上線才發現。

判斷標準很簡單：如果這一層失敗時，不該讓排程器以為任務成功了，它就需要自己的狀態回報路徑。

留給下次的一件事

觸發記錄正常，不等於任務開始執行。大部分時候這兩件事同步發生，但在分層架構裡它們可以分開：排程層只管觸發，執行層有自己的生命週期，中間那條通道可以無聲斷掉。

下次排程任務靜默沒產出、觸發記錄卻一片綠燈，我第一個查的會是：執行環境本身有沒有成功啟動。不是任務邏輯跑到哪裡，而是它有沒有進門。

— 邱柏宇

Every Trigger Logged Success. Nothing Ever Ran.

It resembles the convenience-store parcel service common across Taiwan: the tracking page keeps showing “dispatched, in transit,” but the package silently stalled at the first relay point and never moved. You check daily for four days. The system never throws an error. Scheduled jobs can do the same thing. Short version: a successful trigger is not a started job. The scheduler only presses the button; whether the execution environment actually came up has to be reported by something else.

The Setup

A common scheduling architecture: a scheduler (cron or any timed trigger) fires a job, a dispatcher answers 202 and spins up an isolated environment for each run (a separate process, container or sandbox), which loads the runtime, tools config and task config before running the task itself. The isolated environment has a startup deadline, say 60 seconds, and gives up if it misses it.

What happened here: after a platform upgrade, one scheduled job produced no output for four days straight. Trigger logs were intact, timestamps accurate, log fields fully populated. Everything looked operational. The only anomaly was that no result ever appeared.

Error Propagation Sequence

Scheduler            Dispatcher             Isolated Worker
    |                       |                      |
    |── trigger job ───────>|                      |
    |<── 202 Accepted ──────|                      |
    |                       |── spawn worker ──────>|
    |                       |                      |── boot runtime...
    |                       |                      |── load tools config...
    |                       |                      |── load task config...
    |                       |                      |── [T+60s timeout] ←── failure point ✗
    |                       |                      |<── silent exit (no output)
    |                       |   (no failure event propagated back)
    |                       |                      |
    trigger log: ✓ success                     task: never started ✗

The scheduler considers its work done on receiving 202; the failure happens in the lower startup layer, which has no obligation to report upward. That gap is why no alert ever fired.

The Boundary Was the Upgrade

Before the upgrade the job ran fine; after it, silent failure began. That boundary is the first thread worth pulling, but it gets skipped because upgrades tend to be treated as neutral background events rather than suspects.

Digging in, the upgrade had changed the isolated environment’s initialization sequence. Startup got slower and hit the 60-second deadline before it was ready. The job never executed a single line of logic. Not a mid-run timeout: it never got through the door.

Why No One Was Notified

The scheduler did its job: it triggered the task and logged a success. The startup timeout happened at a different, lower layer, one with no obligation to report failure back up. No bridge between them, so no alert.

Viewed one at a time, every layer had nothing to report. This kind of cross-layer silence is harder to catch than a single-layer error precisely because neither layer is wrong about its own scope.

The clearest confirmation was a control group: on the same machine, other scheduled jobs that ran inside the long-lived main process, without spawning an isolated environment, kept working all four days. Same host, same schedule, same scheduler. The only difference was the run mode.

The Fix Was Simpler Than Expected

Switching the failing job from “spawn an isolated environment per run” to “run inside the already-running main process” bypassed the slow startup entirely. Many schedulers re-read config automatically, so no restart was needed.

This is a workaround, not a root fix. Why the isolated environment takes over 60 seconds to start on some machines after this upgrade is still open; low-spec hardware, long uptime and a stuck initialization step are all candidates, but none is confirmed as the sole cause.

The tradeoff is real: the main process is shared, so the isolation the separate environment provided is gone. Acceptable here, but not free.

Config Diff: Before and After (illustrative; field names depend on your scheduler)

Before (isolated environment per run; startup timeout fails silently)

{
  "name": "nightly-report",
  "schedule": "0 13 * * *",
  "runMode": "isolated",        // ← new isolated environment per trigger, 60s startup deadline
  "payload": { "task": "build today's report" }
}

After (runs inside the long-lived main process; no isolated startup)

{
  "name": "nightly-report",
  "schedule": "0 13 * * *",
  "runMode": "main",            // ← hand it to the process that is already running
  "payload": { "task": "build today's report" }
}

If you want to keep the isolation, the other route is to have the job write a “started” record as its very first step, and run a small watchdog that compares “triggered” against “started”:

# Watchdog: alert when a job was triggered but never recorded a start within N minutes
for job in triggered_jobs(since=now - 2*TIMEOUT):
    if job.triggered_at < now - TIMEOUT and not started(job.run_id):
        alert(f"{job.name} was triggered but never started (run {job.run_id})")

Side Effects That Should Be Isolated

Trigger event logging: Writing a trigger record must be decoupled from execution; a successful trigger log must never stand in for a successful run.
Lifecycle hooks: Startup success, timeout exit and completion should each emit their own upstream event, so the scheduler sees more than “202 accepted.”
Cross-layer failure propagation: When an isolated environment times out on startup, a failure event needs a defined path back to the scheduler instead of being dropped.
Observability sink: Stage timestamps, exit reasons and exit codes belong in a queryable log store, not only in the environment’s own memory.
Health probes: Measure startup duration continuously and alert as it approaches the deadline, before results go missing.
Retry queue: A startup timeout is usually transient and should be retried automatically rather than burning the execution window.
Host resource alerting: Low-spec or long-uptime machines should warn when startup latency nears the threshold.
Upgrade regression gate: Any upgrade that touches initialization should get a startup-time benchmark across every run mode in staging first.

The test: if this layer’s failure should not let the scheduler believe the job succeeded, it needs its own status reporting path.

One Thing for Next Time

A clean trigger log does not mean the job started. Most of the time the two happen together; in a layered architecture they can come apart. The scheduler handles triggers, the execution environment has its own lifecycle, and the channel between them can go dark without a sound.

The next time a scheduled job quietly produces nothing while the trigger log stays green, the first thing I check is whether the execution environment itself started. Not where the logic stalled, but whether it ever got through the door.