# robots.txt 一直都在，只是裡面裝的是 HTML
- URL: https://justfly.idv.tw/robots-txt-%e4%b8%80%e7%9b%b4%e9%83%bd%e5%9c%a8%ef%bc%8c%e5%8f%aa%e6%98%af%e8%a3%a1%e9%9d%a2%e8%a3%9d%e7%9a%84%e6%98%af-html/
- 日期: 2026-05-01
- 分類: WEB&RIA
- 標籤: server, web, 神島
![robots.txt 一直都在，只是裡面裝的是 HTML]
夜市攤位旁貼了 QR code，掃下去跳出的是隔壁補習班的招生頁。你要的是菜單，拿到的是廣告，狀態碼還是 200，一切看起來正常。這次的問題結構一模一樣。
##### 五天，零成功
4 月 24 日起，IG 發文連續五天 0 成功。排程卡在 QUEUE，4 月 28 日回的錯誤碼是 subcode 2207076，訊息寫「robots.txt blocked」。第一反應是外部平台的問題，或者 Integration 設定錯了，或者 token 過期了。
robots.txt 本身從來不是懷疑對象。路徑一直有回應，HTTP 200，沒有任何錯誤訊號。
##### 分界點在反向代理
根本原因在 Caddy 沒有為 /robots.txt 設專屬路由。請求進來，fall through 到 SPA 的 catch-all，SPA 把所有路徑都回傳同一份 HTML，Content-Type 是 text/html，狀態碼 200。
Meta 的爬蟲收到的不是 text/plain 的爬蟲指令，而是整頁 HTML。保守的實作把這個回應解讀為「robots.txt blocked」，然後對整個 domain 的媒體停止抓取。
任何監控都不會對 200 回應發出警報。回應看起來完全正常，問題因此藏住了。要發現這件事，需要有人主動去看回應的 Content-Type，或者去看回應的 body。
##### 容易誤診的原因
SPA 的 catch-all 路由設計本來就是要接住所有未定義路徑，防止 404。這個設計在大多數情境下是對的。問題只在 /robots.txt 這個例外——它是協定層級的約定，外部爬蟲對它的 Content-Type 有嚴格預期，跟一般頁面路徑不同。
兩個設計之間的衝突沒有人特別注意，因為兩邊各自都工作正常。Caddy 轉發了，SPA 回應了，HTTP 200 出去了。只有爬蟲那一側默默做了判斷，然後停了。
##### 確認與修法
驗證方式直接：curl -I 那個路徑，看 Content-Type。如果拿到的是 text/html 而不是 text/plain，問題就在這裡。
修法在反向代理層加一條明確路由，讓 /robots.txt 在 SPA catch-all 之前被攔截，直接回傳靜態文字檔，狀態碼 200，Content-Type text/plain，內容包含對 facebookexternalhit、meta-externalagent、Instagram 的明確 Allow 規則。SPA 的其他路由不用動。
下次碰到「外部服務突然停止抓取」的症狀，先確認 /robots.txt 的回應格式，不只是狀態碼。200 不代表對方拿到了想要的東西。
— 邱柏宇
##### 延伸閱讀
- [明知贏不了K-pop，台灣偶像為什麼還要做自己](https://justfly.idv.tw/%e6%98%8e%e7%9f%a5%e8%b4%8f%e4%b8%8d%e4%ba%86k-pop%ef%bc%8c%e5%8f%b0%e7%81%a3%e5%81%b6%e5%83%8f%e7%82%ba%e4%bb%80%e9%ba%bc%e9%82%84%e8%a6%81%e5%81%9a%e8%87%aa%e5%b7%b1/)
- [一月八日的出廠日期，讓三個月後的每個請求都回 400](https://justfly.idv.tw/%e4%b8%80%e6%9c%88%e5%85%ab%e6%97%a5%e7%9a%84%e5%87%ba%e5%bb%a0%e6%97%a5%e6%9c%9f%ef%bc%8c%e8%ae%93%e4%b8%89%e5%80%8b%e6%9c%88%e5%be%8c%e7%9a%84%e6%af%8f%e5%80%8b%e8%ab%8b%e6%b1%82%e9%83%bd%e5%9b%9e-4/)
- [2000隻青蛙如何把殖民時代的錯誤分類變成台灣的生態主權](https://justfly.idv.tw/2000%e9%9a%bb%e9%9d%92%e8%9b%99%e5%a6%82%e4%bd%95%e6%8a%8a%e6%ae%96%e6%b0%91%e6%99%82%e4%bb%a3%e7%9a%84%e9%8c%af%e8%aa%a4%e5%88%86%e9%a1%9e%e8%ae%8a%e6%88%90%e5%8f%b0%e7%81%a3%e7%9a%84%e7%94%9f/)
### robots.txt Was Always There, Just Full of HTML
A QR code on a night market stall, scanned, and up comes an ad for a cram school next door. You wanted the menu. You got something else. HTTP 200 the whole time, no error, nothing visibly wrong. This incident had the same structure.
##### Five Days, Zero Posts
Starting April 24, IG publishing hit zero successful posts for five consecutive days. Jobs stacked in QUEUE. By April 28, the error code was subcode 2207076 — “robots.txt blocked”. The obvious suspects were an external platform issue, a broken integration, an expired token.
The robots.txt path itself was never under suspicion. It always responded. HTTP 200. No alerts, no errors.
##### The Break Point Was in the Reverse Proxy
Caddy had no dedicated route for /robots.txt. Requests fell through to the SPA’s catch-all, which returned the same HTML document for every path — Content-Type: text/html, status 200.
Meta’s crawler received HTML instead of plain-text crawl directives. A conservative implementation read that as “robots.txt blocked” and stopped fetching media from the entire domain.
Nothing in the system was throwing an error. A 200 response doesn’t trigger monitoring alerts. Catching this required someone to actively inspect the Content-Type — or the body — of the robots.txt response.
##### Why It Gets Misdiagnosed
SPA catch-all routing exists precisely to prevent 404s on unmatched paths. It does its job correctly in nearly every case. The conflict only surfaces at /robots.txt, which isn’t a page path — it’s a protocol-level convention. External crawlers expect text/plain. The SPA delivers text/html. Both sides behave as designed; only the crawler silently draws a conclusion.
##### Confirming It, Fixing It
One check: `curl -I` the path, read the Content-Type. If it returns text/html instead of text/plain, the cause is confirmed.
The fix goes in the reverse proxy layer — a dedicated route that intercepts /robots.txt before the SPA catch-all, serving a static plain-text file with explicit Allow rules for facebookexternalhit, meta-externalagent, and Instagram. Nothing else in the SPA routing needs to change.
Next time a crawler stops fetching without an obvious error: check robots.txt’s response format, not just its status code. 200 only means the server responded. It says nothing about what the client was expecting to receive.
— 邱柏宇
##### Related Posts
- [明知贏不了K-pop，台灣偶像為什麼還要做自己](https://justfly.idv.tw/%e6%98%8e%e7%9f%a5%e8%b4%8f%e4%b8%8d%e4%ba%86k-pop%ef%bc%8c%e5%8f%b0%e7%81%a3%e5%81%b6%e5%83%8f%e7%82%ba%e4%bb%80%e9%ba%bc%e9%82%84%e8%a6%81%e5%81%9a%e8%87%aa%e5%b7%b1/)
- [一月八日的出廠日期，讓三個月後的每個請求都回 400](https://justfly.idv.tw/%e4%b8%80%e6%9c%88%e5%85%ab%e6%97%a5%e7%9a%84%e5%87%ba%e5%bb%a0%e6%97%a5%e6%9c%9f%ef%bc%8c%e8%ae%93%e4%b8%89%e5%80%8b%e6%9c%88%e5%be%8c%e7%9a%84%e6%af%8f%e5%80%8b%e8%ab%8b%e6%b1%82%e9%83%bd%e5%9b%9e-4/)
- [2000隻青蛙如何把殖民時代的錯誤分類變成台灣的生態主權](https://justfly.idv.tw/2000%e9%9a%bb%e9%9d%92%e8%9b%99%e5%a6%82%e4%bd%95%e6%8a%8a%e6%ae%96%e6%b0%91%e6%99%82%e4%bb%a3%e7%9a%84%e9%8c%af%e8%aa%a4%e5%88%86%e9%a1%9e%e8%ae%8a%e6%88%90%e5%8f%b0%e7%81%a3%e7%9a%84%e7%94%9f/)