# 週報第一行偶發亂碼：UTF-8 字元被 TCP chunk 切半

- URL: https://justfly.idv.tw/%e9%80%b1%e5%a0%b1%e7%ac%ac%e4%b8%80%e8%a1%8c%e7%9a%84%e4%ba%82%e7%a2%bc%ef%bc%8c%e6%98%af-tcp-%e9%82%84%e6%b2%92%e8%aa%aa%e5%ae%8c%e8%a9%b1/
- 日期: 2026-06-08
- 分類: WEB&RIA
- 標籤: javascript, JSON, server, 神島

![週報第一行偶發亂碼：UTF-8 字元被 TCP chunk 切半]

每週定時跑的報告腳本，多數週正常。偶爾——不是每次——輸出的第一行中文變成 `�`。沒有 exception，也沒有錯誤日誌，只有輸出內容異常。

##### 現象

多數週輸出正常。問題偶發：第一行某個中文字變成替代符號 `�`。不固定哪個字、不固定哪一週，無可預測的觸發條件。重跑、換執行時間、換資料量，皆無法穩定複現。

腳本寫法很直覺：監聽 HTTP 回應的 `data` 事件，每來一個 `chunk` 就用 `+=` 串成字串。資料到達即處理，邏輯上無漏接，當下看不出問題。

##### 先查錯的方向

第一個懷疑：Content-Type 或伺服器端編碼設定。檢查後排除。伺服器回應正常，UTF-8 宣告存在，問題不在 `response header`。

第二個方向：「偶發即外部問題」——網路抖動、CDN 快取、第三方服務異常。以上方向皆未命中。真正問題在本地腳本的 response 收集策略，跟外部無關。

##### 真正的原因

多數常見中文字在 UTF-8 中是三個 byte，UTF-8 是變長編碼。真正觸發條件是：多位元組字元剛好被切在兩個 `chunk` 的邊界上。

前一個 `chunk` 先到，尾端帶著不完整的 `byte sequence`。腳本立刻轉字串，Node.js 遇到不合法序列，填入 `U+FFFD`。後一個 `chunk` 到達時，原本的 `byte sequence` 已被拆開處理，輸出結果因此出現 `�` 或局部亂碼。

`chunk` 多大由網路層決定，不固定，這就是偶發的來源。多數時候字元落在完整邊界，輸出正常；偶爾切在中間，就產生亂碼。兩種情況在程式碼層一致，因此難以重現與定位。簡單說：字元還沒收完，就先解碼。

##### 修法

確認方式：在 `data` callback 印出每個 `chunk` 的 byte 長度與尾端 bytes，確認是否存在尾端不完整的 UTF-8 `byte sequence`。

修法只有一個方向：不要在 `data` 事件裡轉字串。處理方式：

- `data` 階段：只收 `Buffer`

- `end` 階段：`Buffer.concat(chunks)`

- 最後：`.toString('utf8')`

這樣不管 TCP 怎麼切，解碼時看到的都是完整 `byte sequence`。改用此法後，未再出現亂碼。

##### 留給下次的一件事

「資料到達即處理」這個直覺在多數情境成立，但在邊界條件下不可靠——當資料單位（多位元組字元）與傳輸單位（TCP `chunk`）沒有對齊保證時。下次碰到偶發的編碼問題，先確認：解碼發生在「片段」階段，還是「完整資料集合」階段？若解碼發生在片段階段，偶發亂碼屬於預期結果，不是隨機異常。

— 邱柏宇

##### 延伸閱讀

- [川崎夜市不再是異國展覽，它成了日本人自己的祭典](https://justfly.idv.tw/%e5%b7%9d%e5%b4%8e%e5%a4%9c%e5%b8%82%e4%b8%8d%e5%86%8d%e6%98%af%e7%95%b0%e5%9c%8b%e5%b1%95%e8%a6%bd%ef%bc%8c%e5%ae%83%e6%88%90%e4%ba%86%e6%97%a5%e6%9c%ac%e4%ba%ba%e8%87%aa%e5%b7%b1%e7%9a%84%e7%a5%ad/)

- [spec 寫對了，女角開口卻是男聲](https://justfly.idv.tw/spec-%e5%af%ab%e5%b0%8d%e4%ba%86%ef%bc%8c%e5%a5%b3%e8%a7%92%e9%96%8b%e5%8f%a3%e5%8d%bb%e6%98%af%e7%94%b7%e8%81%b2/)

- [阿嬤的手與AI的眼，守著同一塊土地](https://justfly.idv.tw/taiwan-grandma-orchid-ai-conservation/)

### The Garbled First Line: A UTF-8 Character Split Across TCP Chunks

A scheduled weekly report script. Most weeks the output is normal. Occasionally — not every run — the first line of Chinese comes out as `�`. No exception, no error log, only abnormal output content.

##### The Phenomenon

Output is normal most weeks. The corruption is intermittent: one Chinese character in the first line becomes a replacement symbol `�`. No fixed character, no fixed week, no predictable trigger. Re-running, changing the run time, changing the data size — none reproduces it reliably.

The script was written the obvious way: listen to the HTTP response’s `data` event, concatenate each `chunk` with `+=` as it arrives. Data is processed on arrival; logically nothing is dropped; nothing looks wrong at the time.

##### Wrong Directions First

First suspect: Content-Type or server-side encoding. Checked and ruled out. The server response is normal, UTF-8 is declared; the problem is not in the `response header`.

Second direction: “intermittent means external” — network jitter, CDN caching, upstream instability. None of these hit. The actual problem is in the local script’s response-collection strategy, unrelated to anything external.

##### The Actual Cause

Most common Chinese characters are three bytes in UTF-8, but UTF-8 is fundamentally a variable-length encoding. The real trigger condition: a multi-byte character is split across two `chunk` boundaries.

The previous `chunk` arrives first, its tail carrying an incomplete `byte sequence`. The script converts it to a string immediately; Node.js hits an invalid sequence and inserts `U+FFFD`. When the next `chunk` arrives, the original `byte sequence` has already been split and processed, so the output shows `�` or partial corruption.

`chunk` size is decided by the network layer and is not fixed — that is the source of the intermittence. Most of the time characters land on complete boundaries and output is normal; occasionally one is cut mid-character, and corruption appears. Both cases are identical at the code level, which makes it hard to reproduce and locate. In short: decoding starts before the character is fully received.

##### The Fix

Confirmation: in the `data` callback, log each `chunk`‘s byte length and tail bytes, and check whether an incomplete UTF-8 `byte sequence` exists at the tail.

One direction for the fix: do not convert to string inside the `data` event. Handling:

- `data` stage: collect `Buffer`s only

- `end` stage: `Buffer.concat(chunks)`

- finally: `.toString('utf8')`

This way, no matter how TCP splits the stream, decoding always sees a complete `byte sequence`. After switching to this approach, the corruption did not recur.

##### One Thing Worth Remembering

The instinct to “process data on arrival” holds in most situations but is unreliable at the boundary — when the unit of meaning (a multi-byte character) has no alignment guarantee with the unit of transmission (the TCP `chunk`). Next time an intermittent encoding bug appears, confirm one thing first: does decoding happen at the fragment stage or the complete-dataset stage? If decoding happens at the fragment stage, intermittent corruption is an expected result, not a random anomaly.

— 邱柏宇

##### Related Posts

- [川崎夜市不再是異國展覽，它成了日本人自己的祭典](https://justfly.idv.tw/%e5%b7%9d%e5%b4%8e%e5%a4%9c%e5%b8%82%e4%b8%8d%e5%86%8d%e6%98%af%e7%95%b0%e5%9c%8b%e5%b1%95%e8%a6%bd%ef%bc%8c%e5%ae%83%e6%88%90%e4%ba%86%e6%97%a5%e6%9c%ac%e4%ba%ba%e8%87%aa%e5%b7%b1%e7%9a%84%e7%a5%ad/)

- [spec 寫對了，女角開口卻是男聲](https://justfly.idv.tw/spec-%e5%af%ab%e5%b0%8d%e4%ba%86%ef%bc%8c%e5%a5%b3%e8%a7%92%e9%96%8b%e5%8f%a3%e5%8d%bb%e6%98%af%e7%94%b7%e8%81%b2/)

- [阿嬤的手與AI的眼，守著同一塊土地](https://justfly.idv.tw/taiwan-grandma-orchid-ai-conservation/)