Repository navigation
fix(jobs): scrub build logs to valid UTF-8 before persisting - #13706
Conversation
Build output can contain raw binary, and truncating to the last N bytes can split a multibyte character. MySQL rejects both, so the log callback threw and the deployment's build logs stopped updating (CLOUD-3QXN).
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
|
|
||
| return \substr($logs, -$limit); | ||
| // Build output can be binary, and the byte cut can split a multibyte character; MySQL rejects either. | ||
| return \mb_scrub($logs, 'UTF-8'); |
There was a problem hiding this comment.
This production-incident fix has no behavioral regression coverage, so a later removal or reordering of the scrub could silently restore failed build-log persistence. Add a worker-level test that processes raw invalid bytes and a multibyte character split at the size boundary, then verifies that the persisted buildLogs is valid UTF-8 and remains bounded. Do not merely unit-test mb_scrub() or mirror this helper's implementation.
Knowledge Base Used: Functions and execution processing
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/Appwrite/Platform/Modules/Functions/Workers/Jobs.php
Line: 846
Comment:
**Missing Behavioral Regression Coverage**
This production-incident fix has no behavioral regression coverage, so a later removal or reordering of the scrub could silently restore failed build-log persistence. Add a worker-level test that processes raw invalid bytes and a multibyte character split at the size boundary, then verifies that the persisted `buildLogs` is valid UTF-8 and remains bounded. Do not merely unit-test `mb_scrub()` or mirror this helper's implementation.
**Knowledge Base Used:** [Functions and execution processing](https://app.greptile.com/appwrite/-/custom-context/knowledge-base/appwrite/appwrite/-/docs/functions-execution-api.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
✨ Benchmark resultsComparing
Per-scenario breakdown & investigation detailsMetrics below reflect the current branch (after). Δ P95 compares against the base.
Top API waits (after) |
2 participants
Summary
Fixes CLOUD-3QXN:
SQLSTATE[HY000]: General error: 1366 Incorrect string value: '\x82 gzip...' for column 'buildLogs'in the jobs worker, ~90 events over the last two weeks and still ongoing.Two ways invalid UTF-8 reaches
buildLogs. A build step can print raw binary (the gzip bytes in the Sentry title), andtruncate()keeps the lastAPP_LOG_LENGTH_LIMITbytes withsubstr, which can land inside a multibyte character. MySQL rejects either on write, soonLogthrows and the deployment's build logs stop updating.truncate()now passes the result throughmb_scrub(). EverybuildLogswrite in this worker goes through it (the log chunk, the exit trailer, and cloud's site-distribution override), so this one spot covers all of them.Testing
Pint passes. No test added for a one-line sanitisation.