bug: `COPY INTO … ON_ERROR = ABORT_N` skips bad rows (fewer than N) but reports `ERRORS_SEEN = 0` and no `FIRST_ERROR` · Issue #20782 · databendlabs/databend · GitHub
You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
bug: COPY INTO … ON_ERROR = ABORT_N skips bad rows (fewer than N) but reports ERRORS_SEEN = 0 and no FIRST_ERROR #20782
I had searched in the issues and found no similar issues.
Version
main e9b49fd (source build, v1.2.953-nightly-e9b49fde); also v1.2.952-nightly-8862a6285b and v1.2.925-patch-15.
What's Wrong?
With ON_ERROR = ABORT_N, rows that fail to parse are skipped as long as fewer than N errors are seen. That part is
intended. But the result of COPY INTO then reports the file with ERRORS_SEEN = 0 and FIRST_ERROR = NULL, even
though rows were dropped (the docs define ERRORS_SEEN as "Number of error rows in the source file"). With RETURN_FAILED_ONLY = TRUE the file is not listed at all. The same file loaded with ON_ERROR = CONTINUE correctly reports ERRORS_SEEN = 2 and the first error. Load pipelines that check ERRORS_SEEN / FIRST_ERROR (or use RETURN_FAILED_ONLY) therefore see a clean load, the file is marked as loaded, and nothing
records that rows are missing. NDJSON behaves the same.
Cause: on_error in src/query/storages/stage/src/read/error_handler.rs. In the OnErrorMode::Continue branch it
calls file_status.add_error(e.error, line). In the OnErrorMode::AbortNum(n) branch, when the error is tolerated, it
only increments on_error_count and returns Ok(()) without recording the error in file_status. Calling file_status.add_error(...) there as well would make ERRORS_SEEN/FIRST_ERROR (and RETURN_FAILED_ONLY) correct.
How to Reproduce?
CREATE STAGE IF NOT EXISTS s115;
COPY INTO @s115/a.csvFROM (SELECTnumber::STRING AS id, 'v'AS s FROM numbers(10)
UNION ALLSELECT'bad1', 'x'UNION ALLSELECT'bad2', 'y')
FILE_FORMAT = (TYPE = CSV) SINGLE = TRUE USE_RAW_PATH = TRUE; -- 12 rows, 2 of them not INTCREATE OR REPLACETABLEt_abort (id INT, s STRING);
COPY INTO t_abort FROM @s115 FILES = ('a.csv') FILE_FORMAT = (TYPE = CSV) ON_ERROR = ABORT_3;
-- a.csv | 10 | 0 | NULL | NULL <-- rows_loaded 10, errors_seen 0, first_error NULLSELECTcount(*) FROM t_abort; -- 10CREATE OR REPLACETABLEt_continue (id INT, s STRING);
COPY INTO t_continue FROM @s115 FILES = ('a.csv') FILE_FORMAT = (TYPE = CSV) ON_ERROR = CONTINUE;
-- a.csv | 10 | 2 | Invalid value 'bad1' for column 0 (id Int32 NULL): invalid text for number | 11CREATE OR REPLACETABLEt_abort2 (id INT, s STRING);
COPY INTO t_abort2 FROM @s115 FILES = ('a.csv') FILE_FORMAT = (TYPE = CSV) ON_ERROR = ABORT_3 RETURN_FAILED_ONLY = TRUE;
-- (empty result)
Expected: ABORT_3 reports errors_seen = 2 and the first error (as CONTINUE does), and RETURN_FAILED_ONLY lists
the file.
Activity
fix(storage): report tolerated errors for ON_ERROR=ABORT_N (databendl…