[chore] dsdgen generation signed integer overflow fix by taniabogatsch · Pull Request #20279 · duckdb/duckdb · GitHub
Skip to content

[chore] dsdgen generation signed integer overflow fix - #20279

Merged
Mytherin merged 3 commits into
duckdb:v1.4-andiumfrom
taniabogatsch:fuzzer-fixes
Jan 6, 2026
Merged

Mytherin merged 3 commits into
duckdb:v1.4-andiumfrom
taniabogatsch:fuzzer-fixes

Conversation

@taniabogatsch

Copy link
Copy Markdown
Member

@taniabogatsch

taniabogatsch commented Dec 21, 2025 •

Copy link
Copy Markdown
Member Author

@Mytherin

Copy link
Copy Markdown
Member

Thanks for the PR!

Generally for TPC-H / TPC-DS we want to potentially generate higher scale factors, so having the scale factor to 777 could actually be limiting. The code is copied as-is, so it wouldn't surprise me if this overflow also exists in the original implementation.

I'm not sure how to best deal with this - we could fix the overflow by e.g. using unsigned integer arithmetic so that we wrap instead, but who knows in how many places these overflows exist. We also need to make sure we don't change the data that is generated which is time consuming to check at such high scale factors.

@taniabogatsch

Copy link
Copy Markdown
Member Author

Hmmm - so currently we already cannot generate any scale factor beyond 777. With the changes here in place we at least make the problem apparent if we ever want to generate them by throwing an easy-to-understand exception... 🤔

we could fix the overflow by e.g. using unsigned integer arithmetic so that we wrap instead, but who knows in how many places these overflows exist

As for fixing this - I agree that I don't see a "good" solution that does not require fixing the copied-in code/changing the data generation...

@taniabogatsch

Copy link
Copy Markdown
Member Author

I am also not sure if we actually want to go down the path of fixing this until we actually need to generate it?

@Mytherin

Copy link
Copy Markdown
Member

Hmmm - so currently we already cannot generate any scale factor beyond 777. With the changes here in place we at least make the problem apparent if we ever want to generate them by throwing an easy-to-understand exception... 🤔

Is that due to a different reason than the signed integer overflow? Signed integer overflows, while technically undefined behavior, are in practice effectively always wrapping. In debug mode the sanitizers trigger, but in release mode they don't. If the original dsdgen contains the signed integer overflow then that means that is "expected" in a way, since all larger TPC-DS datasets including official ones would be generated with this overflow (likely triggering the wrapping behavior).

We could throw this exception in debug mode only using e.g. #ifdef DEBUG. Then we can still generate the large sets in release mode. Maybe we want to eventually fix this but agreed this is low priority.

@taniabogatsch

Copy link
Copy Markdown
Member Author

In debug mode the sanitizers trigger, but in release mode they don't. If the original dsdgen contains the signed integer overflow then that means that is "expected" in a way, [...] We could throw this exception in debug mode only using e.g. #ifdef DEBUG.

Okay - I've removed the test now since I don't think we can tag it as "only run in debug mode" (AFAIK). I've also wrapped the exception in #ifdef DEBUG. Then, we can close the fuzzer issue so the fuzzer no longer triggers on this, I'd say?

@duckdb-draftbot
duckdb-draftbot marked this pull request as draft December 27, 2025 16:16
@taniabogatsch
taniabogatsch marked this pull request as ready for review December 27, 2025 16:18
@Mytherin
Mytherin merged commit be28a10 into duckdb:v1.4-andium Jan 6, 2026
104 checks passed
@Mytherin

Mytherin commented Jan 6, 2026

Copy link
Copy Markdown
Member

@taniabogatsch
taniabogatsch deleted the fuzzer-fixes branch January 6, 2026 09:15
github-actions Bot pushed a commit to duckdb/duckdb-r that referenced this pull request Jan 6, 2026
Add QueryContext also to Write(void *buffer, idx_t nr_bytes), so that BytesWritten are updated also there (duckdb/duckdb#20393)
Give preference do variables set in config (duckdb/duckdb#20396)
[chore] dsdgen generation signed integer overflow fix (duckdb/duckdb#20279)
Fixup comparison to wrong iterator on abdc's driver (duckdb/duckdb#20360)
github-actions Bot added a commit to duckdb/duckdb-r that referenced this pull request Jan 6, 2026
Add QueryContext also to Write(void *buffer, idx_t nr_bytes), so that BytesWritten are updated also there (duckdb/duckdb#20393)
Give preference do variables set in config (duckdb/duckdb#20396)
[chore] dsdgen generation signed integer overflow fix (duckdb/duckdb#20279)
Fixup comparison to wrong iterator on abdc's driver (duckdb/duckdb#20360)

Co-authored-by: krlmlr <krlmlr@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants