Repository navigation
[chore] dsdgen generation signed integer overflow fix - #20279
Conversation
|
Thanks for the PR! Generally for TPC-H / TPC-DS we want to potentially generate higher scale factors, so having the scale factor to 777 could actually be limiting. The code is copied as-is, so it wouldn't surprise me if this overflow also exists in the original implementation. I'm not sure how to best deal with this - we could fix the overflow by e.g. using unsigned integer arithmetic so that we wrap instead, but who knows in how many places these overflows exist. We also need to make sure we don't change the data that is generated which is time consuming to check at such high scale factors. |
|
Hmmm - so currently we already cannot generate any scale factor beyond 777. With the changes here in place we at least make the problem apparent if we ever want to generate them by throwing an easy-to-understand exception... 🤔
As for fixing this - I agree that I don't see a "good" solution that does not require fixing the copied-in code/changing the data generation... |
|
I am also not sure if we actually want to go down the path of fixing this until we actually need to generate it? |
Is that due to a different reason than the signed integer overflow? Signed integer overflows, while technically undefined behavior, are in practice effectively always wrapping. In debug mode the sanitizers trigger, but in release mode they don't. If the original dsdgen contains the signed integer overflow then that means that is "expected" in a way, since all larger TPC-DS datasets including official ones would be generated with this overflow (likely triggering the wrapping behavior). We could throw this exception in debug mode only using e.g. |
Okay - I've removed the test now since I don't think we can tag it as "only run in debug mode" (AFAIK). I've also wrapped the exception in |
Add QueryContext also to Write(void *buffer, idx_t nr_bytes), so that BytesWritten are updated also there (duckdb/duckdb#20393) Give preference do variables set in config (duckdb/duckdb#20396) [chore] dsdgen generation signed integer overflow fix (duckdb/duckdb#20279) Fixup comparison to wrong iterator on abdc's driver (duckdb/duckdb#20360)
Add QueryContext also to Write(void *buffer, idx_t nr_bytes), so that BytesWritten are updated also there (duckdb/duckdb#20393) Give preference do variables set in config (duckdb/duckdb#20396) [chore] dsdgen generation signed integer overflow fix (duckdb/duckdb#20279) Fixup comparison to wrong iterator on abdc's driver (duckdb/duckdb#20360) Co-authored-by: krlmlr <krlmlr@users.noreply.github.com>

Fix https://github.com/duckdblabs/duckdb-internal/issues/6497