gh-95778: Correctly pre-check for int-to-str conversion (#96537) · python/cpython@b126196 · GitHub
Skip to content

Commit b126196

Browse files
mdickinsongpshead
mdickinson
and
gpshead
authored
gh-95778: Correctly pre-check for int-to-str conversion (#96537)
Converting a large enough `int` to a decimal string raises `ValueError` as expected. However, the raise comes _after_ the quadratic-time base-conversion algorithm has run to completion. For effective DOS prevention, we need some kind of check before entering the quadratic-time loop. Oops! =) The quick fix: essentially we catch _most_ values that exceed the threshold up front. Those that slip through will still be on the small side (read: sufficiently fast), and will get caught by the existing check so that the limit remains exact. The justification for the current check. The C code check is: ```c max_str_digits / (3 * PyLong_SHIFT) <= (size_a - 11) / 10 ``` In GitHub markdown math-speak, writing $M$ for `max_str_digits`, $L$ for `PyLong_SHIFT` and $s$ for `size_a`, that check is: $$\left\lfloor\frac{M}{3L}\right\rfloor \le \left\lfloor\frac{s - 11}{10}\right\rfloor$$ From this it follows that $$\frac{M}{3L} < \frac{s-1}{10}$$ hence that $$\frac{L(s-1)}{M} > \frac{10}{3} > \log_2(10).$$ So $$2^{L(s-1)} > 10^M.$$ But our input integer $a$ satisfies $|a| \ge 2^{L(s-1)}$, so $|a|$ is larger than $10^M$. This shows that we don't accidentally capture anything _below_ the intended limit in the check. <!-- gh-issue-number: gh-95778 --> * Issue: gh-95778 <!-- /gh-issue-number --> Co-authored-by: Gregory P. Smith [Google LLC] <greg@krypto.org>
1 parent 6adb89f commit b126196

4 files changed

Lines changed: 107 additions & 7 deletions

File tree

‎Include/internal/pycore_long.h‎

Lines changed: 2 additions & 2 deletions

‎Lib/test/test_int.py‎

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
import sys
2+
import time
23

34
import unittest
45
from test import support
@@ -632,6 +633,87 @@ def test_max_str_digits(self):
632633
with self.assertRaises(ValueError):
633634
str(i)
634635

636+
def test_denial_of_service_prevented_int_to_str(self):
637+
"""Regression test: ensure we fail before performing O(N**2) work."""
638+
maxdigits = sys.get_int_max_str_digits()
639+
assert maxdigits < 50_000, maxdigits # A test prerequisite.
640+
get_time = time.process_time
641+
if get_time() <= 0: # some platforms like WASM lack process_time()
642+
get_time = time.monotonic
643+
644+
huge_int = int(f'0x{"c"*65_000}', base=16) # 78268 decimal digits.
645+
digits = 78_268
646+
with support.adjust_int_max_str_digits(digits):
647+
start = get_time()
648+
huge_decimal = str(huge_int)
649+
seconds_to_convert = get_time() - start
650+
self.assertEqual(len(huge_decimal), digits)
651+
# Ensuring that we chose a slow enough conversion to measure.
652+
# It takes 0.1 seconds on a Zen based cloud VM in an opt build.
653+
if seconds_to_convert < 0.005:
654+
raise unittest.SkipTest('"slow" conversion took only '
655+
f'{seconds_to_convert} seconds.')
656+
657+
# We test with the limit almost at the size needed to check performance.
658+
# The performant limit check is slightly fuzzy, give it a some room.
659+
with support.adjust_int_max_str_digits(int(.995 * digits)):
660+
with self.assertRaises(ValueError) as err:
661+
start = get_time()
662+
str(huge_int)
663+
seconds_to_fail_huge = get_time() - start
664+
self.assertIn('conversion', str(err.exception))
665+
self.assertLess(seconds_to_fail_huge, seconds_to_convert/8)
666+
667+
# Now we test that a conversion that would take 30x as long also fails
668+
# in a similarly fast fashion.
669+
extra_huge_int = int(f'0x{"c"*500_000}', base=16) # 602060 digits.
670+
with self.assertRaises(ValueError) as err:
671+
start = get_time()
672+
# If not limited, 8 seconds said Zen based cloud VM.
673+
str(extra_huge_int)
674+
seconds_to_fail_extra_huge = get_time() - start
675+
self.assertIn('conversion', str(err.exception))
676+
self.assertLess(seconds_to_fail_extra_huge, seconds_to_convert/8)
677+
678+
def test_denial_of_service_prevented_str_to_int(self):
679+
"""Regression test: ensure we fail before performing O(N**2) work."""
680+
maxdigits = sys.get_int_max_str_digits()
681+
assert maxdigits < 100_000, maxdigits # A test prerequisite.
682+
get_time = time.process_time
683+
if get_time() <= 0: # some platforms like WASM lack process_time()
684+
get_time = time.monotonic
685+
686+
digits = 133700
687+
huge = '8'*digits
688+
with support.adjust_int_max_str_digits(digits):
689+
start = get_time()
690+
int(huge)
691+
seconds_to_convert = get_time() - start
692+
# Ensuring that we chose a slow enough conversion to measure.
693+
# It takes 0.1 seconds on a Zen based cloud VM in an opt build.
694+
if seconds_to_convert < 0.005:
695+
raise unittest.SkipTest('"slow" conversion took only '
696+
f'{seconds_to_convert} seconds.')
697+
698+
with support.adjust_int_max_str_digits(digits - 1):
699+
with self.assertRaises(ValueError) as err:
700+
start = get_time()
701+
int(huge)
702+
seconds_to_fail_huge = get_time() - start
703+
self.assertIn('conversion', str(err.exception))
704+
self.assertLess(seconds_to_fail_huge, seconds_to_convert/8)
705+
706+
# Now we test that a conversion that would take 30x as long also fails
707+
# in a similarly fast fashion.
708+
extra_huge = '7'*1_200_000
709+
with self.assertRaises(ValueError) as err:
710+
start = get_time()
711+
# If not limited, 8 seconds in the Zen based cloud VM.
712+
int(extra_huge)
713+
seconds_to_fail_extra_huge = get_time() - start
714+
self.assertIn('conversion', str(err.exception))
715+
self.assertLess(seconds_to_fail_extra_huge, seconds_to_convert/8)
716+
635717
def test_power_of_two_bases_unlimited(self):
636718
"""The limit does not apply to power of 2 bases."""
637719
maxdigits = sys.get_int_max_str_digits()

‎Misc/NEWS.d/next/Security/2022-08-07-16-53-38.gh-issue-95778.ch010gps.rst‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,4 +11,4 @@ limitation <int_max_str_digits>` documentation. The default limit is 4300
1111
digits in string form.
1212

1313
Patch by Gregory P. Smith [Google] and Christian Heimes [Red Hat] with feedback
14-
from Victor Stinner, Thomas Wouters, Steve Dower, and Ned Deily.
14+
from Victor Stinner, Thomas Wouters, Steve Dower, Ned Deily, and Mark Dickinson.

‎Objects/longobject.c‎

Lines changed: 22 additions & 4 deletions

0 commit comments

Comments
 (0)