{{ message }}
Repository navigation
Do we need a timeout mechanism? #636
Description
Is your feature request related to a problem? Please describe.
Some attack recipes are extremely slow, so a timeout mechanism for a single attack on one sample is needed.
Describe the solution you'd like
Use a timeout context manager like
@contextmanager
def timeout(duration: float):
"""
A context manager that raises a `TimeoutError` after a specified time.
Parameters
----------
duration: float,
the time duration in seconds,
should be non-negative,
0 for no timeout
References
----------
https://stackoverflow.com/questions/492519/timeout-on-a-function-call
"""
if np.isinf(duration):
duration = 0
elif duration < 0:
raise ValueError("duration must be non-negative")
elif duration > 0: # granularity is 1 second, so round up
duration = max(1, int(duration))
def timeout_handler(signum, frame):
raise TimeoutError(f"block timedout after {duration} seconds")
signal.signal(signal.SIGALRM, timeout_handler)
signal.alarm(duration)
yield
signal.alarm(0)and replace this line
result = self._attack(goal_function_result)with, for example,
try:
with timeout(duration=time_out):
result = self._attack(goal_function_result)
except TimeoutError:
result = FailedAttackResult(goal_function_result, goal_function_result)
except Exception as e:
if ignore_errors:
# print(e)
result = FailedAttackResult(
goal_function_result, goal_function_result
)
else:
raise eOne can also add a time-out argument in commands.textattack_cli, and pass this argument in related classes and functions.
Describe alternatives you've considered
NA
Additional context
NA
Activity
jxmorris12 commentedon May 25, 2022
I think this is a great idea @wenh06. The attack search process is exponential in nature so it's easy to get into a state where the attack will never finish in a reasonable amount of time. It could also be nice to try and predict how long the attack would take, and warn the user if they're running something that is very very slow.
jxmorris12 commentedon Jun 9, 2022
Just thinking about this a little more:
TimedOutAttackResultorExpiredAttackResultor something like that, and show it in the results (X skipped, Y succeeded, Z failed, and M timed out) if users specify a timeout.kenneth-ge commentedon Mar 10, 2024
Bump on this! I really like this, and have had to implement a custom timeout feature in my own production code, so this would definitely be really useful. The issue is that most attacks finish in just a couple minutes, but some small proportion of attacks take more than 2 days to finish, so having a timeout mechanism allows us to finish most of our attacks in a reasonable amount of time
qiyanjun commentedon Aug 13, 2026
Still relevant, and the design thread above (new `TimedOutAttackResult` vs. treating timeout as failure, cross-platform interrupt mechanism, interaction with `--parallel` worker timeouts) never reached a conclusion — that's real design work, not something I want to unilaterally decide and merge as a "fix." No code change from me here; flagging that this is still open and well-scoped for whoever picks it up, with the open design questions being: (1) new result type or reuse `FailedAttackResult`, (2) signal-based vs. thread/process-based timeout (signal-based doesn't work well cross-platform or inside multiprocessing workers), (3) per-example vs. whole-dataset budget.