WIP: update API reference · QData/TextAttack@b430d76 · GitHub
Skip to content

Commit b430d76

Browse files
jinyongyoojinyongyoo
committed
WIP: update API reference
1 parent 1d09830 commit b430d76

9 files changed

Lines changed: 171 additions & 57 deletions

File tree

‎docs/api/datasets.rst‎

Lines changed: 4 additions & 5 deletions

‎docs/api/goal_functions.rst‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
Goal Functions API Reference
2+
============================
3+
4+
:class:`~textattack.goal_functions.GoalFunction` determines both the conditions under which the attack is successful (in terms of the model outputs)
5+
and the heuristic score that we want to maximize when searching for the solution.
6+
7+
GoalFunction
8+
------------
9+
.. autoclass:: textattack.goal_functions.GoalFunction
10+
:members:
11+
12+
ClassifiedGoalFunction
13+
------------
14+
.. autoclass:: textattack.goal_functions.ClassifiedGoalFunction
15+
:members:
16+
17+
TargetedClassification
18+
------------
19+
.. autoclass:: textattack.goal_functions.TargetedGoalFunction
20+
:members:
21+
22+
UntargetedClassification
23+
------------
24+
.. autoclass:: textattack.goal_functions.UntargetedGoalFunction
25+
:members:
26+
27+
InputReduction
28+
------------
29+
.. autoclass:: textattack.goal_functions.InputReduction
30+
:members:
31+
32+
TextToTextGoalFunction
33+
------------
34+
.. autoclass:: textattack.goal_functions.TextToTextGoalFunction
35+
:members:
36+
37+
MinimizeBleu
38+
------------
39+
.. autoclass:: textattack.goal_functions.MinimizeBleu
40+
:members:
41+
42+
NonOverlappingOutput(
43+
------------
44+
.. autoclass:: textattack.goal_functions.NonOverlappingOutput(
45+
:members:
46+

‎docs/api/search_methods.rst‎

Lines changed: 40 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,46 @@
11
Search Methods API Reference
22
============================
33

4+
:class:`~textattack.search_methods.SearchMethod` attempts to find the optimal set of perturbations that will produce an adversarial example.
5+
Finding such optimal perturbations becomes a combinatorial optimization problem, and search methods are typically heuristic search algorithms designed
6+
to solve the underlying combinatorial problem.
7+
8+
More in-depth study of search algorithms for NLP adversarial attacks can be found in the following work
9+
`Searching for a Search Method: Benchmarking Search Algorithms for Generating NLP Adversarial Examples <https://arxiv.org/abs/2009.06368>`_
10+
by Jin Yong Yoo, John X. Morris, Eli Lifland, and Yanjun Qi.
11+
412
SearchMethod
513
------------
6-
.. automodule:: textattack.search_methods.SearchMethod
14+
.. autoclass:: textattack.search_methods.SearchMethod
15+
:members:
16+
17+
BeamSearch
18+
------------
19+
.. autoclass:: textattack.search_methods.BeamSearch
720
:members:
21+
22+
GreedySearch
23+
------------
24+
.. autoclass:: textattack.search_methods.GreedySearch
25+
:members:
26+
27+
GreedyWordSwapWIR
28+
------------------
29+
.. autoclass:: textattack.search_methods.GreedyWordSwapWIR
30+
:members:
31+
32+
AlzantotGeneticAlgorithm
33+
-------------------------
34+
.. autoclass:: textattack.search_methods.AlzantotGeneticAlgorithm
35+
:members:
36+
37+
ImprovedGeneticAlgorithm
38+
-------------------------
39+
.. autoclass:: textattack.search_methods.ImprovedGeneticAlgorithm
40+
:members:
41+
42+
ParticleSwarmOptimization
43+
--------------------------
44+
.. autoclass:: textattack.search_methods.ParticleSwarmOptimization
45+
:members:
46+

‎docs/index.rst‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,5 +59,7 @@ TextAttack Documentation
5959
Trainer <api/trainer.rst>
6060
Models <api/models/models.rst>
6161
Datasets <api/datasets.rst>
62+
GoalFunction <api/goal_functions.rst>
63+
SearchMethod <api/search_methods.rst>
6264

6365

‎textattack/datasets/dataset.py‎

Lines changed: 33 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -9,19 +9,28 @@ class Dataset(torch.utils.data.Dataset):
99
data via :meth:`__getitem__` and :meth:`__len__` methods.
1010
1111
Args:
12-
dataset (list_like): A list-like iterable of ``(input, output)`` pairs. Here, `output` can either be an integer representing labels for classification
13-
or a string for seq2seq tasks. If input consists of multiple sequences (e.g. SNLI), iterable should be of the form ``([input_1, input_2, ...], output)`` and ``input_columns`` parameter must be set.
14-
input_columns (:obj:`list[str]`, `optional`, defaults to :obj:`["text"]`):
12+
dataset (:obj:`list[tuple]`):
13+
A list of :obj:`(input, output)` pairs.
14+
If :obj:`input` consists of multiple fields (e.g. "premise" and "hypothesis" for SNLI),
15+
:obj:`input` must be of the form :obj:`(input_1, input_2, ...)` and :obj:`input_columns` parameter must be set.
16+
:obj:`output` can either be an integer representing labels for classification or a string for seq2seq tasks.
17+
input_columns (:obj:`list[str]`, `optional`, defaults to :obj:`["text"]`):
1518
List of column names of inputs in order.
1619
label_map (:obj:`dict[int, int]`, `optional`, defaults to :obj:`None`):
17-
Mapping if output labels of the dataset should be re-mapped. Useful if model was trained with a different label arrangement than
18-
provided in the ``datasets`` version of the dataset. For example, if dataset's arrangement is 0 for negative and 1 for positive, but model's label
19-
arrangement is 1 for negative and 0 for positive, pass ``{0: 1, 1: 0}``. Could also be used to remap literal labels to numerical labels,
20-
(e.g. ``{"positive": 1, "negative": 0}``)
21-
label_names (list[str], optional): List of label names in corresponding order (e.g. ``["World", "Sports", "Business", "Sci/Tech"] for AG-News dataset).
22-
If not set, labels will printed as is (e.g. "0", "1", ...). This should be set to ``None`` for non-classification datasets.
23-
output_scale_factor (float): Factor to divide ground-truth outputs by. Generally, TextAttack goal functions require model outputs between 0 and 1. Some datasets are regression tasks, in which case this is necessary.
24-
shuffle (bool): Whether to shuffle the dataset on load.
20+
Mapping if output labels of the dataset should be re-mapped. Useful if model was trained with a different label arrangement.
21+
For example, if dataset's arrangement is 0 for `Negative` and 1 for `Positive`, but model's label
22+
arrangement is 1 for `Negative` and 0 for `Positive`, passing :obj:`{0: 1, 1: 0}` will remap the dataset's label to match with model's arrangements.
23+
Could also be used to remap literal labels to numerical labels (e.g. :obj:`{"positive": 1, "negative": 0}`).
24+
label_names (:obj:`list[str]`, `optional`, defaults to :obj:`None`):
25+
List of label names in corresponding order (e.g. :obj:`["World", "Sports", "Business", "Sci/Tech"]` for AG-News dataset).
26+
If not set, labels will printed as is (e.g. "0", "1", ...). This should be set to :obj:`None` for non-classification datasets.
27+
output_scale_factor (:obj:`float`, `optional`, defaults to :obj:`None`):
28+
Factor to divide ground-truth outputs by. Generally, TextAttack goal functions require model outputs between 0 and 1.
29+
Some datasets are regression tasks, in which case this is necessary.
30+
shuffle (:obj:`bool`, `optional`, defaults to :obj:`False`): Whether to shuffle the underlying dataset.
31+
32+
.. note::
33+
Generally not recommended to shuffle the underlying dataset. Shuffling can be performed using DataLoader or by shuffling the order of indices we attack.
2534
2635
Examples::
2736
@@ -93,13 +102,21 @@ def shuffle(self):
93102
random.shuffle(self._dataset)
94103
self.shuffled = True
95104

96-
def __getitem__(self, i):
97-
if isinstance(i, int):
98-
return self._format_as_dict(self._dataset[i])
105+
def __getitem__(self, idx):
106+
"""Retrieves the specified item from dataset.
107+
108+
Args:
109+
idx (:obj:`int` or :obj:`slice`): Index or slice.
110+
Returns:
111+
:obj:`tuple[OrderedDict, Union[int, str]]` - Tuple of input as :obj:`collections.OrderedDict` and output as :obj:`int` or :obj:`str`.
112+
"""
113+
if isinstance(idx, int):
114+
return self._format_as_dict(self._dataset[idx])
99115
else:
100-
# `i` could be a slice or an integer. if it's a slice,
116+
# `idx` could be a slice or an integer. if it's a slice,
101117
# return the formatted version of the proper slice of the list
102-
return [self._format_as_dict(ex) for ex in self._dataset[i]]
118+
return [self._format_as_dict(ex) for ex in self._dataset[idx]]
103119

104120
def __len__(self):
121+
"""Returns the size of dataset."""
105122
return len(self._dataset)

‎textattack/datasets/huggingface_dataset.py‎

Lines changed: 37 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -57,27 +57,38 @@ class HuggingFaceDataset(Dataset):
5757
"""Loads a dataset from 🤗 Datasets and prepares it as a TextAttack dataset.
5858
5959
Args:
60-
name_or_dataset (Union[datasets.Dataset, str]): the dataset name or actual ``datasets.Dataset`` object. If it's your custom ``datasets.Dataset`` object, please pass the input and output columns via ``dataset_columns`` argument.
61-
subset (:obj:`str`, optional, defaults to :obj:`None`): the subset of the main dataset. Dataset will be loaded as ``datasets.load_dataset(name, subset)``. Default is ``None``.
62-
split (str, optioanl): the split of the dataset. Default is "train".
63-
dataset_columns (tuple(list[str], str)), optional): Pair of ``list[str]`` representing list of input column names (e.g. ["premise", "hypothesis"]) and ``str`` representing the output column name (e.g. ``label``). If not set, we will try to automatically determine column names from known designs.
64-
label_map (dict, optional): Mapping if output labels of the dataset should be re-mapped. Useful if model was trained with a different label arrangement than
65-
provided in the ``datasets`` version of the dataset. For example, if dataset's arrangement is 0 for negative and 1 for positive, but model's label
66-
arrangement is 1 for negative and 0 for positive, pass ``{0: 1, 1: 0}``. Could also be used to remap literal labels to numerical labels,
67-
(e.g. ``{"positive": 1, "negative": 0}``)
68-
label_names (list[str], optional): List of label names in corresponding order (e.g. ``["World", "Sports", "Business", "Sci/Tech"] for AG-News dataset).
69-
If ``datasets.Dataset`` object already has label names, then this is not required. Also, this should be set to ``None`` for non-classification datasets.
70-
output_scale_factor (float): Factor to divide ground-truth outputs by.
71-
Generally, TextAttack goal functions require model outputs between 0 and 1.
60+
name_or_dataset (:obj:`Union[str, datasets.Dataset]`):
61+
The dataset name as :obj:`str` or actual :obj:`datasets.Dataset` object.
62+
If it's your custom :obj:`datasets.Dataset` object, please pass the input and output columns via :obj:`dataset_columns` argument.
63+
subset (:obj:`str`, `optional`, defaults to :obj:`None`):
64+
The subset of the main dataset. Dataset will be loaded as :obj:`datasets.load_dataset(name, subset)`.
65+
split (:obj:`str`, `optional`, defaults to :obj:`"train"`):
66+
The split of the dataset.
67+
dataset_columns (:obj:`tuple(list[str], str))`, `optional`, defaults to :obj:`None`):
68+
Pair of :obj:`list[str]` representing list of input column names (e.g. :obj:`["premise", "hypothesis"]`)
69+
and :obj:`str` representing the output column name (e.g. :obj:`label`). If not set, we will try to automatically determine column names from known designs.
70+
label_map (:obj:`dict[int, int]`, `optional`, defaults to :obj:`None`):
71+
Mapping if output labels of the dataset should be re-mapped. Useful if model was trained with a different label arrangement.
72+
For example, if dataset's arrangement is 0 for `Negative` and 1 for `Positive`, but model's label
73+
arrangement is 1 for `Negative` and 0 for `Positive`, passing :obj:`{0: 1, 1: 0}` will remap the dataset's label to match with model's arrangements.
74+
Could also be used to remap literal labels to numerical labels (e.g. :obj:`{"positive": 1, "negative": 0}`).
75+
label_names (:obj:`list[str]`, `optional`, defaults to :obj:`None`):
76+
List of label names in corresponding order (e.g. :obj:`["World", "Sports", "Business", "Sci/Tech"]` for AG-News dataset).
77+
If not set, labels will printed as is (e.g. "0", "1", ...). This should be set to :obj:`None` for non-classification datasets.
78+
output_scale_factor (:obj:`float`, `optional`, defaults to :obj:`None`):
79+
Factor to divide ground-truth outputs by. Generally, TextAttack goal functions require model outputs between 0 and 1.
7280
Some datasets are regression tasks, in which case this is necessary.
73-
shuffle (bool): Whether to shuffle the dataset on load.
81+
shuffle (:obj:`bool`, `optional`, defaults to :obj:`False`): Whether to shuffle the underlying dataset.
82+
83+
.. note::
84+
Generally not recommended to shuffle the underlying dataset. Shuffling can be performed using DataLoader or by shuffling the order of indices we attack.
7485
"""
7586

7687
def __init__(
7788
self,
7889
name_or_dataset,
7990
subset=None,
80-
split="train",
91+
split="test",
8192
dataset_columns=None,
8293
label_map=None,
8394
label_names=None,
@@ -139,14 +150,21 @@ def _format_as_dict(self, example):
139150

140151
return (input_dict, output)
141152

142-
def __getitem__(self, i):
143-
if isinstance(i, int):
144-
return self._format_as_dict(self._dataset[i])
153+
def __getitem__(self, idx):
154+
"""Retrieves the specified item from dataset.
155+
156+
Args:
157+
idx (:obj:`int` or :obj:`slice`): Index or slice.
158+
Returns:
159+
:obj:`tuple[OrderedDict, Union[int, str]]` - Tuple of input as :obj:`collections.OrderedDict` and output as :obj:`int` or :obj:`str`.
160+
"""
161+
if isinstance(idx, int):
162+
return self._format_as_dict(self._dataset[idx])
145163
else:
146-
# `i` could be a slice or an integer. if it's a slice,
164+
# `idx` could be a slice or an integer. if it's a slice,
147165
# return the formatted version of the proper slice of the list
148166
return [
149-
self._format_as_dict(self._dataset[j]) for j in range(i.start, i.stop)
167+
self._format_as_dict(self._dataset[j]) for j in range(idx.start, idx.stop)
150168
]
151169

152170
def shuffle(self):

‎textattack/goal_functions/classification/classification_goal_function.py‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,8 +14,6 @@
1414
class ClassificationGoalFunction(GoalFunction):
1515
"""A goal function defined on a model that outputs a probability for some
1616
number of classes.
17-
18-
model: The PyTorch or TensorFlow model used for evaluation.
1917
"""
2018

2119
def _process_model_outputs(self, inputs, scores):

‎textattack/goal_functions/goal_function.py‎

Lines changed: 8 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,3 @@
1-
"""
2-
goal_function: Goal functions determine if an attack has been successful.
3-
==============================================================================
4-
"""
5-
61
from abc import ABC, abstractmethod
72

83
import lru
@@ -21,12 +16,14 @@ class GoalFunction(ABC):
2116
specified goal.
2217
2318
Args:
24-
model_wrapper: The model used for evaluation.
25-
maximizable: Whether the goal function is maximizable, as opposed to a boolean result
26-
of success or failure.
27-
query_budget (float): The maximum number of model queries allowed.
28-
model_cache_size (int): The maximum number of items to keep in the model
29-
results cache at once
19+
model_wrapper (:class:`~textattack.models.wrappers.ModelWrapper`):
20+
The victim model to attack.
21+
maximizable(:obj:`bool`, `optional`, defaults to :obj:`False`):
22+
Whether the goal function is maximizable, as opposed to a boolean result of success or failure.
23+
query_budget (:obj:`float`, `optional`, defaults to :obj:`float("in")`):
24+
The maximum number of model queries allowed.
25+
model_cache_size (:obj:`int`, `optional`, defaults to :obj:`2**20`):
26+
The maximum number of items to keep in the model results cache at once.
3027
"""
3128

3229
def __init__(

‎textattack/search_methods/search_method.py‎

Lines changed: 1 addition & 3 deletions

0 commit comments

Comments
 (0)