text/docs/api_docs/python/text/Trimmer.md at master · tensorflow/text · GitHub
Skip to content

Latest commit

 

History

History
121 lines (94 loc) · 3.13 KB

File metadata and controls

121 lines (94 loc) · 3.13 KB

description: Truncates a list of segments using a pre-determined truncation strategy.

text.Trimmer

View source

Truncates a list of segments using a pre-determined truncation strategy.

Methods

generate_mask

View source

@abc.abstractmethod
generate_mask(
    segments
)

Generates a boolean mask specifying which portions of segments to drop.

Users should be able to use the results of generate_mask() to drop items in segments using tf.ragged.boolean_mask(seg, mask).

Args
`segments` A list of `RaggedTensor` each w/ a shape of [num_batch, (num_items)].
Returns
a list with len(segments) number of items and where each item is a `RaggedTensor` with the same shape as its counterpart in `segments` and with a boolean dtype where each value is True if the corresponding value in `segments` should be kept and False if it should be dropped instead.

trim

View source

trim(
    segments
)

Truncate the list of segments.

Truncate the list of segments using the truncation strategy defined by generate_mask.

Args
`segments` A list of `RaggedTensor`s w/ shape [num_batch, (num_items)].
Returns
a list of `RaggedTensor`s with len(segments) number of items and where each item has the same shape as its counterpart in `segments` and with unwanted values dropped. The values are dropped according to the `TruncationStrategy` defined.