Welcome to fasttransform – fasttransform

Welcome to fasttransform

The main building block of data pipelines in fastai. And elsewhere if you like.

fasttransform provides reusable data transformations and pipelines. It is the main building block of fastai’s data pipelines and can also be used independently. A Transform combines a function with optional inverse, setup, and type-handling behaviour. A Pipeline composes transforms.

Installation

Install latest from the GitHub repository:

$ pip install git+https://github.com/AnswerDotAI/fasttransform.git

or from pypi:

$ pip install fasttransform

Quick start

Transform

Create a Transform by passing a function to its constructor or using it as a decorator. The function becomes the transform’s encodes method.

A transform supports:

  • Reversibility: keep a function and its inverse in one object.
  • Setup: configure a transform instance using the dataset.
  • Type-based multiple dispatch: select a function based on argument types.
  • Type conversion and preservation: control the result’s type, including subclasses.

To create a transform with a decorator:

from fasttransform import Transform, Pipeline
@Transform
def add_one(x): 
    return x + 1

# Usage
add_one(2)
3

Reversibility

Pass a function and its inverse to make a transform reversible. Use this to normalize and de-normalize numerical values, or to encode categories as indices and decode them again:

def enc(x): return x*2
def dec(x): return x//2

t = Transform(enc,dec)

t(2), t.decode(2), t.decode(t(2))
(4, 1, 2)

Setup

A transform’s setups method can calculate properties from a dataset. This z-score normalization transform stores the mean and standard deviation. Its encodes and decodes methods use those values:

import statistics

class NormalizeMean(Transform):
    def setups(self, items): 
        self.mean = statistics.mean(items)
        self.std  = statistics.stdev(items)
    
    def encodes(self, x): 
        return (x - self.mean) / self.std
    
    def decodes(self, x): 
        return x * self.std + self.mean

normalize = NormalizeMean()
normalize.setup([1, 2, 3, 4, 5])
normalize.mean
3

Type-based multiple dispatch

Pass multiple functions with different parameter annotations to select behaviour by input type. This is useful for handling different image formats or numerical types in one transform.

This transform selects a function for an int or a str:

def inc1(x:int): return x+1
def inc2(x:str): return x+"a"

t = Transform(enc=(inc1,inc2))

t(5), t('b')
(6, 'ba')

When no type annotation matches an input, the transform returns that input unchanged.

add_one(2.0)
3.0
normalize(3.0)
0.0

Type conversion and preservation

Transform uses the wrapped function’s return type to control conversion in encodes and decodes. The return type can be explicit or implicit. The rules are:

  1. The result has the function’s return type, with conversion when needed.
  2. When the input’s runtime type is a subtype of that return type, the result preserves the input’s type.
  3. A return annotation of None disables type conversion and preservation.

Return type

FS is a subclass of float. Normal Python multiplication of an FS and a float returns a float:

class FS(float):
    def __repr__(self): return f'FS({float(self)})'
 

f1 = float(1)
FS2 = FS(2)

val = f1 * FS2
type(val) # => float
float

With Transform, an FS return annotation makes the multiplication return an FS:

def double_FS(x)->FS: return FS(2)*x
t = Transform(double_FS)
val = t(1) 
assert isinstance(val,FS)
val
FS(2.0)

Type preservation

Without a return annotation, this multiplication transform preserves the input’s runtime type. Passing an FS returns an FS. The wrapped function alone would return a float:

def double(x): return x*2.0  # no type annotation
t = Transform(double)
fs1 = FS(1)
val = t(fs1)
assert isinstance(val,FS)
val # => FS(2), an FS value of 2
FS(2.0)

Disabling conversion

Use a return annotation of None to disable type conversion and preservation:

def double_none(x) -> None: return x*2.0  # "None" returnt type means "no conversion"
t = Transform(double_none)
fs1 = FS(1)
val = t(fs1)
assert isinstance(val,float)
val # => 2.0, a float of 2, because of fallback to standard Python type logic
2.0

Pipelines

A Pipeline applies transforms in sequence. This pipeline doubles a value and then normalizes it. decode reverses the transformations:

def double(x): return x*2.0 
def halve(x): return x/2.0
dt = Transform(double,halve)

class NormalizeMean(Transform):
    def setups(self, items): 
        self.mean = statistics.mean(items)
        self.std  = statistics.stdev(items)
    
    def encodes(self, x):
        return (x - self.mean) / self.std
    
    def decodes(self, x):
        return x * self.std + self.mean

normalize = NormalizeMean()
normalize.setup([1, 2, 3, 4, 5])

p = Pipeline((dt, normalize))

v = p(5)
v
4.427188724235731
p.decode(v)
5.0

Documentation

See the documentation for the full API.