This is a home for thoughts and results on table performance. It is intended to capture ideas quickly without being polished. Eventually some of these will turn into code and/or performance tips.

Performance-related PRs:

• Row #8494
• Table slicing #8493
• Table meta #8404

This is addressed to some extent in #8404

More than half the time required to slice a table is spent in DataInfo just getting / setting attributes. It is being fancy about managing attributes (e.g. allowing for attributes to live on parent, checking for allowed attributes), but this is all time consuming.

MHvK: one option to speed it up is to remove __getattr__; instead, one would have properties for attributes that live on the parent. These could be auto-generated from _parent_attrs using a metaclass.

Astropy 3.1

>>> tb = t['b']
>>> timeit tb.info.name
4.16 µs ± 119 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

>>> timeit tb.name
155 ns ± 2.14 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

Astropy Table uses the strategy of slicing each column individually and making a new table. This makes table slicing time scale with the number of columns.

>>> t = simple_table(cols=2)
>>> %timeit t[1:2]
1.59 ms ± 48.6 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

>>> t = simple_table(cols=2)
>>> %timeit t[1:2]
156 µs ± 2.2 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

The time to slice a pandas DataFrame is roughly independent of the number of columns, about 80-90 usec. They are presumably using some sort of stride / slice object which references the original DataFrame.

Using direct column attribute access instead of info makes a huge difference, see #8493

__array_finalize__ and _copy_attrs are the losers here.

Astropy 3.1

>>> %timeit t['b'][1:2]
26.1 µs ± 401 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

# Viewing as ndarray speeds it up alot.  No meta
>>> %timeit t['b'].data[1:2]
1.48 µs ± 8.8 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

Though pandas column slice is also slow (compared to numpy), about the 90 usec (similar to time for slicing a 25-column DataFrame).

See #8494

A limiting factor in creating a Row is determining the table length, which in turn comes down to getting the length for at least one column. It turns out that OrderedDict is very slow for returning a values() dictview compared to dict:

In [46]: timeit t.columns.values()
1.3 µs ± 28.3 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

In [47]: t_asdict = dict(t)

In [48]: timeit t_asdict.values()
90.7 ns ± 0.795 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

In [51]: len(t.columns)
Out[51]: 25