Python version upgrades

This doc contains instructions for how to upgrade pytype to support a new Python version, using the Python 3.6->3.7 upgrade as an example.

pytype

The process is as follows:

1. Build pytype in the new version, so that you can try out your changes. Have a test file, such as pytype/test_data/simple.py, handy. For the open-source build, first update setup.cfg (only locally for now!), then install from source using the new version’s pip.
2. Make the minimal changes for pytype to start analyzing code in the new version.
3. Run the regression tests and fix failures.
4. Update setup.cfg to declare support for the new version. We can start offering type-checking for backwards-compatible code, even before any new features have been implemented.
5. Implement new features.

minimal changes

version validation

The pytype.utils.validate_version method should be updated to accept the new version.

opcode changes

If the new version adds or removes any opcodes, then updated opcodes should be added to pytype/pyc/opcodes.py, an opcode mapping to pycnite/mapping.py and new opcodes implemented in pytype/vm.py.

TIP: pytype/pyc/generate_opcode_diffs.py will generate the changes you need to make to opcodes.py, as well as stub implementations for vm.py.

If the above script doesn’t work, you can figure out the necessary changes by playing around with the opcode library:

• opcode.opmap contains a name->index mapping of all opcodes.
• Opcodes with an index of at least opcode.HAVE_ARGUMENT have an argument.
• The opcode.has{property} attributes correspond to HAS_{PROPERTY} flags in pytype’s opcodes module.

The documentation for the dis library is a good reference for bytecode changes. dis is also handy for disassembling a piece of code and comparing the bytecode between two versions. For example, to disassemble a function foo.f:

import dis
import foo
dis.dis(foo.f)  # pretty-prints the bytecode to stdout

Finally, if all else fails, you can consult the CPython source code.

magic numbers

Magic numbers for the new version should be copied from the CPython source code to pycnite/magic.py.

stubs

We maintain custom pytd stubs for some modules in pytype/stubs/{builtins,stdlib}. Compare them to the typeshed versions and make sure all types for the new Python version are present in our stubs.

regression tests

For the open-source project, navigate to the root of your cloned pytype repository and run using the new Python version:

python build_scripts/run_tests.py

The new version should also be added to the CI test matrix. Even if the tests do not pass yet, it is helpful to see the failures. You can can configure them to not fail the workflow by marking the check as experimental and using || exit 0 to always report a success.

GitHub release

Update the classifiers and install_requires fields in setup.cfg to include the new version.

new features

Changes may be needed to pytype to support new features that affect typing. For 3.7, such features included postponed evaluation of type annotations, dataclasses, and typing.OrderedDict. New features can be found on the “What’s New in Python 3.x” page and by searching for “New in version 3.x” in the typing module documentation.