test: add oss-scanner support · coveragepy/coveragepy@202e161 · GitHub
Skip to content

Commit 202e161

Browse files
nedbatnedbat
committed
test: add oss-scanner support
1 parent 55014fe commit 202e161

9 files changed

Lines changed: 217 additions & 1 deletion

File tree

‎MANIFEST.in‎

Lines changed: 1 addition & 0 deletions

‎tests/oss-scanner/Dockerfile‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
# Build image for the OSS Scanner: https://github.com/anthropics/oss-scanner
2+
# The scanner builds this with the repository as the build context. Network
3+
# access is only available while building this image, so everything needed to
4+
# build and test coverage.py has to be installed here.
5+
6+
# PYVERSIONS: This runs on 3.13. Update throughout this file as needed.
7+
8+
FROM python:3.13-bookworm
9+
10+
ENV DEBIAN_FRONTEND=noninteractive \
11+
PIP_DISABLE_PIP_VERSION_CHECK=1 \
12+
PYTHONWARNDEFAULTENCODING=1 \
13+
PYTHON_COLORS=0
14+
15+
# The scanner runs the image with no network.
16+
ENV COVERAGE_TEST_NO_NETWORK=1
17+
18+
RUN apt-get update \
19+
&& apt-get install -y --no-install-recommends build-essential gdb git \
20+
&& rm -rf /var/lib/apt/lists/*
21+
22+
COPY . /src
23+
WORKDIR /src
24+
25+
RUN python -m pip install -r requirements/tox.txt
26+
27+
# Debug info and frame pointers make C tracer crashes easier to diagnose.
28+
ENV CFLAGS="-g -O1 -fno-omit-frame-pointer"
29+
30+
# Without build isolation, tox can run again later with no network.
31+
ENV COVERAGE_PIP_ARGS=--no-build-isolation
32+
33+
# Run the full test suite as tox.ini defines it, but don't fail the build on
34+
# test failures.
35+
RUN (tox -e py313 || echo "WARNING: tests reported failures")
36+
37+
# tox removes the C extension to test the other cores, so build it again.
38+
RUN .tox/py313/bin/python setup.py --quiet build_ext --inplace \
39+
&& .tox/py313/bin/python -c "import coverage.tracer"
40+
41+
ENV PATH="/src/.tox/py313/bin:$PATH"

‎tests/oss-scanner/README.rst‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
.. Licensed under the Apache License: http://www.apache.org/licenses/LICENSE-2.0
2+
.. For details: https://github.com/coveragepy/coveragepy/blob/main/NOTICE.txt
3+
4+
OSS Scanner
5+
===========
6+
7+
These files enroll coverage.py in Anthropic's `OSS Scanner`_. project.yaml is
8+
a copy of the file in that repo; Dockerfile and threat_model.md are read from
9+
here.
10+
11+
To check that the Dockerfile still works, build the image and run the tests in
12+
it with no network, as the scanner will::
13+
14+
$ source tests/oss-scanner/local_run.sh
15+
16+
This builds from the last commit, so commit your changes first. Tests that
17+
need the network or a non-root user are marked xfail.
18+
19+
.. _OSS Scanner: https://github.com/anthropics/oss-scanner

‎tests/oss-scanner/local_run.sh‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
# Licensed under the Apache License: http://www.apache.org/licenses/LICENSE-2.0
2+
# For details: https://github.com/coveragepy/coveragepy/blob/main/NOTICE.txt
3+
4+
# Build the OSS Scanner image from the last commit, then run the tests in it
5+
# with no network, as the scanner will.
6+
#
7+
# This is meant to be sourced, so it avoids `set -e` and `exit`.
8+
# git archive packs only committed files, like the scanner's clean checkout.
9+
# In a subdirectory it would pack only that subdirectory, so use the top.
10+
11+
# Uncommitted changes wouldn't be tested, which is easy to forget.
12+
if [ -n "$(git status --porcelain)" ]; then
13+
echo "Commit these changes first, they won't be in the image:"
14+
git status --short
15+
else
16+
git -C "$(git rev-parse --show-toplevel)" archive HEAD \
17+
| DOCKER_CLI_HINTS=false docker build -f tests/oss-scanner/Dockerfile -t coveragepy-oss-scanner - \
18+
&& docker run --rm --network none coveragepy-oss-scanner tox -e py313
19+
fi

‎tests/oss-scanner/project.yaml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
# Enrollment for the OSS Scanner: https://github.com/anthropics/oss-scanner
2+
# This file belongs in that repository at projects/coveragepy/project.yaml.
3+
# It is kept here as the source of truth.
4+
repo: https://github.com/coveragepy/coveragepy
5+
primary_contact: ned@nedbatchelder.com
6+
homepage: https://coverage.readthedocs.io
7+
dockerfile: tests/oss-scanner/Dockerfile
8+
threat_model: tests/oss-scanner/threat_model.md

‎tests/oss-scanner/threat_model.md‎

Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
# Threat model for coverage.py
2+
3+
## What this project does and where untrusted input enters
4+
5+
coverage.py measures code coverage of Python programs. It runs (or is started
6+
inside) a Python process, records which lines and branches execute, stores the
7+
results in a SQLite data file (`.coverage`), and produces reports in text,
8+
HTML, XML (Cobertura), JSON, LCOV, and annotated-source formats.
9+
10+
coverage.py runs with the full privileges of the program being measured. By
11+
design, the following are **trusted** and are not security boundaries:
12+
13+
- The Python code being measured. coverage.py executes it (`coverage run`) and
14+
imports it.
15+
- Configuration files (`.coveragerc`, `setup.cfg`, `tox.ini`,
16+
`pyproject.toml`) and command-line arguments.
17+
- Plugins named in configuration. They are imported and run as ordinary
18+
Python code.
19+
- Environment variables (`COVERAGE_*`, etc.).
20+
- The `.pth` file coverage.py may install to measure subprocesses.
21+
22+
Inputs that can reasonably be **less trusted** than the person running
23+
coverage:
24+
25+
- **Source file contents and file names.** Reports are often generated in CI
26+
for pull requests from outside contributors and then published (for example
27+
HTML reports hosted on a web site, or XML/LCOV consumed by other services).
28+
Source text, file paths, function and class names, and context labels flow
29+
into report output. These must not be able to inject markup or script into
30+
HTML reports, break the structure of XML/JSON/LCOV output, or inject terminal
31+
control sequences into text output.
32+
- **Data files from other machines.** `coverage combine` reads `.coverage.*`
33+
SQLite files that are often collected from many CI jobs or downloaded as
34+
artifacts. A malformed or malicious data file should produce an error, not
35+
memory corruption, code execution, writing outside of the expected
36+
directories, or unbounded resource use beyond the size of the input.
37+
- **Path remapping during combine and reporting** (`[paths]` settings) applied
38+
to file names recorded in data files.
39+
40+
## Components that matter most / least
41+
42+
Most important:
43+
44+
- `coverage/ctracer/` — the C extension tracer (`coverage.tracer`). It is
45+
called on every line event of the measured program. Memory-safety bugs here
46+
(reference-counting errors, use-after-free, buffer overflows) are the highest
47+
priority.
48+
- `coverage/html.py`, `coverage/templite.py`, and `coverage/htmlfiles/` — HTML
49+
report generation and escaping.
50+
- `coverage/xmlreport.py`, `coverage/jsonreport.py`, `coverage/lcovreport.py`,
51+
`coverage/report.py`, `coverage/annotate.py` — other report writers.
52+
- `coverage/sqldata.py`, `coverage/sqlitedb.py`, `coverage/numbits.py`,
53+
`coverage/data.py` — reading, writing, and combining data files.
54+
- `coverage/files.py` — path handling, globbing, and remapping.
55+
- `coverage/phystokens.py`, `coverage/parser.py` — tokenizing and parsing
56+
source files for reports.
57+
58+
Less important:
59+
60+
- `coverage/pytracer.py` and `coverage/sysmon.py` — pure-Python tracers; bugs
61+
here are usually correctness issues, not security issues.
62+
- `coverage/execfile.py`, `coverage/plugin_support.py`, `coverage/config.py`,
63+
`coverage/tomlconfig.py` — they process trusted input by design.
64+
65+
Out of scope:
66+
67+
- `lab/`, `ci/`, `doc/`, `igor.py`, and `tests/` — developer tooling, not
68+
shipped in the package.
69+
- The vendored JavaScript and CSS in `coverage/htmlfiles/` beyond how
70+
coverage.py fills in the data it renders.
71+
72+
## How to exercise it
73+
74+
- The checkout is in `/src`. The project is installed editable in the tox
75+
environment `/src/.tox/py313`, which is first on `PATH`, with the C
76+
extension built in place. `coverage` is the command-line entry point.
77+
- `tox -e py313` runs the full test suite with each tracer: the C tracer
78+
(`ctrace`), the Python tracer (`pytrace`), and `sys.monitoring` (`sysmon`).
79+
It removes the C extension to test the other tracers; rebuild it with
80+
`python setup.py build_ext --inplace`.
81+
- `python igor.py test_with_core ctrace` runs the tests with just the C
82+
tracer. Standard pytest arguments can be added, for example `-k html`.
83+
- Typical flow to produce reports from a program:
84+
`coverage run --branch prog.py && coverage html && coverage xml && coverage json && coverage lcov`.
85+
- To exercise combining: create several data files with `coverage run -p` and
86+
run `coverage combine`.
87+
- Tests that need network access or a non-root user are skipped or marked
88+
xfail.
89+
90+
## How you rate severity
91+
92+
- Memory corruption in the C tracer that a measured program can trigger
93+
through ordinary Python code: high. (The measured program is trusted, but
94+
coverage.py should never crash it.)
95+
- Script or markup injection into HTML reports from source contents, file
96+
names, or data files: high.
97+
- Code execution, or reading/writing files outside the expected locations,
98+
triggered by a crafted data file during combine or reporting: high.
99+
- Corrupting the structure of XML, JSON, or LCOV output, or terminal control
100+
sequence injection in text output: medium.
101+
- Crashes or excessive resource use from malformed data files: low.
102+
- Anything that requires control of configuration, plugins, environment
103+
variables, or the measured code itself to cause effects that code could
104+
already cause directly: not a vulnerability.
105+
106+
## Anything to leave alone
107+
108+
- Use of `exec`, `compile`, and imports to run the measured program and
109+
plugins is intentional.
110+
- Non-cryptographic hashing (for example for file fingerprints) is
111+
intentional and is marked `usedforsecurity=False`.
112+
- SQL in `sqldata.py` is built from internal constants; table and column names
113+
are not user-controlled.

‎tests/test_api.py‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -482,7 +482,14 @@ def make_good_data_files(self) -> None:
482482

483483
@pytest.mark.parametrize(
484484
"problem, msgpattern",
485-
[("badcontent", BAD_SQLITE_REGEX), ("noperms", NO_OPEN_REGEX)],
485+
[
486+
("badcontent", BAD_SQLITE_REGEX),
487+
pytest.param(
488+
"noperms",
489+
NO_OPEN_REGEX,
490+
marks=pytest.mark.xfail(testenv.ROOT_USER, reason="root can read any file"),
491+
),
492+
],
486493
)
487494
def test_combining_bad_data(self, problem: str, msgpattern: str) -> None:
488495
# If you try to combine a bad data file, then you will get a warning,

‎tests/test_venv.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -203,6 +203,7 @@ def coverage_command_fixture(request: pytest.FixtureRequest) -> str:
203203
bool(os.getenv("COVERAGE_PIP_ARGS")),
204204
reason="COVERAGE_PIP_ARGS setting means we shouldn't try pip",
205205
)
206+
@pytest.mark.xfail(testenv.NO_NETWORK, reason="Installing packages needs the network")
206207
class VirtualenvTest(CoverageTest):
207208
"""Tests of virtualenv considerations."""
208209

‎tests/testenv.py‎

Lines changed: 7 additions & 0 deletions

0 commit comments

Comments
 (0)