dmca_tri

Downloads every PDF attachment from a public Regulations.gov docket and converts PDFs with embedded text into UTF-8 text files. Reruns reuse previously downloaded PDFs.

Petition library: https://margaret-tucker.github.io/dmca_tri/

Setup

cd ~/dmca_tri
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt

Run

python download_pdfs.py COLC-2026-0100

Results are written to output/pdfs, output/text, and output/manifest.jsonl. Use --resource both to include docket documents as well as public-comment attachments.

The shared DEMO_KEY works for light use. For a full crawl, request an API key from Regulations.gov and set it without putting it in source control:

export REGULATIONS_GOV_API_KEY="your-key"
python download_pdfs.py COLC-2026-0100 --resource both

For scanned PDFs, install OCRmyPDF (brew install ocrmypdf on macOS) and add --ocr. The original PDF is preserved; searchable OCR PDFs go under output/ocr.

If the download host returns HTTP 403, the Regulations.gov CloudFront distribution is blocking the current network path. The API may still list files normally; retry later or from another network. Already completed files are retained for the next run.

For public PDFs, the existing manifest can instead be converted to Markdown through Jina Reader without repeating the API crawl:

python convert_with_reader.py

This sends each public PDF URL to the third-party Jina Reader service. Do not use this fallback for private or sensitive documents.

Petition library

Run the local browser application:

python app.py

Open http://127.0.0.1:8000. The library supports full-text search, New and Renewal filters, petition summaries, extracted text, and inline access to each source PDF.

To make the app available to coworkers on the same trusted network, bind it to all interfaces:

python app.py --host 0.0.0.0

GitHub Pages

Build the same application as a static site:

python build_site.py
python -m http.server 8001 --directory _site

The GitHub Actions workflow rebuilds and deploys the site whenever main is pushed. In the repository settings, set Pages > Build and deployment > Source to GitHub Actions. The generated _site directory is ignored because the workflow recreates it from the reviewed source text and PDFs.