Add a lexer for FASTA by Madin-H23 · Pull Request #3420 · pygments/pygments · GitHub
Skip to content

Add a lexer for FASTA - #3420

Open
Madin-H23 wants to merge 1 commit into
pygments:masterfrom
Madin-H23:add/fasta-lexer
Open

Madin-H23 wants to merge 1 commit into
pygments:masterfrom
Madin-H23:add/fasta-lexer

Conversation

@Madin-H23

@Madin-H23 Madin-H23 commented Oct 20, 2026 •

Copy link
Copy Markdown

What

Fixes #3019 — adds a lexer for FASTA sequence files, which Pygments had no support for at all (there is no bio/fasta module in the tree).

A FASTA file holds one or more sequences. Every sequence starts with a header line beginning with > that carries the sequence identifier and an optional free-form description; the sequence itself follows on the next lines, wrapped at an arbitrary width. The new FastaLexer marks:

  • the > marker as punctuation, the sequence identifier as a tag, and the description as a string
  • the sequence data as constants, with gaps (- and .) in a low-contrast comment color, digits as integers, and the * stop-codon marker as a keyword

It is registered under the alias fasta and the usual extensions (*.fa, *.fasta, *.fas, *.faa, *.fna, *.ffn, *.frn, *.pep, *.seq). analyse_text recognizes a FASTA file by its header line followed by sequence-only lines, so guess_lexer works on content as well as on file names.

A note on the colouring

The original request also asked for per-residue palettes — one for amino acids and one for nucleotides. I did not implement that part, and I would rather explain why than guess: a per-residue palette needs roughly 24 residue-specific token types, and token types that are not part of the standard set are not styled by the common themes, so in practice they would all render as plain text rather than as a palette.

Correction to the above, after measuring instead of assuming: that statement holds for custom token types, but I had not checked the other route — reusing existing token types as palette slots. Reading the token→style mapping of the stock themes (pygments 2.21.0, 30 candidate existing types) yields 26 distinct colours under default and friendly, 24 under emacs, 23 under pastie, 15 under vim, 11 under solarized-dark, 9 under monokai and 4 under bw — so 20 standard residues do fit inside the palette the common themes hand out. The price of that route is that it borrows token types whose meaning has nothing to do with residues, and it flattens on themes with a smaller palette.

That leaves three shapes: a grouped/semantic palette (what this change does), per-residue custom token types (clean taxonomy, but invisible until someone ships a style), and per-residue reuse of existing token types (per-residue identity under stock themes, at the price of the borrowed taxonomy). @Artoria2e5 has asked for the per-residue variant, so I am glad to rework this change into whichever of the two per-residue shapes is preferred — I did not want to swap the implementation silently, since the taxonomy trade-off is a maintainer-level call. If the maintainers want that, the sensible route is to agree on a small set of standard token types to reuse for the residue groups (and, if needed, to teach the popular themes about them), and I would happily add that on top of this change.

Tests

tests/examplefiles/fasta/fasta.fasta (+ generated golden) covers several records, a long wrapped sequence, a header with pipes and a description, gaps, lowercase residues, ambiguity codes and a stop codon.

Verified locally:

  • pytest -q --ignore=tests/contrast — 5335 passed, 16 skipped
  • pytest tests/examplefiles -k fasta — 1 passed
  • ruff check --ignore UP031 ., scripts/check_crlf.py, scripts/detect_missing_analyse_text.py --skip-no-aliases, scripts/check_sources.py, scripts/count_token_references.py — all clean
  • the token stream round-trips to the input exactly, with no Error tokens, including for CRLF files, files with a lone CR, an empty file, a header without a sequence, and several records in one file

AI usage disclosure: the lexer and its tests were prepared with AI assistance; all test results above were produced locally and reviewed.

FASTA sequence files had no lexer at all.  The header line is marked with
the sequence identifier as a tag and the free-form description as a string,
the sequence data as constants, gaps in a low-contrast comment color, and
the stop codon marker as a keyword.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Lexer request: FASTA (aa and nt)

1 participant