Modernize parser to use phpstan/phpdoc-parser by johnbillion · Pull Request #247 · WordPress/phpdoc-parser · GitHub
Skip to content

Modernize parser to use phpstan/phpdoc-parser - #247

Draft
johnbillion wants to merge 44 commits into
masterfrom
modernize-parser-php8
Draft

johnbillion wants to merge 44 commits into
masterfrom
modernize-parser-php8

Conversation

@johnbillion

@johnbillion johnbillion commented Jul 13, 2025 •

Copy link
Copy Markdown
Member

Summary

Complete modernization of the WordPress PHPDoc parser from PHP 5.4/legacy libraries to phpstan/phpdoc-parser while maintaining 100% backward compatibility.

  • Complete API compatibility maintained
  • Modern PHP 8 support
  • Increased test coverage

Changes Made

Dependencies

  • PHP: >=5.4 → >=8.1
  • Parser: phpdocumentor/reflection v3 → nikic/php-parser v5 with phpstan/phpdoc-parser v2
  • Testing: PHPUnit v7 → v9

Architecture

  • File reflector: Complete rewrite using PHPParser NodeVisitorAbstract for AST traversal
  • Modern parsing: Replace legacy reflection with AST-based parsing for improved accuracy
  • Backward compatibility: Maintained through runner.php bridge layer

AI disclosure

These changes were generated primarily by an AI coding assistant. I have yet to manually review the changes thoroughly.

Complete rewrite of the parser architecture from legacy phpdocumentor/reflection
to modern PHP libraries while maintaining 100% backward compatibility.

## Major Changes

### Core Dependencies
- PHP requirement: 5.4+ → 8.1+
- Replace phpdocumentor/reflection v3.0 with phpstan/phpdoc-parser v2.0
- Add nikic/php-parser v5.0 for AST-based parsing
- Update PHPUnit: v7 → v9 for WordPress compatibility

### Parser Architecture
- Rewrite File_Reflector to use PHPParser NodeVisitorAbstract
- Implement modern AST traversal for improved accuracy
- Add advanced PHPDoc parsing with type support
- Maintain backward compatibility through runner.php bridge

### Features Added
- Namespace detection and tracking
- Property docblock parsing with visibility
- File-level docblock detection
- Advanced tag parsing (@PARAM, @return with types)
- Class name resolution (self, parent, $this)
- Modern PHP syntax support

### Development Environment
- Update to Node.js 20+ with .nvmrc and .npmrc
- Modernize GitHub Actions for PHP 8.1-8.3 testing
- Add comprehensive documentation and contribution guidelines
- Add Dependabot for automated dependency updates

### Test Results
- All 22 tests passing (100% success rate)
- Full WordPress integration via wp-env
- Complete API compatibility maintained
- Remove Posts-to-Posts from wp-env.json to eliminate plugin conflicts
- Add function existence checks in Relationships class for graceful degradation
- Keep P2P composer dependencies for full functionality when plugin is available
- Add class existence checks in plugin activation hooks

This allows tests to run without P2P while maintaining full functionality
when the WordPress plugin is installed. Core parsing tests all pass.
- Remove scribu/lib-posts-to-posts and scribu/scb-framework from composer.json
- Add class/function existence checks in Relationships class for graceful degradation
- Add P2P class existence check in plugin activation hooks
- Rely solely on WordPress Posts-to-Posts plugin (via wp-env) instead of conflicting Composer packages

This resolves the "function redeclared" errors in development while maintaining full
functionality when the WordPress plugin is available. Tests pass, development environment
works without conflicts, and production compatibility is preserved.
- Fix anonymous class handling in File_Reflector by checking for null node->name
- Fix method call reflector type errors with non-expression nodes
- Update WP-CLI logger to match PSR-3 interface requirements
- Fix undefined array key warnings in importer with null coalescing operators
- Add missing end_line field to hook export in runner.php
- Fix undefined namespace warnings in relationships
- Add wp-cli.yml configuration for development environment connection
- Update README.md with clarified wp-env usage instructions

All PHP warnings eliminated during WordPress core parsing.
Processed 3,338 files successfully: 4,826 functions, 2,112 classes, 14,169 methods, 2,815 hooks.
…atibility

Allows CI environments to generate their own lock files based on the specific PHP version being tested.
- Tests parser import command using wp-env
- Verifies functions, classes, methods, and hooks are imported correctly
- Checks for PHP warnings and errors during parsing
- Validates database integrity and WordPress admin functionality
- Uses subset of WordPress core files for faster CI execution
- Fix npm run wp-env command syntax with proper -- -- usage
- Remove admin area tests, focus on WP-CLI verification only
- Test parser functionality without requiring authentication
- Verify import counts, error detection, and database integrity
Prevents duplicate runs while ensuring tests run on pull requests and master branch pushes
- Add 10-minute timeout to prevent hanging jobs
- Replace hanging 'npm run wp-env logs' with 'docker ps' in cleanup
- Improves CI reliability and debugging capability
Prevents duplicate runs - now only runs on pull requests and master branch pushes
Check plugin status and available commands to diagnose why parser command is not found
The import test was failing because composer dependencies weren't installed in the Docker container, causing the WP-CLI parser command to not register.

Changed from 'npm run wp-env start' to 'npm run setup' which automatically:
1. Starts the WordPress environment
2. Installs composer dependencies in the container

This ensures the parser command is properly registered when the plugin activates.
Removed --quick flag from:
- CI workflow import test
- README examples
- WP-CLI command synopsis documentation

The functionality remains in the codebase for backward compatibility,
but is no longer promoted in documentation or examples.
@johnbillion johnbillion linked an issue Sep 8, 2025 that may be closed by this pull request
@dd32

dd32 commented Sep 9, 2025 •

Copy link
Copy Markdown
Member

@joehoyle

joehoyle commented Dec 5, 2025

Copy link
Copy Markdown

These changes were generated primarily by an AI coding assistant. I have yet to manually review the changes thoroughly.

@johnbillion is this still the case, or are you done with correcting AI! Happy to also give this a review

@johnbillion

Copy link
Copy Markdown
Member Author

This is still the case. It needs a full review.

@ockham

ockham commented Dec 18, 2025 •

Copy link
Copy Markdown

One high-level note, the original structure of the code strongly reflects (no pun intended) phpdocumentor’s - e.g. the File_Reflector class extends phpDocumentor\Reflection\FileReflector, and most of the other reflector classes extend phpDocumentor\Reflection\BaseReflector.

This PR retains that structure, which is fair enough. However, it also means shoehorning PHPStan\PhpDocParser to fit in with what is now kind of a legacy structure. I think that this is particularly visible in the File_Reflector class, which serves as an entry point and now does a lot of the heavy lifting (i.e. parsing at the top level; lots of helper methods such as processXYZ). It’s possible that PHPStan\PhpDocParser might offer a more idiomatic way to do these things, and that the same result could be achieved with “thinner” wrappers around its classes.

(I apologize for the drive-by comment and lofty words. I’m aware that the code here was initially AI-generated, and that AI has an easier time generating code that fits into an existing structure compared to “re-thinking” that structure. However, the ensuing verbosity of the code might’ve played a role in this PR not receiving a lot of reviews yet. I’d also like to acknowledge that a verbose solution is better than none 🙂 and that I probably won’t be able to offer an alternative myself any time soon. If anything, maybe the above suggestions can be used to guide an LLM to a solution with a lower LOC count.)


Edit: The DeepWiki docs (which I realize are also AI-generated) seem quite helpful in understanding the overall architecture of PHPStan\PhpDocParser. (Assuming they're correct 😅)

@johnbillion

Copy link
Copy Markdown
Member Author

Good observation. It definitely felt like getting Claude to reproduce the existing structure with a new parser was going to be simpler than rearchitecting the approach.

@johnbillion johnbillion changed the title Modernize parser to PHP 8.1+ with phpstan/phpdoc-parser Modernize parser to use phpstan/phpdoc-parser Dec 19, 2025
@johnbillion

Copy link
Copy Markdown
Member Author

These changes are still building on top of legacy code such as Posts 2 Posts and its output needs to work with the developer.wordpress.org theme unless you want to also rewrite that at the same time. I think that means that the best we can do is add good test coverage for whatever format is required of developer.wordpress.org and ensure they pass. The wrappers around phpstan/phpdoc-parser and nikic/php-parser and their resulting data structure remain necessary unless we can start from scratch, which is even more impractical than what we're doing here.

@dd32 I have merged your commits into this branch, made some tweaks, and updated the tests. How can we determine what's left to do?

@jonathanbossenger

Copy link
Copy Markdown

I've managed to get a working version of both the original docparser and the updated version (from this PR) so I can run exports to compare the json files. Next step is to review the import process to determine what fields are important, and then compare that to the export. At the same time I'm leaving comments on the PR around onboarding documentation for contributors. I will pick up testing in the new year after I return from my year end leave.

@tmdk

tmdk commented Jan 11, 2026 •

Copy link
Copy Markdown

After some digging, I do not really understand the decision to drop phpDocumentor. The latest version of the phpDocumentor docblock parser even uses the phpstan phpdoc-parser internally for type parsing. Even phpstubs/wordpress-stubs is using phpDocumentor to create phpstan compatible array shape type docs from the legacy WordPress params. Meanwhile, out of the box, the phpstan docblock-parser does really strange things to the legacy WordPress array shape docs.

Seems to me that phpDocumentor provides better backwards compatibility and arguably an easier upgrade path

@tmdk

tmdk commented Jan 21, 2026

Copy link
Copy Markdown

Seems to me that phpDocumentor provides better backwards compatibility and arguably an easier upgrade path

I was wrong.

While I got pretty close to achieving the same output as the old parser, getting to 100% identical output (with some whitespace and type related issues excluded) seems to stay frustratingly out of reach.

Most of the issues are caused by phpDocumentor not considering any nodes that are not the direct child of a function or class method or at the root of the file, and some issues with type name resolution. Since the API of the reflection library is very restrictive and locked down by design, workarounds for these issues are simply not possible or require ugly hacks or lots of duplicated code.

It might still be a good idea to use the phpdocumentor/reflection-docblock library for docblock parsing, because it handles the WordPress-style array shape param tags better than the phpdoc docblock parser alone

@sirreal

sirreal commented Feb 13, 2026

Copy link
Copy Markdown
Member

bordoni pushed a commit to bordoni/phpdoc-parser that referenced this pull request May 31, 2026
- Fully-qualify class-name argument typehints (\WP_Post), leaving
  built-in Identifier types (int, string, array) bare. Argument types
  previously dropped the leading backslash the legacy parser emitted.
- Strip reflection-docblock's FQSEN normalization backslash from @see
  references so they read as written, matching the legacy output.

Both are the leading-backslash discrepancy dd32 flagged on upstream
PR WordPress#247 (present in our parser too); verified against the legacy oracle.
bordoni pushed a commit to bordoni/phpdoc-parser that referenced this pull request May 31, 2026
- Add type-tags.inc fixture + legacy golden: the WordPress @type hash
  @PARAM stays inline in content and is not extracted, per dd32 and
  johnbillion (the wporg-developer theme depends on this).
- Extend docblocks.inc (additive) with @see references, a typed-hash
  method, and a markdown-heavy description; regenerate the legacy oracle.
- Unit tests lock the modern @PARAM syntaxes the old parser mangled
  (?type, parenthesized unions) and modern code typehints (?WP_Post,
  union, return types) that php-parser v1 could not parse at all.
bordoni added a commit to bordoni/phpdoc-parser that referenced this pull request May 31, 2026
- Fully-qualify class-name argument typehints (\WP_Post), leaving
  built-in Identifier types (int, string, array) bare. Argument types
  previously dropped the leading backslash the legacy parser emitted.
- Strip reflection-docblock's FQSEN normalization backslash from @see
  references so they read as written, matching the legacy output.

Both are the leading-backslash discrepancy dd32 flagged on upstream
PR WordPress#247 (present in our parser too); verified against the legacy oracle.
bordoni added a commit to bordoni/phpdoc-parser that referenced this pull request May 31, 2026
- Add type-tags.inc fixture + legacy golden: the WordPress @type hash
  @PARAM stays inline in content and is not extracted, per dd32 and
  johnbillion (the wporg-developer theme depends on this).
- Extend docblocks.inc (additive) with @see references, a typed-hash
  method, and a markdown-heavy description; regenerate the legacy oracle.
- Unit tests lock the modern @PARAM syntaxes the old parser mangled
  (?type, parenthesized unions) and modern code typehints (?WP_Post,
  union, return types) that php-parser v1 could not parse at all.
bordoni added a commit to bordoni/phpdoc-parser that referenced this pull request May 31, 2026
- Extract file-level constants (define() calls anywhere + the const
  keyword) and include/require statements with their legacy type labels
  ("Include", "Require Once", ...), via new Include_Reflector and
  Constant_Reflector wrappers. File_Reflector previously returned [] for
  both, silently dropping these from the export contract.
- Fix the file-docblock heuristic: a docblock attached to the open tag
  floats to the file only when the first statement does not claim it.
  Hooks, define(), and include/require claim it (as the legacy parser
  does); only plain calls and assignments leave it for the file. The old
  check missed bare hooks/define()/require, mis-attributing their
  docblock to the file on real wp-load.php-style files.
- Lock both with a golden fixture (constants-includes.inc, minted from
  the legacy parser on PHP 7.4) and unit tests.

Found while mining upstream PR WordPress#247 for export-contract gaps.
bordoni added a commit to bordoni/phpdoc-parser that referenced this pull request May 31, 2026
Mining upstream PR WordPress#247: probed two contract corners our corpus never
exercised; both already match the legacy parser byte-for-byte.

- hooks-extra: all 6 hook variants (action/filter x ref_array/deprecated),
  argument shapes, and hook-name normalization (concat -> interpolation).
- class-features: abstract/final, extends + multiple implements, method
  aliases, multi-property declarations, and trait/interface exclusion.
bordoni added a commit to bordoni/phpdoc-parser that referenced this pull request May 31, 2026
…parser

Mining upstream PR WordPress#247 surfaced four name-resolution gaps in namespaced
code (which in wp-includes means the bundled SimplePie/Requests/PHPMailer
libraries dd32 diffed):

- Method namespace now reports its enclosing namespace (My\Plugin), not ''
  — '' is correct only for the global namespace.
- Exported namespace aliases on functions and methods are fully-qualified
  ("\Other\Thing"), matching the legacy output (dd32's leading-backslash note).
- Function-use names resolve like the legacy parser: unqualified global-
  fallback calls stay bare (count), while fully-qualified, qualified, and
  use-function-imported calls become a leading-backslash FQN (\do_action,
  \Other\helper, \My\Plugin\Sub\thing). The previous code read the wrong
  resolver attribute and left them all bare.
- A fully-qualified \do_action is a plain function call, not a hook.

Locked with a golden fixture (namespaced-uses.inc, minted from the legacy
parser on PHP 7.4) and a unit test.
bordoni added a commit to bordoni/phpdoc-parser that referenced this pull request May 31, 2026
Mining upstream PR WordPress#247: calls inside closures (anonymous functions used
as hook callbacks, assigned to variables, or nested in functions) attribute
to the enclosing named scope (file or function), not the closure — already
matching the legacy parser byte-for-byte.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Update dependencies for phpdocumentor/reflection

7 participants