Add Submodules Fuzz Target by DaveLak · Pull Request #1919 · gitpython-developers/GitPython · GitHub
Skip to content

Add Submodules Fuzz Target#1919

Merged
Byron merged 1 commit into
gitpython-developers:mainfrom
DaveLak:improve-fuzzer-coverage
May 30, 2024
Merged

Add Submodules Fuzz Target#1919
Byron merged 1 commit into
gitpython-developers:mainfrom
DaveLak:improve-fuzzer-coverage

Conversation

@DaveLak

@DaveLak DaveLak commented May 30, 2024

Copy link
Copy Markdown
Contributor

Fuzz Introspector heuristics suggest the Submodule API code represent "optimal analysis targets" that should yield a meaningful increase in code coverage. The changes here introduce a first pass at implementing a fuzz harness that cover the primary APIs/methods related to Submodules. Of particular interest to me is the Submodule.config_writer() coverage.

Please note however, there is likely plenty of room for improvement in this harness in terms of both code coverage as well as performance; the latter of which will see significant benefit from a well curated seed corpus of .gitmodules file like inputs. The ParsingError raised by the fuzzer without a good seed corpus hinders test efficacy significantly.

I have a draft PR up with a seed corpus here: gitpython-developers/qa-assets#5

Fuzz Introspector heuristics suggest the Submodule API code represent
"optimal analysis targets" that should yield a meaningful increase in
code coverage. The changes here introduce a first pass at implementing a
fuzz harness that cover the primary APIs/methods related to Submodules.
Of particular interest to me is the `Submodule.config_writer()`
coverage.

Please note however, there is likely plenty of room for improvement in
this harness in terms of both code coverage as well as performance; the
latter of which will see significant benefit from a well curated seed
corpus of `.gitmodules` file like inputs. The `ParsingError` raised by
the fuzzer without a good seed corpus hinders test efficacy
significantly.

@Byron Byron left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for taking such great care!

Part of me thinks that the submodule implementation is so riddled with inaccuracies and and incorrectness that fuzzing it seems like a waste. The fuzzer can only try to find unexpected exceptions, and maybe that's a small win, but at what cost?

Part of that feeling also stems for the incredible sluggishness of Python in general, so any fuzzing feels wasteful. But that's besides the point I suppose, apologies for the ramblings.

@Byron Byron merged commit 2345c1a into gitpython-developers:main May 30, 2024
@DaveLak DaveLak deleted the improve-fuzzer-coverage branch May 30, 2024 14:33
@DaveLak

DaveLak commented May 30, 2024

Copy link
Copy Markdown
Contributor Author

@Byron

Byron commented May 31, 2024

Copy link
Copy Markdown
Member

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants