Improve extract performance via ignoring directories early during os.walk by akx · Pull Request #968 · python-babel/babel · GitHub
Skip to content

Improve extract performance via ignoring directories early during os.walk#968

Merged
akx merged 1 commit into
masterfrom
p694-redux
Dec 29, 2025
Merged

Improve extract performance via ignoring directories early during os.walk#968
akx merged 1 commit into
masterfrom
p694-redux

Conversation

@akx

@akx akx commented Feb 7, 2023

Copy link
Copy Markdown
Member

Closes #694 (rebases and reimplements it)

@codecov

codecov Bot commented Feb 7, 2023 •
edited
Loading

Copy link
Copy Markdown

@akx akx marked this pull request as ready for review February 16, 2023 19:10
@akx

akx commented Feb 20, 2023

Copy link
Copy Markdown
Member Author

@DenverCoder1 If you have a moment, could you review this? :)

@DenverCoder1

DenverCoder1 commented Feb 24, 2023

Copy link
Copy Markdown
Contributor

Looks good to me (I haven't tested)

I noticed a similar method in frontend.py -

babel/babel/messages/frontend.py

Lines 277 to 288 in 373a52f

def _make_directory_filter(ignore_patterns):
"""
Build a directory_filter function based on a list of ignore patterns.
"""
def cli_directory_filter(dirname):
basename = os.path.basename(dirname)
return not any(
fnmatch.fnmatch(basename, ignore_pattern)
for ignore_pattern
in ignore_patterns
)
return cli_directory_filter
- it seems to not need to create relative paths there, but maybe it's different. Possibly some logic could be combined or renamed to be consistent, but besides that, it seems like this should work.

@akx akx marked this pull request as draft February 28, 2023 06:59
@akx akx force-pushed the p694-redux branch 2 times, most recently from 6cb29ea to c68ca01 Compare January 13, 2025 12:15
@akx akx marked this pull request as ready for review January 14, 2025 15:18
@akx akx requested a review from tomasr8 January 14, 2025 15:19
tomasr8
tomasr8 reviewed Jan 16, 2025
View reviewed changes
Comment thread babel/messages/extract.py
Comment thread babel/messages/extract.py
@akx akx self-assigned this Feb 1, 2025
@akx akx added this to the Babel 2.18 milestone Feb 1, 2025
@akx akx force-pushed the p694-redux branch 3 times, most recently from 522e42b to 95fc9b3 Compare December 29, 2025 10:40
@akx akx requested a review from Copilot December 29, 2025 11:00
Copilot AI reviewed Dec 29, 2025
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves message extraction performance by filtering out ignored directories early during the os.walk traversal, rather than checking each file individually. The implementation introduces a new directory filter factory and deprecates the old default_directory_filter function.

Key Changes:

  • New make_default_directory_filter factory function that creates directory filters aware of ignore patterns from the method map
  • Deprecation of the standalone default_directory_filter function with a clear warning
  • Refactored tests to validate both file-based ignore patterns (**/ignored/**.*) and simple directory patterns (ignored)

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File Description
babel/messages/extract.py Added make_default_directory_filter factory to create pattern-aware directory filters, deprecated old default_directory_filter, and updated extract_from_dir to use the new filter factory for early directory pruning during tree traversal
tests/messages/frontend/test_extract.py Refactored test_extraction_with_mapping_dict into a parameterized helper method and added new test case test_extraction_with_entire_directory_ignored to verify different ignore pattern styles

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread babel/messages/extract.py
Comment thread babel/messages/extract.py
Comment thread babel/messages/extract.py Outdated
Comment thread babel/messages/extract.py
@akx akx force-pushed the p694-redux branch 2 times, most recently from 8b1b28a to 8c455bb Compare December 29, 2025 11:12
@akx @stkao05
Improve extract performance via ignoring directories early during os.…
2021bf1 
…walk

Co-authored-by: Steven Kao <st.kao.05@gmail.com>
@akx akx requested a review from Copilot December 29, 2025 11:13
Copilot AI reviewed Dec 29, 2025
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@akx akx merged commit e91c346 into master Dec 29, 2025
43 of 44 checks passed
@akx akx deleted the p694-redux branch April 20, 2026 05:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments
+1 more reviewer
@tomasr8 tomasr8 tomasr8 left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

@akx akx

Labels

area/messages enhancement

Projects

None yet

Milestone

Babel 2.18

Development

Successfully merging this pull request may close these issues.

4 participants

@akx @DenverCoder1 @tomasr8