{{ message }}
#1475 -- Regular crawling should work when autodiscovery of sitemaps is turned off#1477
Merged
Conversation
tballison
changed the title
jnioche
requested changes
Feb 22, 2025
jnioche
left a comment
jnioche
left a comment
Contributor
There was a problem hiding this comment.
Instead of passing a configuration value through the metadata, I think we should instead make use of the configure method which all filters inherit.
We can get the value of the conf from there and if the sitemap detection is off, have a simple check at the beginning of the filter method and exit.
Contributor
Author
jnioche
approved these changes
Feb 28, 2025
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I tested this offline with https://www.cdc.gov and https://www.fda.gov.
I confirmed that when
sitemap.discovery=false, I could set one in the seed file totrue, and the behavior was as expected.I also tested the opposite, where the default was
true, but the seed for one of them wasfalse, and the behavior was as expected.I'm not sure this is the best solution. I don't like tightly coupling logic for the SitemapFilter in the FetcherBolts, but so it goes.
And, as usual, unit tests are, well, hard.
Let me know what you think.
Thank you for contributing to Apache StormCrawler.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:
Is there a issue associated with this PR? Is it referenced in the commit message?
Does your PR title start with
#XXXXwhereXXXXis the issue number you are trying to resolve?Has your PR been rebased against the latest commit within the target branch (typically main)?
Is your initial contribution a single, squashed commit?
Is the code properly formatted with
mvn git-code-format:format-code -Dgcf.globPattern="**/*" -Dskip.format.code=false?For code changes:
mvn clean verify?Note:
Please ensure that once the PR is submitted, you check GitHub Actions for build issues and submit an update to your PR as soon as possible.