{{ message }}
Repository navigation
Breaking changes for v0.6 #906
Description
Remove unused fields in the Configuration
- Fields to remove:
chrome_executable_path,xvfb. - Additionally, consider removing the
verbose_logfield:- The
verbose_logfield is only used to derive the log level, and alog_leveloption already exists. So I consider it as redundant.
- The
Refactor abstract class naming
- Remove "Base" from the names of abstract classes (e.g.,
BaseStorage,BaseStorageClient, ...). - Together with @janbuchar we came to a conclusion not to use the Hungarian notation (where the name of a variable indicates its intention or kind). @Pijukatel, feel free to share your thoughts.
Rename enqueue_links for clarity
enqueue_links for clarityThe naming ofenqueue_linkscauses confusion due to its similarity toadd_requests.Proposed solution:Renameenqueue_linksto something more descriptive, such asextract_links(indicating what it does more precisely).
I am open to name suggestions here.
Update of enqueue_links
enqueue_linkswill have the same interface as its JavaScript counterpart, enqueueLinks.add_requestswill remain dedicated to adding requests to the RQ only.- A new
extract_linksfunction can be introduced for link extraction only. - Internally,
enqueue_linkscan utilize bothadd_requestsandextract_links.
Activity
Pijukatel commentedon Jan 15, 2025
Regarding abstract class naming, just one more thing to think about. It will involve both BaseSomeClass and AbstractSomeClass. What will we do when we have default implementation of such class? For example HttpCrawler is one implementation of AbstractHttpCrawler. (Where HttpCrawler was reserved for backwards compatibility)
vdusek commentedon Jan 15, 2025
Maybe for
AbstractHttpCrawlerwe would have to make an exception. Or rename theHttpCrawlerto something else.Mantisus commentedon Feb 5, 2025
I'm thinking of moving from a
dictto aCookieJar-based approach for storing cookies inSessionfor #933, would be appropriate for 0.6refactor!: remove unused config properties (#978)
refactor!: remove Base prefix from abstract class names (#980)
vdusek commentedon Feb 12, 2025
One more thing from Ruff 0.9:
Since
statisticsis a public module, we should consider renaming it for v0.6 release. Maybe juststats? Or do you have any suggestions?@Pijukatel @janbuchar
janbuchar commentedon Feb 12, 2025
I wouldn't mind disabling the rule in this case. It is unlikely that anyone will do
from crawlee import statistics- that would be cumbersome to work with.vdusek commentedon Mar 4, 2025
Closing, as all the breaking changes were resolved. (There are no breaking changes in #1024.)