Make CLI libraries optional dependencies · Issue #703 · apify/crawlee-python · GitHub
Skip to content

Make CLI libraries optional dependencies #703

Closed
#1011
Closed
Make CLI libraries optional dependencies#703
#1011
Labels
t-toolingIssues with this label are in the ownership of the tooling team.

Description

@vdusek
vdusek
opened on Nov 15, 2024

Since cookiecutter, along with other libraries like inquirer and typer, is used exclusively in the CLI for templating new projects, could we make these optional dependencies? This would reduce the installation footprint.

Activity

  1. vdusek
    added
    t-toolingIssues with this label are in the ownership of the tooling team.
    on Nov 15, 2024

  2. B4nan commented on Nov 15, 2024

    @B4nan
    B4nan
    on Nov 15, 2024
    Member

    Wouldn't this mean that pipx run crawlee create my-crawler stop working?

  3. vdusek commented on Nov 18, 2024

    @vdusek
    vdusek
    on Nov 18, 2024
    CollaboratorAuthor

    The mentioned packages & their (transitive) dependencies.

    Cookiecutter

    cookiecutter 2.6.0 A command-line utility that creates projects from project templates, e.g. creating a Python package project from a Python package project template.
├── arrow *
│   ├── python-dateutil >=2.7.0 
│   │   └── six >=1.5 
│   └── types-python-dateutil >=2.8.10 
├── binaryornot >=0.4.4
│   └── chardet >=3.0.2 
├── click >=7.0,<9.0.0
│   └── colorama * 
├── jinja2 >=2.7,<4.0.0
│   └── markupsafe >=2.0 
├── python-slugify >=4.0.0
│   └── text-unidecode >=1.3 
├── pyyaml >=5.3.1
├── requests >=2.23.0
│   ├── certifi >=2017.4.17 
│   ├── charset-normalizer >=2,<4 
│   ├── idna >=2.5,<4 
│   └── urllib3 >=1.21.1,<3 
└── rich *
    ├── markdown-it-py >=2.2.0 
    │   └── mdurl >=0.1,<1.0 
    ├── pygments >=2.13.0,<3.0.0 
    └── typing-extensions >=4.0.0,<5.0

    Inquirer

    inquirer 3.4.0 Collection of common interactive command line user interfaces, based on Inquirer.js
├── blessed >=1.19.0
│   ├── jinxed >=1.1.0 
│   │   └── ansicon * 
│   ├── six >=1.9.0 
│   └── wcwidth >=0.1.4 
├── editor >=1.6.0
│   ├── runs * 
│   │   └── xmod * 
│   └── xmod * (circular dependency aborted here)
└── readchar >=4.2.0

    Typer

    typer 0.13.0 Typer, build great CLIs. Easy to code. Based on Python type hints.
├── click >=8.0.0
│   └── colorama * 
├── rich >=10.11.0
│   ├── markdown-it-py >=2.2.0 
│   │   └── mdurl >=0.1,<1.0 
│   ├── pygments >=2.13.0,<3.0.0 
│   └── typing-extensions >=4.0.0,<5.0 
├── shellingham >=1.3.0
└── typing-extensions >=3.7.4.3

  4. B4nan commented on Nov 18, 2024

    @B4nan
    B4nan
    on Nov 18, 2024
    Member

    Can you also check their install size?

    Also one idea, can we maybe have a script inside crawlee that would install things instead of expecting them to be already part of the crawlee deps? That way only the create command (which would just trigger this script) would install them.

  5. vdusek commented on Dec 3, 2024

    @vdusek
    vdusek
    on Dec 3, 2024
    CollaboratorAuthor

    I used the python-package-size tool to measure the size of each package. This tool creates a separate virtual environment for every package, installs the package along with all its transitive dependencies, and measures the total size. Note that the reported size includes all dependencies, so removing a package does not necessarily mean we will save that much space (if the dependencies are shared with other packages).

    Cookiecutter

    Creating new venv /tmp/tmpbyp2thfj/venv-cookiecutter
Installing cookiecutter:  /tmp/tmpbyp2thfj/venv-cookiecutter/bin/pip install cookiecutter
Size of cookiecutter: 21.8 MB

    Typer

    Creating new venv /tmp/tmpbyp2thfj/venv-typer
Installing typer:  /tmp/tmpbyp2thfj/venv-typer/bin/pip install typer
Size of typer: 11.8 MB

    Inquirer

    Creating new venv /tmp/tmpbyp2thfj/venv-inquirer
Installing inquirer:  /tmp/tmpbyp2thfj/venv-inquirer/bin/pip install inquirer
Size of inquirer: 1.2 MB

  6. janbuchar commented on Dec 5, 2024

    @janbuchar
    janbuchar
    on Dec 5, 2024
    Collaborator

    Wouldn't this mean that pipx run crawlee create my-crawler stop working?

    If we made a cli extra, the command would become pipx run crawlee[cli] create my-crawler. If the cli detected missing dependencies, it could just offer the user the right command. Wouldn't that be satisfactory?

  7. vdusek commented on Dec 6, 2024

    @vdusek
    vdusek
    on Dec 6, 2024
    CollaboratorAuthor

    If we made a cli extra, the command would become pipx run crawlee[cli] create my-crawler. If the cli detected missing dependencies, it could just offer the user the right command. Wouldn't that be satisfactory?

    I am ok with that.

  8. janbuchar commented on Dec 6, 2024

    @janbuchar
    janbuchar
    on Dec 6, 2024
    Collaborator

    If we decide to mention uvx as well, the corresponding command would be uvx --with crawlee[cli] crawlee create my-crawler, which is... unsexy. See astral-sh/uv#6296 - the --with flag may become unnecessary in the future.

  9. vdusek commented on Dec 6, 2024

    @vdusek
    vdusek
    on Dec 6, 2024
    CollaboratorAuthor

    It seems that uv is still in its early stages. I would wait a few more months before try to migrate. For example, the uv publish feature is currently labeled as experimental.

  10. janbuchar commented on Dec 6, 2024

    @janbuchar
    janbuchar
    on Dec 6, 2024
    Collaborator

    We don't have to support uvx, people can use it already. We might just want to add that to the landing page or whatever. But that's straying from the original topic here...

  11. vdusek commented on Feb 19, 2025

    @vdusek
    vdusek
    on Feb 19, 2025
    CollaboratorAuthor

    Moving it to new issues and let's discuss it on Monday.

  12. vdusek commented on Feb 19, 2025

    @vdusek
    vdusek
    on Feb 19, 2025
    CollaboratorAuthor

    We could utilize the uv's script dependencies, as @Mantisus stated here:

    Oh, got it. But I think it's easier to have cli dependencies like script dependencies for support uvx.

  13. janbuchar commented on Feb 19, 2025

    @janbuchar
    janbuchar
    on Feb 19, 2025
    Collaborator

    That, too - pipx and uvx should both support script dependencies.

    But with the new release of uv, both of these should work equally if we make that cli extra:

    pipx run 'crawlee[cli]' create mycrawler # https://pipx.pypa.io/stable/examples/#pipx-run-examples
uvx 'crawlee[cli]' create mycrawler
  14. vdusek
    closed this as completedin #1011on Mar 3, 2025
  15. vdusek
    closed this as completedin 4382959on Mar 3, 2025
  16. Mantisus
    added a commit that references this issue on Mar 3, 2025

    refactor!: move `cli` dependencies to optional dependencies (apify#1011)

    4382959
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    t-toolingIssues with this label are in the ownership of the tooling team.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions