PIP dependency support (#1) · climacell/rules_python@7f4cc92 · GitHub
Skip to content
This repository was archived by the owner on Sep 3, 2019. It is now read-only.

Commit 7f4cc92

Browse files
mattmoormattmoor
authored
PIP dependency support (bazel-contrib#1)
* Check in an initial version of the pip rules. This is broken off of my prototype repository, and there are a handful of TODOs left to resolve. * Remove some vestigial convenience code. Elaborate on why the TODO to rewrite {pip,whl}.sh as Python is harder than it looks. * Incorporate Code Review Feedback * First cut at skydoc-based documentation. * Move whl.sh into a Python script that has unit testing. * Migrate pip.sh into piptool.py This migrates the logic from pip.sh into piptool.py, which should improve portability by removing the bash dependency. This also has the beginnings of wrapping piptool as a closed redistributable that doesn't rely on a system-installed copy of PIP, but instead uses these rules to pull pip into a PAR bundle. Besides needing to work out the details of releasing and redistributing the PAR, we have two unresolved issues: * When bundled as a PAR (vs. py_binary), piptool seems to pick up the system-installed version of pip. * When bundled as a PAR, piptool sometimes sees cert issues resolving requirements (similar to what we see with httplib2). * Address the cert issue in piptool as a PAR. With this change I am able to build/test my PR on my macbook without pip installed. The only additional change I have locally is to switch from running piptool.py as a simple .py file to downloading/using a PAR built from this change. I believe we still have the problem that the .par picks the host's version of pip instead of our embedded copy, but I haven't reverified that issue still exists (this just does nothing to address that issue). * Fix assorted buildifier issues. * Fix a typo in docs * Incorporate @duggelz code review feedback. * Incorporate review feedback move python tools under a top-level rules_python package simplify version_test * Adopt a canonical naming format for imported whl_library rules. `whl_library` rules generated by `pip_import` are now named as: ``` pypi__{distribution}_{version} ``` Substituting illegal characters (e.g. `-`, `.`) with underscores. * Move the piptool dependency into a .PAR file. Add a script for updating the tools and document this and the docs script. * Fix buildifier issues.
1 parent 9160fc7 commit 7f4cc92

37 files changed

Lines changed: 2118 additions & 12 deletions

‎.ci/rules_python.json‎

Lines changed: 10 additions & 0 deletions

‎.gitignore‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
# Compiled Object files
2+
*.slo
3+
*.lo
4+
*.o
5+
*.obj
6+
7+
# Precompiled Headers
8+
*.gch
9+
*.pch
10+
11+
# Compiled Dynamic libraries
12+
*.so
13+
*.dylib
14+
*.dll
15+
16+
# Fortran module files
17+
*.mod
18+
*.smod
19+
20+
# Compiled Static libraries
21+
*.lai
22+
*.la
23+
*.a
24+
*.lib
25+
26+
# Executables
27+
*.exe
28+
*.out
29+
*.app
30+
31+
# Emacs garbage
32+
*~
33+
34+
# Bazel directories
35+
bazel-*
36+
bazel-bin
37+
bazel-genfiles
38+
bazel-out
39+
bazel-testlogs

‎.travis.yml‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
sudo: required
2+
dist: trusty
3+
language:
4+
- java
5+
jdk:
6+
- oraclejdk8 # Building Bazel requires JDK8.
7+
addons:
8+
apt:
9+
sources:
10+
- sourceline: 'deb [arch=amd64] http://storage.googleapis.com/bazel-apt stable jdk1.8'
11+
key_url: 'https://storage.googleapis.com/bazel-apt/doc/apt-key.pub.gpg'
12+
packages:
13+
- bazel
14+
15+
install:
16+
- go get -u github.com/bazelbuild/buildifier/buildifier
17+
18+
script:
19+
# Check that all of our tools and samples build
20+
- bazel clean && bazel build //...
21+
# Check that all of our tests pass
22+
- bazel clean && bazel test //...
23+
24+
# Check for issues with the format of our bazel config files.
25+
- buildifier -mode=check $(find . -name BUILD -type f)
26+
- buildifier -mode=check $(find . -name WORKSPACE -type f)
27+
- buildifier -mode=check $(find . -name '*.bzl' -type f)

‎README.md‎

Lines changed: 84 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -4,14 +4,17 @@
44

55
## Rules
66

7-
* [py_library](#py_library)
8-
* [py_binary](#py_binary)
7+
* [pip_import](docs/python/pip.md#pip_import)
8+
* [py_library](docs/python/python.md#py_library)
9+
* [py_binary](docs/python/python.md#py_binary)
10+
* [py_test](docs/python/python.md#py_test)
911

1012
## Overview
1113

12-
This is a placeholder repository that provides aliases for the native Bazel
13-
python rules. In the future, this will also become the home for rules that
14-
download `pip` packages, and other non-Core Python functionality.
14+
This repository provides Python rules for Bazel. Currently, support for
15+
rules that are available from Bazel core are simple aliases to that bundled
16+
functionality. On top of that, this repository provides support for installing
17+
dependencies typically managed via `pip`.
1518

1619
## Setup
1720

@@ -23,14 +26,19 @@ git_repository(
2326
remote = "https://github.com/bazelbuild/rules_python.git",
2427
commit = "{HEAD}",
2528
)
29+
30+
# Only needed for PIP support:
31+
load("//python:pip.bzl", "pip_repositories")
32+
33+
pip_repositories()
2634
```
2735

2836
Then in your `BUILD` files load the python rules with:
2937

3038
``` python
3139
load(
3240
"@io_bazel_rules_python//python:python.bzl",
33-
"py_binary", "py_library"
41+
"py_binary", "py_library", "py_test",
3442
)
3543

3644
py_binary(
@@ -39,12 +47,76 @@ py_binary(
3947
)
4048
```
4149

42-
<a name="py_library"></a>
43-
## py_library
50+
## Importing `pip` dependencies
51+
52+
These rules are designed to have developers continue using `requirements.txt`
53+
to express their dependencies in a Python idiomatic manner. These dependencies
54+
are imported into the Bazel dependency graph via a two-phased process in
55+
`WORKSPACE`:
56+
57+
```python
58+
load("@io_bazel_rules_python//python:pip.bzl", "pip_import")
59+
60+
# This rule translates the specified requirements.txt into
61+
# @my_deps//:requirements.bzl, which itself exposes a pip_install method.
62+
pip_import(
63+
name = "my_deps",
64+
requirements = "//path/to:requirements.txt",
65+
)
66+
67+
# Load the pip_install symbol for my_deps, and create the dependencies'
68+
# repositories.
69+
load("@my_deps//:requirements.bzl", "pip_install")
70+
pip_install()
71+
```
72+
73+
## Consuming `pip` dependencies
74+
75+
Once a set of dependencies has been imported via `pip_import` and `pip_install`
76+
we can start consuming them in our `py_{binary,library,test}` rules. In support
77+
of this, the generated `requirements.bzl` also contains a `package` method,
78+
which can be used directly in `deps=[]` to reference an imported `py_library`.
79+
80+
```python
81+
load("@my_deps//:requirements.bzl", "package")
82+
83+
py_library(
84+
name = "mylib",
85+
srcs = ["mylib.py"],
86+
deps = [
87+
":myotherlib",
88+
# This takes the name as specified in requirements.txt
89+
package("importeddep"),
90+
]
91+
)
92+
```
93+
94+
## Canonical `whl_library` naming
4495

45-
See Bazel core [documentation](https://docs.bazel.build/versions/master/be/python.html#py_library).
96+
It is notable that `whl_library` rules imported via `pip_import` are canonically
97+
named, following the pattern: `pypi__{distribution}_{version}`. Characters in
98+
these components that are illegal in Bazel label names (e.g. `-`, `.`) are
99+
replaced with `_`.
46100

47-
<a name="py_binary"></a>
48-
## py_binary
101+
This canonical naming helps avoid redundant work to import the same library
102+
multiple times. It is expected that this naming will remain stable, so folks
103+
should be able to reliably depend directly on e.g. `@pypi__futures_3_1_1//:pkg`
104+
for dependencies, however, it is recommended that folks stick with the `package`
105+
pattern in case the need arises for us to make changes to this format in the
106+
future.
49107

50-
See Bazel core [documentation](https://docs.bazel.build/versions/master/be/python.html#py_binary).
108+
## Updating `docs/`
109+
110+
All of the content (except `BUILD`) under `docs/` is generated. To update the
111+
documentation simply run this in the root of the repository:
112+
```shell
113+
./update_docs.sh
114+
```
115+
116+
## Updating `tools/`
117+
118+
All of the content (except `BUILD`) under `tools/` is generated. To update the
119+
documentation simply run this in the root of the repository:
120+
```shell
121+
./update_tools.sh
122+
```

‎WORKSPACE‎

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
# Copyright 2017 The Bazel Authors. All rights reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
workspace(name = "io_bazel_rules_python")
15+
16+
# Skydoc stuff
17+
git_repository(
18+
name = "io_bazel_rules_sass",
19+
remote = "https://github.com/bazelbuild/rules_sass.git",
20+
tag = "0.0.2",
21+
)
22+
23+
load("@io_bazel_rules_sass//sass:sass.bzl", "sass_repositories")
24+
25+
sass_repositories()
26+
27+
git_repository(
28+
name = "io_bazel_skydoc",
29+
remote = "https://github.com/bazelbuild/skydoc.git",
30+
tag = "0.1.3",
31+
)
32+
33+
load("@io_bazel_skydoc//skylark:skylark.bzl", "skydoc_repositories")
34+
35+
skydoc_repositories()
36+
37+
# Requirements for building our piptool.
38+
load("//python:pip.bzl", "pip_import")
39+
40+
pip_import(
41+
name = "piptool_deps",
42+
requirements = "//python:requirements.txt",
43+
)
44+
45+
load(
46+
"@piptool_deps//:requirements.bzl",
47+
_piptool_install = "pip_install",
48+
)
49+
50+
_piptool_install()
51+
52+
git_repository(
53+
name = "subpar",
54+
remote = "https://github.com/google/subpar",
55+
tag = "1.0.0",
56+
)
57+
58+
# Test data for WHL tool testing.
59+
http_file(
60+
name = "grpc_whl",
61+
sha256 = "c232d6d168cb582e5eba8e1c0da8d64b54b041dd5ea194895a2fe76050916561",
62+
# From https://pypi.python.org/pypi/grpcio/1.6.0
63+
url = ("https://pypi.python.org/packages/c6/28/" +
64+
"67651b4eabe616b27472c5518f9b2aa3f63beab8f62100b26f05ac428639/" +
65+
"grpcio-1.6.0-cp27-cp27m-manylinux1_i686.whl"),
66+
)
67+
68+
http_file(
69+
name = "futures_whl",
70+
sha256 = "c4884a65654a7c45435063e14ae85280eb1f111d94e542396717ba9828c4337f",
71+
# From https://pypi.python.org/pypi/futures
72+
url = ("https://pypi.python.org/packages/a6/1c/" +
73+
"72a18c8c7502ee1b38a604a5c5243aa8c2a64f4bba4e6631b1b8972235dd/" +
74+
"futures-3.1.1-py2-none-any.whl"),
75+
)
76+
77+
http_file(
78+
name = "mock_whl",
79+
sha256 = "5ce3c71c5545b472da17b72268978914d0252980348636840bd34a00b5cc96c1",
80+
# From https://pypi.python.org/pypi/mock
81+
url = ("https://pypi.python.org/packages/e6/35/" +
82+
"f187bdf23be87092bd0f1200d43d23076cee4d0dec109f195173fd3ebc79/" +
83+
"mock-2.0.0-py2.py3-none-any.whl"),
84+
)
85+
86+
# Imports for examples
87+
pip_import(
88+
name = "examples_helloworld",
89+
requirements = "//examples/helloworld:requirements.txt",
90+
)
91+
92+
load(
93+
"@examples_helloworld//:requirements.bzl",
94+
_helloworld_install = "pip_install",
95+
)
96+
97+
_helloworld_install()
98+
99+
pip_import(
100+
name = "examples_version",
101+
requirements = "//examples/version:requirements.txt",
102+
)
103+
104+
load(
105+
"@examples_version//:requirements.bzl",
106+
_version_install = "pip_install",
107+
)
108+
109+
_version_install()

‎docs/BUILD‎

Lines changed: 60 additions & 0 deletions

0 commit comments

Comments
 (0)