Allow the buildpack use miniconda when `environment.yml` is present. · snippet-java/python-buildpack@07c5b15 · GitHub
Skip to content

Commit 07c5b15

Browse files
geramirezpivotal
geramirez
authored and
pivotal
committed
Allow the buildpack use miniconda when environment.yml is present.
Also, warns the users when they have listed Python runtimes in both `runtime.txt` and `environment.yml` [#118752873] Signed-off-by: John Shahid <jvshahid@gmail.com>
1 parent 3cc5f5b commit 07c5b15

19 files changed

Lines changed: 405 additions & 1 deletion

File tree

‎bin/compile‎

Lines changed: 8 additions & 0 deletions

‎bin/detect‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ BUILD_DIR=$1
1616
BP=$(dirname $(dirname $0))
1717

1818
# Exit early if app is clearly not Python.
19-
if [ ! -f $BUILD_DIR/requirements.txt ] && [ ! -f $BUILD_DIR/setup.py ]; then
19+
if [ ! -f $BUILD_DIR/requirements.txt ] && [ ! -f $BUILD_DIR/setup.py ] && [ ! -f $BUILD_DIR/environment.yml ]; then
2020
exit 1
2121
fi
2222

‎bin/steps/conda-install‎

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
#!/usr/bin/env bash
2+
# Cloud Foundry Python Conda Buildpack
3+
# Copyright (c) 2014-2015 the original author or authors.
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
#
17+
18+
echo "-----> Starting compile step..."
19+
BUILD_DIR=$1
20+
CACHE=$2
21+
22+
BASH=$(which bash)
23+
WGET=$(which wget)
24+
CONDA_HOME="$1/.conda"
25+
CONDA_BIN="$CONDA_HOME/bin"
26+
RUNTIME="$BUILD_DIR/runtime.txt"
27+
28+
# Get the runtime version and download appropriate Miniconda
29+
if [ -e $RUNTIME ]; then
30+
PYTHON_VERSION=$(cut -d- -f2 $RUNTIME)
31+
if [ ${PYTHON_VERSION:0:1} -eq 3 ]; then
32+
PYTHON_MAJOR_VERSION=3
33+
else
34+
PYTHON_MAJOR_VERSION=""
35+
fi
36+
MINICONDA_FILE="Miniconda$PYTHON_MAJOR_VERSION-latest-Linux-x86_64.sh"
37+
else
38+
MINICONDA_FILE="Miniconda-latest-Linux-x86_64.sh"
39+
fi
40+
41+
MINICONDA_URI="http://repo.continuum.io/miniconda/$MINICONDA_FILE"
42+
MINICONDA_CACHE="$CACHE/$MINICONDA_FILE"
43+
44+
PROFILE_PATH="$BUILD_DIR/.profile.d/conda.sh"
45+
46+
echo "-----> Preparing Python Environment..."
47+
if [ ! -e $MINICONDA_CACHE ] ; then
48+
echo "-----> Downloading Miniconda..."
49+
if [ ! -d $CACHE ]; then mkdir $CACHE; fi
50+
$WGET -q -O $MINICONDA_CACHE $MINICONDA_URI
51+
chmod +x $MINICONDA_CACHE
52+
fi
53+
if [ -e $CONDA_HOME ]; then rm -rf $CONDA_HOME; fi
54+
# Install miniconda
55+
$MINICONDA_CACHE -b -p $CONDA_HOME #&> /dev/null
56+
57+
echo "-----> Installing Dependencies..."
58+
$CONDA_BIN/conda update --yes --quiet conda
59+
$CONDA_BIN/conda install --yes --quiet pip
60+
61+
# Default Conda env is root
62+
CONDA_ENV="root"
63+
64+
# Install application dependencies
65+
echo "-----> Installing conda environment from environment.yml..."
66+
$CONDA_BIN/conda env update -n root -f "$BUILD_DIR/environment.yml"
67+
68+
$CONDA_BIN/conda clean -pt
69+
70+
#
71+
echo "-----> Fixing paths..."
72+
grep -rlI $BUILD_DIR $BUILD_DIR | xargs sed -i.bak "s|$BUILD_DIR|/home/vcap/app|g"
73+
#
74+
75+
# Add Conda path to profile
76+
mkdir -p $(dirname $PROFILE_PATH)
77+
echo "export PATH=$HOME/app/.conda/bin:\$PATH" >> $PROFILE_PATH
78+
79+
if test -f $BUILD_DIR/runtime.txt && grep 'python=' $BUILD_DIR/environment.yml; then
80+
echo "WARNING: you have specified the version of Python runtime both in 'runtime.txt and 'environment.yml'. You should remove one of the two versions"
81+
fi
82+
83+
echo "-----> Finished compile step"

‎cf_spec/fixtures/miniconda_simple_app_python_2/Procfile‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
web: python app.py

‎cf_spec/fixtures/miniconda_simple_app_python_2/app.py‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
from flask import Flask
2+
import pytest
3+
import os
4+
import importlib
5+
import sys
6+
7+
MODULE_NAMES = ['numpy', 'scipy', 'sklearn', 'pandas']
8+
modules = {}
9+
10+
for m in MODULE_NAMES:
11+
try:
12+
modules[m] = importlib.import_module(m)
13+
except ImportError:
14+
modules[m] = None
15+
16+
app = Flask(__name__)
17+
18+
19+
@app.route('/<module_name>')
20+
def in_module_tests(module_name):
21+
if module_name not in modules:
22+
return "This module is not listed"
23+
try:
24+
module = modules[module_name]
25+
if module_name == 'sklearn':
26+
result = pytest.main('--pyargs sklearn.tests')
27+
result_string = "sklearn: passed" if result == 0 else "sklearn: failed"
28+
else:
29+
result = module.test()
30+
num_failures = result.failures
31+
result_string = "{}: number of failures={}".format(module_name, len(num_failures))
32+
except (NameError, ImportError, AttributeError):
33+
result_string = "{}: Error running test!".format(module_name)
34+
return result_string
35+
36+
37+
@app.route('/all')
38+
def run_all():
39+
results = "<br>\n".join([in_module_tests(m) for m in MODULE_NAMES])
40+
return str(results)
41+
42+
43+
def module_version(module_name):
44+
m = modules[module_name]
45+
if m is None:
46+
version_string = "{}: unable to import".format(module_name)
47+
else:
48+
version_string = "{}: {}".format(module_name, m.__version__)
49+
return version_string
50+
51+
52+
@app.route('/')
53+
def root():
54+
versions = "<br>\n".join([module_version(m) for m in MODULE_NAMES])
55+
python_version = "\npython-version%s\n" % sys.version
56+
r = """<br><br>
57+
Imports Successful!<br>
58+
59+
To test each module go to /numpy, /scipy, /sklearn and /pandas
60+
or test all at /all.<br>
61+
Test suites can take up to 10 minutes to run, main output is in app logs."""
62+
return python_version + versions + r
63+
64+
if __name__ == '__main__':
65+
# Get port from environment variable or choose 9099 as local default
66+
port = int(os.getenv("PORT", 9099))
67+
# Run the app, listening on all IPs with our chosen port number
68+
app.run(host='0.0.0.0', port=port, debug=True)

‎cf_spec/fixtures/miniconda_simple_app_python_2/environment.yml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
name: pydata_test
2+
dependencies:
3+
- pip
4+
- pytest
5+
- flask
6+
- nose
7+
- numpy=1.10.4
8+
- scipy=0.17.0
9+
- scikit-learn=0.17.1
10+
- pandas=0.18.0

‎cf_spec/fixtures/miniconda_simple_app_python_2/manifest.yml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
applications:
3+
- name: pydata_cf_test
4+
memory: 1GB
5+
disk_quota: 1GB
6+
command: python app.py

‎cf_spec/fixtures/miniconda_simple_app_python_2/runtime.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
python-2.7.11

‎cf_spec/fixtures/miniconda_simple_app_python_2_3/Procfile‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
web: python app.py

‎cf_spec/fixtures/miniconda_simple_app_python_2_3/app.py‎

Lines changed: 68 additions & 0 deletions

0 commit comments

Comments
 (0)