Dependency Management

Dependency Management

Bernd Doser

November 5, 2019

HITS RGB eng

Who am I?

Outline

  • Introduction into dependency management

  • Docker development environment

  • Visual Studio Code

  • Python

    • pip, virtualenv, setup.py, wheels

  • Conan.io

    • install, create, upload

Introduction

Dependency management is a technique for declaring, resolving and using dependencies required by the project in an automated fashion.

(Source: Devopedia)

  • Dependencies are project related

  • Keep your dependencies up to date to reduce technical debts

  • Distinguish build-time and runtime requirements

Package vs Dependency

  • Package manager: Install applications, libraries and tools on a system

    • apt (Linux)

    • brew (MacOS)

  • Dependency manager: Handle project dependencies across environments

    • pip, conda (Python)

    • conan (C++)

What is a software dependency

  • Module A uses functionality of module B

  • Transitive dependency → Dependency graph

    transitive

Dependency hell

  • Too many dependencies

  • Conflicting dependencies

  • Circular dependencies

    circular

Version numbering

major.minor[.patch]

  • Bugfixes in minor version

  • New interfaces and features in major version

Version specifier

  • Comparison

    ==: exact match
!=: exclusion
<=,>=: inclusive ordered
<,>: exclusive ordered

  • Compatibility (for stability)

    ~= 1.4.5
>= 1.4.5, == 1.4.*

  • Combination

    ~=3.1.0, != 3.1.3: version 3.1.0 or later,
                   but not version 3.1.3
                   and not version 3.2.0
                   or later

Docker

  • Virtualization of applications (lightweight VM)

  • Coded environment by Dockerfile

  • Important tool for software development

  • Images shared at DockerHub

  • Flexible deployment (Orchestration) via Kubernetes

  • Continuous Integration (Jenkins) by Jenkinsfile

Docker Development Environment

docker devel env

Visual Studio Code

  • Free and open source (not Microsoft Visual Studio)

  • Most popular development environment 2019

  • Language Server Protocol (LSP) as open standard for language specific features

    • code completion and navigation

    • refactoring, syntax highlighting, error markers

  • embedded git and GitHub support

Package installer for Python (PIP)

  • Python Package Index

  • Build recipe as code 'setup.py' from setuptools

  • Wheels for platform-specific C extensions (replace eggs, which was building everything from scratch)

  • Docker image 'manylinux' with old 'glibc' to support most Linux distributions

Exercise 1

Install TensorFlow in virtualenv

  • Install virtualenv

  • Create and activate virtualenv 'tensorflow'

  • Install tensorflow

  • Test tensorflow

Python packaging

setup.py

import setuptools

with open("README.md", "r") as fh:
    long_description = fh.read()

setuptools.setup(
    name="example-pkg-your-username",
    version="0.0.1",
    author="Example Author",
    author_email="author@example.com",
    description="A small example package",
    long_description=long_description,
    long_description_content_type="text/markdown",
    url="https://github.com/pypa/sampleproject",
    packages=setuptools.find_packages(),
    classifiers=[
        "Programming Language :: Python :: 3",
        "License :: OSI Approved :: MIT License",
        "Operating System :: OS Independent",
    ],
)

Docu / Example Tensorflow

Exercise 2

Install tqdm with setup.py

  • Clone tqdm from GitHub

  • Install via setup.py

  • Test tqdm

Exercise 3

Create package python-example

C++ dependency management with conan.io

  • Decentralized package manager

  • Client-server architecture similar to git push/pull

  • CMake integration with cmake-conan

  • Support all platforms (Linux, Apple, Windows, Android, embedded, …​)

  • Support all build systems (CMake, Makefile, Visual Studio, …​)

Conan repositories

  • conan-center: Official maintained by the Conan team (178 packages)

  • bincrafters: Group of OSS developers (370 packages)

  • braintwister: Personal repository at Bintray for OSS

  • Running conan_server for on-site repository

Installing dependencies

conanfile.txt

[requires]
Poco/1.9.0@pocoproject/stable

[generators]
cmake

name / version @ user / channel

Creating package

conanfile.py

from conans import ConanFile, CMake

class PackageConan(ConanFile):
    name = "<package name>"
    version = "0.1"
    license = "<Put the package license here>"
    url = "<Package recipe repository url>"
    description = "<Description of Hello here>"
    settings = "os", "compiler", "build_type", "arch"
    options = {"shared": [True, False]}
    default_options = {"shared": False}
    generators = "cmake"

    def source(self):
        self.run("git clone https://github.com/memsharded/hello.git")
        self.run("cd hello")

    def build(self):
        cmake = CMake(self)
        cmake.configure(source_folder="hello")
        cmake.build()

    def package(self):
        self.copy("*.h", dst="include", src="hello")
        self.copy("*.so", dst="lib", keep_path=False)

    def package_info(self):
        self.cpp_info.libs = ["hello"]

Exercise 4

Install range-v3 with conan

  • Clone conan-example from GitHub

  • Install with conanfile.txt

  • Compile and run

Exercise 5

Build package conan-example

  • Write conanfile.py

  • Create package

  • Upload to Bintray

Thank you

https://bernddoser.github.io/workshop-dependency-management