CDP setup in one click

This repository contains a set of scripts that will create CDP minimal assets for demo in one wrapper script, including:

• Cloud pre-requisites (bucket, policies, roles, network)
• Cloud CDP Environmment
• CDP Data Lake
• Any CDP Data Hub cluster definition

Pre-Requisites

AWS

• AWS CLI (Instructions)
  • You must run aws configure after install, and ensure your region is set
• AWS ssh key (Instructions); Alternatively, you can use the field or _field keys setup in our AWS SE accounts

Azure

• Azure CLI (Instructions
  • Use az login after install to login
• ssh Key: you will need to paste the public key into your parameters file

Note: Azure does not support CML yet, so don't add it in your parameters file :)

CDP

• CDP CLI (Instructions)

• CDP Credential (Instructions)

  • You must set your Workload Password in your CDP Profile (Shortcut)
  • You must generate a CLI Access Key in your CDP Profile, and configure it to your local CDP CLI (Shortcut)

Parameters file format

Detailed format

{
    Required parameters: 
    "required": {
                
        Prefix used for cdp assets creation:  
        "prefix":         "pvi",
        
        Name of credential to use: 
        "credential":     "pvidal-aws-se-credential",

        Region to use (should also be the default region of your cloud provider cli profile): 
        "region":         "us-east-1",
        
        ssh key to use for cdp instances setup: 
        "key":            "field",

        Workload password to use in CDP: 
        "workload_pwd":   "cdpw0rksh0p",

        Array of datahub to setup (can be empty): 
        "datahub_list": [

            Element 1: 
            {
                Definition from cdp-cluster-definitions folder: 
                "definition": "data-mart.json",

                Custom script from cdp-dh-custom-scripts folder: 
                "custom_script": ""
            },

            Element 2: 
            {
                Definition from cdp-cluster-definitions folder: 
                "definition": "cdp-mod-workshop.json",

                Custom script from cdp-dh-custom-scripts folder: 
                "custom_script": "cdp_mod_wkp.sh"
            },
        ],

        Array of ml workspaces to setup (can be empty): 
        "ml_workspace_list": [

            Element 1: 
            {
                Definition from cml-workspace-definitions folder: 
                "definition": "small_workspace.json"
            }

        ]
    },

    Optional (defaulted) parameters (can be empty): 
    "optional": {

        Cloud provider (default: aws, possible values: aws, az): 
        "cloud_provider": "aws", 

        Cloud provider cli profile (only supported in AWS) (default: default): 
        "cloud_profile":    "default",

        CDP cli profile (default: default): 
        "cdp_profile":    "default",

        Flag to create cdp credential or not (default: no, possible values: yes, no) 
        "generate_credential": "no",

        External ID for cross account role creation, can be found in register environment screen (only needed for AWS)
        "external_id": "YOUR_EXTERNAL_ID",

        NOT SUPPORTED YET Flag to generate minimal cross account role policy or not (default: no, possible values: yes, no) 
        "generate_minimal_cross_account": "no",

        Flag to create network in cloud provider or not (only supported in AWS) (default: no, possible values: yes, no) 
        "create_network": "no",

        Tags to setup: 
        "tags": {

            End date tag (default: today's date +3 days): 
            "end_date":       "04102020",
            End date tag (default: [prefix]_one_click_project): 
            "project":        "pvi_test"
        },

    }

}

Parameters file samples

See parameters_sample folder

Doing all the things (full wrapper)

Creation

Run the source target wrapper script:

cdp_create_all_the_things.sh <your_param_file> 

Deletion

Run the deletion script:

cdp_delete_all_the_things.sh <your_param_file>

Doing some of the things (individual wrappers)

AWS things

Pre-requisites

cdp_aws_pre_reqs.sh.sh <your_param_file>

SDX

cdp_aws_sdx.sh <your_param_file> [<network_file>]

Azure things

Pre-requisites

cdp_az_pre_reqs.sh.sh <your_param_file>

SDX

cdp_az_sdx.sh <your_param_file>

CDP things

Datahub

cdp_create_datahub_things.sh <your_param_file>

ML

cdp_create_ml_things.sh <your_param_file> 

Starting / Stopping (work in progress)

cdp_stop_all_the_things.sh <your_param_file> 

cdp_start_all_the_things.sh <your_param_file> 

Future Improvements

• Add support for Azure ML
• Add support for minimal set of policies for AWS
• Add dynamic definition updates
• Create a nifi flow wrapper?

Author & Contributors

Paul Vidal - LinkedIn

Dan Chaffelson - LinkedIn

Chris Perro - LinkedIn

André Araújo - LinkedIn

Nathan Anthony - LinkedIn

Steffen Maerkl - LinkedIn

Mike Riggs - LinkedIn