iframe-proxy | Sunbelt Computer Software

Repository navigation

History

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

<!DOCTYPE html>

<title>Scaling Python ML | Blog of my adventures working with different tools for scaling Python ML workloads.</title>

{"@type":"WebSite","headline":"Scaling Python ML","url":"https://scalingpythonml.com/","name":"Scaling Python ML","description":"Blog of my adventures working with different tools for scaling Python ML workloads.","@context":"https://schema.org"}</script>

if(!(window.doNotTrack === "1" || navigator.doNotTrack === "1" || navigator.doNotTrack === "yes" || navigator.msDoNotTrack === "1")) {

(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){

(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),

m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)

})(window,document,'script','https://www.google-analytics.com/analytics.js','ga');

ga('create', 'UA-175499613-1', 'auto');

ga('send', 'pageview');

}

</script>

document.addEventListener("DOMContentLoaded", function() {

renderMathInElement( document.body, {

delimiters: [

{left: "$$", right: "$$", display: true},

{left: "[%", right: "%]", display: true},

{left: "$", right: "$", display: false}

]}

);

});

</script>

function wrap_img(fn) {

if (document.attachEvent ? document.readyState === "complete" : document.readyState !== "loading") {

var elements = document.querySelectorAll(".post img");

Array.prototype.forEach.call(elements, function(el, i) {

if (el.getAttribute("title")) {

const caption = document.createElement('figcaption');

var node = document.createTextNode(el.getAttribute("title"));

caption.appendChild(node);

const wrapper = document.createElement('figure');

wrapper.className = 'image';

el.parentNode.insertBefore(wrapper, el);

el.parentNode.removeChild(el);

wrapper.appendChild(el);

wrapper.appendChild(caption);

}

});

} else { document.addEventListener('DOMContentLoaded', fn); }

}

window.onload = wrap_img;

</script>

</head>

<div class="wrapper"><a class="site-title" rel="author" href="/">Scaling Python ML</a><nav class="site-nav">

</svg>

</span>

</label>

<div class="trigger"><a class="page-link" href="/about.html">About</a><a class="page-link" href="/mailinglist.html">Mailinglist</a></div>

</nav></div>

</header>

<div class="home"><p>Hello and welcome to my adventures in Scaling Python Machine Learning (ML).</p>

<h2 id="scaling-python-ml">Scaling Python ML</h2>

<h2 class="post-list-heading">Posts</h2>

<h3>

Initial Steps at Getting polynote and ZeroToJupyterHub to work together (ish)

</a>

</h3><div class="paragraph">

<p>JupyterHub has long been the de facto leader in the notebook space. Polynote is a new approach to notebooks with multi-language kernels and a focus on first-class Scala support. There is a lot of tooling to make deploying & managing JupyterHub easy for various environments. Since Polynote is relatively new, the same tooling does not yet exist. Since I am lazy, and I wanted a great Notebook for Spark (w/ Scala & Python), I figured I’d try and see if I could get my ZeroToJupyterHub deployment to launch a Polynote notebook. This way I can have multi-user Kubernetes deployments of Polynote alongside my other Kernels that I’m using (Python w/Dask, Python w/Ray, etc.). It turns out, yes it is possible, and there are of course some things I learned along the way and despite my initial thought this would "take an hour or so" it ended up being an 8 part stream, and I don’t have security set up.</p>

</div></li><li><span class="post-meta">Jun 30, 2021</span>

<h3>

Setting up Per User Secrets and other customizations with Jupyterhub on Kubernetes

</a>

</h3><div class="paragraph">

<p>While we’ve already set up Jupyterhub using zero-to-jupyterhub on Kubernetes, I wanted to expand access to my other prospective co-authors as part of trying to convince them to write with me (again) :). While I generally speaking trust my co-authors, I still like having some kind of access controls, so we don’t accidentally stomp on each-others work.</p>

</div></li><li><span class="post-meta">Jun 29, 2021</span>

<h3>

A Quick Look at DF I/O (basic ETL w/JSON over http to CSV and Parquet on MinIO) in Dask

</a>

</h3><div class="paragraph">

<p>Now that we’ve got Dask installed, it’s time to try some simple data preparation and extract, transform, load(ETL). While ETL is often not the most exciting thing, getting data is the first step of most adventures. Data tools don’t exist in a vacuum; the data normally comes from somewhere else, and the data or models we make need to be useable with other tools. Because of this, the formats and systems that a tool can interact with can make a difference between it being a fit or needing to keep looking. To simplify your life with I/O, you should make sure your notebook (or client) runs inside the same cluster as the workers.</p>

</div></li><li><span class="post-meta">Feb 22, 2021</span>

<h3>

Tagging my ARM NVidia Jetson machines with GPUs in my Kubernetes (k3s) cluster

</a>

</h3><div class="paragraph">

<p>We previously setup the cluster with GPUs, but since we want to get into using GPU acceleration, it’s important to be able to request these resources from the Kubernetes scheduler. Tagging the nodes will help Kubernetes tell the different between the Raspbery Pi and Jetson Xavier in <a href="#my-cluster-img">[my-cluster-img]</a>. Depending on your type of GPUs there are options for automatically labeling your nodes. Since I’ve got NVidia boards we’ll use the k8s-device-plugin to do the labeling of the nodes. For our machines, though, the containers out of the box do not run on ARM and the code does not detect Jetson chips.</p>

</div></li><li><span class="post-meta">Dec 21, 2020</span>

<h3>

Running Spark Jupyter Notebooks Client Mode inside of a Kubernetes Cluster (with ARM for Extra Fun)

</a>

</h3><div class="paragraph">

<p>Having your Spark Notebook inside the same cluster as the executors can reduce network errors and improve uptime. Since these network issues can result in job failure, this is an important consideration. This post assumes that you’ve already set up the foundation JupyterHub inside of Kubernetes deployment; <a href="https://scalingpythonml.com/2020/12/12/deploying-jupyter-lab-notebook-for-dask-on-arm-on-k8s.html">the Dask-distributed notebook blog post covers that if you haven’t</a>.</p>

</div></li><li><span class="post-meta">Dec 12, 2020</span>

<h3>

Deploying Jupyter Lab/Notebook for Dask on ARM on Kubernetes

</a>

</h3><div class="paragraph">

<p>In this post, we are going to go through how to deploy Jupyter Lab on ARM on Kubernetes. We’ll also build a container for use with Dask, but you can skip/customize this step to meet your own needs. In the <a href="/2020/11/03/a-first-look-at-dask-on-arm-on-k8s.html">previous post, I got Dask on ARM on Kubernetes working</a>, while using remote access to allow the Jupyter notebook to run outside of the cluster. After running into a few issues from having the client code outside of the cluster, I decided it was worth the effort to set up Jupyter on ARM on K8s.</p>

</div></li><li><span class="post-meta">Dec 11, 2020</span>

<h3>

Some sharp corners with docker buildx (especially with qemu)

</a>

</h3><p>Have you been trying out Docker’s wonderful new buildx with QEMU, but are getting an unexpected “exec user process caused: exec format error” or strange segfaults on ARM? If so, this short and sweet blog post is for you. I want to be clear: I think buildx with qemu is amazing, but there are some sharp edges to keep your eyes out on.</p>

</li><li><span class="post-meta">Nov 3, 2020</span>

<h3>

A First Look at Dask on ARM on K8s

</a>

</h3><div class="paragraph">

<p>After getting the cluster set up in the previous post, it was time to finally play with Dask on the cluster. Thankfully, there are <a href="https://github.com/dask/dask-kubernetes">dask-kubernetes</a> and <a href="https://github.com/dask/dask-docker">dask-docker</a> projects that provide the framework to do this. Since I’m still new to Dask, I decided to start off by using Dask from a local notebook (in retrospect maybe not the best choice).</p>

</div></li><li><span class="post-meta">Oct 18, 2020</span>

<h3>

Setting up K3s (lightweight Kubernetes) with Persistent Volumes and Minio on ARM

</a>

</h3><p>After the <a href="http://scalingpythonml.com/2020/09/20/building-the-physical-cluster.html">last adventure</a> of getting the rack built and acquiring the machines, it was time to set up the software. Originally, I had planned to do this in a day or two, but in practice, it ran like so many other “simple” projects and some things I had assumed would be “super quick” ended up taking much longer than planned.</p>

</li><li><span class="post-meta">Sep 20, 2020</span>

<h3>

Building the Test Cluster

</a>

</h3><p>To ensure that the results between tests are as comparable as possible, I’m using a consistent hardware setup whenever possible. Rather than use a cloud provider I (with the help of Nova) set up a rack with a few different nodes. Using my own hardware allows me to avoid the <a href="https://en.wikipedia.org/wiki/Cloud_computing_issues#Performance_interference_and_noisy_neighbors">noisy neighbor problem</a>

with any performance numbers and gives me more control over simulating network partitions. A downside is that the environment is not as easily re-creatable.</p>

</li><li><span class="post-meta">Aug 16, 2020</span>

<h3>

A First (Brief) Look at Ray on Kubernetes

</a>

</h3><p>After my motorcycle/Vespa crash last year I took some time away from work. While I was out and trying to practice getting my typing speed back up, I decided to play with Ray, which was pretty cool. Ray comes out of the same<sup id="fnref:lab" role="doc-noteref"><a href="#fn:lab" class="footnote">1</a></sup> research lab that created the initial work that became the basis of Apache Spark. Like Spark, the primary authors have now started a company (Anyscale) to grow Ray. Unlike Spark, Ray is a Python first library and does not depend on the Java Virtual Machine (JVM) – and as someone who’s spent way more time than she would like getting the JVM and Python to play together, Ray and it’s cohort seem quite promising.</p>

<ol>

<p>Well… same-ish. It’s technically a bit more complicated because of the way the professors choose to run their labs, but if you look at the advisors you’ll notice a lot of overlap. <a href="#fnref:lab" class="reversefootnote" role="doc-backlink">↩</a></p>

</li>

</ol>

</div>

</li></ul>

<p class="rss-subscribe">subscribe <a href="/feed.xml">via RSS</a></p></div>

</div>

</main><footer class="site-footer h-card">

<h2 class="footer-heading">Scaling Python ML</h2>

<li class="p-name">Scaling Python ML</li><li><a class="u-email" href="mailto:holden@pigscanfly.ca">holden@pigscanfly.ca</a></li></ul>

</div>

<li><a href="https://github.com/scalingpythonml"><svg class="svg-icon"><use xlink:href="/assets/minima-social-icons.svg#github"></use></svg> <span class="username">scalingpythonml</span></a></li><li><a href="https://www.twitter.com/holdenkarau"><svg class="svg-icon"><use xlink:href="/assets/minima-social-icons.svg#twitter"></use></svg> <span class="username">holdenkarau</span></a></li><li><a href="https://youtube.com/holdenkarau"><svg class="svg-icon"><use xlink:href="/assets/minima-social-icons.svg#youtube"></use></svg> <span class="username">holdenkarau</span></a></li></ul>

</div>

<p>Blog of my adventures working with different tools for scaling Python ML workloads.</p>

</div>

<br />

If you enjoy this work and have some extra $s, consider sponsoring <a href="https://github.com/sponsors/holdenk">my OSS work</a>.

This blog does not represent any of my employers, software projects, or foundations I'm a part of.

I am one of the developers of Apache Spark <a href="https://amzn.to/2O6KYYH">and have some books published on the topic</a> <a href="https://amzn.to/2IA6Yf0">(plus a new Kubeflow book)</a> that may influence my views.

Some of the links on this blog may generate an affiliate commission. I also earn royalties from my books.

Generally speaking these do not cover the amount I spend on these adventures, but help defray my hardware, <a href="https://amzn.to/31mIOeK">coffee</a>, and <a href="https://dynamodonut.com/">artisinal doughnut</a> costs.

<a rel="license" href="http://creativecommons.org/licenses/by/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by/4.0/88x31.png" /></a><br />The posts on this blog are licensed under a <a rel="license" href="http://creativecommons.org/licenses/by/4.0/">Creative Commons Attribution 4.0 International License</a>. <br />

<a href="https://github.com/scalingpythonml/scalingpythonml.github.io" class="github-corner" aria-label="View source on GitHub"><svg width="80" height="80" viewBox="0 0 250 250" style="fill:#151513; color:#fff; position: absolute; top: 0; border: 0; right: 0;" aria-hidden="true"><path d="M0,0 L115,115 L130,115 L142,142 L250,250 L250,0 Z"></path><path d="M128.3,109.0 C113.8,99.7 119.0,89.6 119.0,89.6 C122.0,82.7 120.5,78.6 120.5,78.6 C119.2,72.0 123.4,76.3 123.4,76.3 C127.3,80.9 125.5,87.3 125.5,87.3 C122.9,97.6 130.6,101.9 134.4,103.2" fill="currentColor" style="transform-origin: 130px 106px;" class="octo-arm"></path><path d="M115.0,115.0 C114.9,115.1 118.7,116.5 119.8,115.4 L133.7,101.6 C136.9,99.2 139.9,98.4 142.2,98.6 C133.8,88.0 127.5,74.4 143.8,58.0 C148.5,53.4 154.0,51.2 159.7,51.0 C160.3,49.4 163.2,43.6 171.4,40.1 C171.4,40.1 176.1,42.5 178.8,56.2 C183.1,58.6 187.2,61.8 190.9,65.4 C194.5,69.0 197.7,73.2 200.1,77.6 C213.8,80.2 216.3,84.9 216.3,84.9 C212.7,93.1 206.9,96.0 205.4,96.6 C205.1,102.4 203.0,107.8 198.3,112.5 C181.9,128.9 168.3,122.5 157.7,114.1 C157.9,116.9 156.7,120.9 152.7,124.9 L141.0,136.5 C139.8,137.7 141.6,141.9 141.8,141.8 Z" fill="currentColor" class="octo-body"></path></svg></a><style>.github-corner:hover .octo-arm{animation:octocat-wave 560ms ease-in-out}@keyframes octocat-wave{0%,100%{transform:rotate(0)}20%,60%{transform:rotate(-25deg)}40%,80%{transform:rotate(10deg)}}@media (max-width:500px){.github-corner:hover .octo-arm{animation:none}.github-corner .octo-arm{animation:octocat-wave 560ms ease-in-out}}</style>

</div>

</footer>

</body>

</html>

Sunbelt Computer Software

PL/B Language Development and Support

Navigation Menu

Repository navigation

index.html

index.html

Sunbelt Computer Software

PL/B Language Development and Support

FilesExpand file tree

index.html

Latest commit

History

index.html

File metadata and controls