Jekyll2020-10-20T17:48:22+00:00https://vsoch.github.io/lessons/lessons/feed.xmlResearch Computing LessonsQuick lessons to learn about reproducible research computingvsochRebasing in Practical Terms2019-10-08T00:00:00+00:002019-10-08T00:00:00+00:00https://vsoch.github.io/lessons/lessons/git-rebase<p>Rebasing is really useful for creating a clean git history. It technically means
changing the original base commit of a branch, but for this post, I want to talk
about it in practical terms. If you want the technical details, there are
plenty of resources available to you.
First, let’s quickly answer some questions.</p>
<h2 id="when-might-i-want-to-rebase">When might I want to rebase?</h2>
<p>Generally, when you are working on a branch and want to clean up your git history,
it’s easy to rebase with master.</p>
<h2 id="what-are-the-general-steps">What are the general steps?</h2>
<p>While this isn’t a copy paste and go example, it will walk you through general steps.
We generally start with checkout of a new branch from an updated master:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># checkout master</span>
git checkout master
<span class="c"># make sure your upstream master is up to date</span>
git pull origin master
<span class="c"># now checkout a new branch</span>
git checkout <span class="nt">-b</span> add/cool-feature
</code></pre></div></div>
<p>And then we make an awesome change! And we commit it.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># make some changes here, and then commit</span>
git commit <span class="nt">-a</span> <span class="nt">-s</span> <span class="nt">-m</span> <span class="s2">"This is my awesome change"</span>
</code></pre></div></div>
<p>Then we maybe push, and open up a pull request, and are told to make more changes.
Our commits get sloppy.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git push origin add/cool-feature
<span class="c"># opens pull request, request for changes</span>
<span class="c"># each of the below is associated with some mistake/more changes</span>
git commit <span class="nt">-a</span> <span class="nt">-s</span> <span class="nt">-m</span> <span class="s1">'making this other change'</span>
git commit <span class="nt">-a</span> <span class="nt">-s</span> <span class="nt">-m</span> <span class="s1">'oops'</span>
git commit <span class="nt">-a</span> <span class="nt">-s</span> <span class="nt">-m</span> <span class="s1">'oops'</span>
git commit <span class="nt">-a</span> <span class="nt">-s</span> <span class="nt">-m</span> <span class="s1">'whyyyyyy'</span>
</code></pre></div></div>
<p>Then our git history is a mess! So we do an interactive rebase against master.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git rebase <span class="nt">-i</span> master
</code></pre></div></div>
<p>And then change all of the crappy commits to “fixup” to squash and disregard the
squashed commit messages. And we push to the branch with force!</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git push origin add/cool-feature <span class="nt">--force</span>
</code></pre></div></div>
<p>That’s it! Happy rebasing, folks.</p>vsochRebasing is really useful for creating a clean git history. It technically means changing the original base commit of a branch, but for this post, I want to talk about it in practical terms. If you want the technical details, there are plenty of resources available to you. First, let’s quickly answer some questions.SLURM Job Manager, smanage2019-02-01T00:00:00+00:002019-02-01T00:00:00+00:00https://vsoch.github.io/lessons/lessons/smanage<p>Today I want to introduce you to a script, <a href="https://www.github.com/esurface/smanage">smanage.sh</a>
created by <a href="https://www.github.com/esurface">@esurface</a> from Harvard Research Computing.
smanage <strong>manage</strong>s jobs running on a
<strong>s</strong>lurm compute cluster. It was developed in bash to take advantage of the automatic output from the <a href="https://slurm.schedmd.com/pdfs/summary.pdf">slurm programs available on the command line</a>, namely sacct and sbatch.</p>
<h2 id="what-do-i-need-this-for">What do I need this for?</h2>
<p>After you launch a bunch of jobs, it’s hard to keep track of them. How many completed? How many crashed
and failed in a horrific ending of memory exceed exceptions? In a nutshell, smanage enables you to submit and track large batches of jobs beyond the MaxArraySize limit set by slurm.</p>
<h2 id="what-is-a-job-array">What is a job array?</h2>
<p>Wait, but what is a job array anyway? If you are used to submitting jobs on a SLURM cluster, you are probably used to the standard sbatch command:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>sbatch myanalysis.job
</code></pre></div></div>
<p>If you are like me, you’ve probably written some kind of Python/R/Bash or other
script that loops through some set of variables and programmatically
generates and/or submits job files. <a href="https://github.com/vsoch/image-comparison-thresholding/blob/master/preprocessing/run_make_group_maps.py#L19">Here is an example</a>.
of some of the nonsense that I (contributor @vsoch) went through in graduate school.
If only I had known about job arrays!</p>
<blockquote>
<p>a job array lets you submit a ton of similar jobs using a template script.</p>
</blockquote>
<p>Actually, it’s just another SBATCH header. It looks like this:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#A job array with index values of 1, 2, 5, 19, 27:</span>
<span class="c">#SBATCH --array=1,2,5,19,27</span>
</code></pre></div></div>
<p>How would we use this? Let’s start with a simple example, and say that we have
100 text files to process. We have them in a folder, and they are labeled
cookie1.txt through cookie100.txt. We could use arrays to process these files
without any extraneous for loops:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/bin/bash</span>
<span class="c">#SBATCH -J cookies-job # A single job name for the array</span>
<span class="c">#SBATCH -n 1 # One Core</span>
<span class="c">#SBATCH -N 1 # All cores on one machine</span>
<span class="c">#SBATCH -p owners # Partition name</span>
<span class="c">#SBATCH --mem 2000 # Memory (2Gb)</span>
<span class="c">#SBATCH -t 0-1:00 # Maximum execution time (D-HH:MM)</span>
<span class="c">#SBATCH -o cookie_%A_%a.out # Standard output</span>
<span class="c">#SBATCH -e cookie_%A_%a.err # Standard error</span>
<span class="c">#SBATCH --array=1-100 # maps 1 to 100 to SLURM_ARRAY_TASK_ID below</span>
/bin/bash <span class="s2">"</span><span class="k">${</span><span class="nv">SCRATCH</span><span class="k">}</span><span class="s2">/cookies/cookie</span><span class="k">${</span><span class="nv">SLURM_ARRAY_TASK_ID</span><span class="k">}</span><span class="s2">"</span>.txt
</code></pre></div></div>
<p>We would then submit that one job file, and 100 jobs would be run to process our
cookie text files!</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>sbatch cookie-job.sbatch
</code></pre></div></div>
<p>That’s the essense of a job array. It’s actually exactly as it sounds - an array
of jobs.</p>
<h2 id="why-a-tool-like-smanage">Why a tool like smanage?</h2>
<p>Once you launch your jobs, you lose them to some extent because they are all individual.
jobs. There are technically command line ways to interact and control them, but
it’s yet another hard-to-learn thing and (wouldn’t it be nice) if there was a tool
to manage arrays for us?</p>
<p>This is the goal of smanage. Now that you understand, let’s briefly look at usage.
For more detailed examples, see the <a href="https://www.github.com/esurface/smanage">smanage repository</a>.
If you want to learn more about Job arrays, check out this <a href="https://ask.cyberinfrastructure.org/t/what-are-good-uses-for-job-arrays/733/3">AskCI post</a>.</p>
<h2 id="usage">Usage</h2>
<p>You can install the script by adding an alias to the program. First, wget it.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
wget https://raw.githubusercontent.com/esurface/smanage/master/smanage.sh
<span class="nb">chmod </span>u+x smanage.sh <span class="c"># makes it executable</span>
</code></pre></div></div>
<p>Run the following line of code or copy it into the file ‘~/.bashrc’ to make it permanent:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">alias </span><span class="nv">smanage</span><span class="o">=</span><span class="s1">'<pathto>/smanage.sh'</span>
</code></pre></div></div>
<p>smanage has two basic modes described below.</p>
<h2 id="step-1-write-a-dummy-job">Step 1. Write a Dummy Job</h2>
<p>To really see how this works, we need to write a dummy job. Let’s write one
that will simply sleep for a bit, and exit. Here is our sbatch job file,
called <code class="language-plaintext highlighter-rouge">sleepy.sbatch</code>.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/bin/bash</span>
<span class="c">#SBATCH -J sleepy-job</span>
<span class="c">#SBATCH -n 1</span>
<span class="c">#SBATCH -N 1</span>
<span class="c">#SBATCH -p owners</span>
<span class="c">#SBATCH --mem 1000</span>
<span class="c">#SBATCH -t 0-0:03</span>
<span class="c">#SBATCH --array=1-100 # maps 1 to 100 to SLURM_ARRAY_TASK_ID below</span>
<span class="nb">echo</span> <span class="s2">"This is job number </span><span class="k">${</span><span class="nv">SLURM_ARRAY_TASK_ID</span><span class="k">}</span><span class="s2">"</span>
<span class="nb">sleep </span>120
</code></pre></div></div>
<p>We will launch this job like this:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sbatch sleepy.sbatch
</code></pre></div></div>
<p>and then use it to learn about the different commands for smanage below.</p>
<h2 id="report-mode-default">Report Mode (default)</h2>
<p>The reporting mode gives you a summary of your jobs. Here is what we see after
launching sleepy.sbatch</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">[</span>vsochat@sh-ln06 login /scratch/users/vsochat/sleepy]<span class="nv">$ </span>smanage report
Finding <span class="nb">jobs </span>using: /usr/bin/sacct <span class="nt">-XP</span> <span class="nt">--noheader</span>
Found <span class="nb">jobs</span>: 35512888
0 COMPLETED <span class="nb">jobs
</span>0 FAILED <span class="nb">jobs
</span>0 TIMEOUT <span class="nb">jobs
</span>98 RUNNING <span class="nb">jobs
</span>0 PENDING <span class="nb">jobs
</span>2 <span class="nb">jobs </span>with untracked status
</code></pre></div></div>
<p>We could have also used old school squeue:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">[</span>vsochat@sh-ln06 login /scratch/users/vsochat/sleepy]<span class="nv">$ </span>squeue <span class="nt">-u</span> vsochat
JOBID PARTITION NAME USER ST TIME NODES NODELIST<span class="o">(</span>REASON<span class="o">)</span>
35386489_[1-100] owners sleepy-j vsochat PD 0:00 1 <span class="o">(</span>None<span class="o">)</span>
</code></pre></div></div>
<p>How does the first commad work? It parses and summarized the output from sacct.
Here is what the output looks like when I’ve launched a TON of jobs - note that
we truncate the job numbers so it’s readable.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>smanage report
Finding <span class="nb">jobs </span>using: /usr/bin/sacct <span class="nt">-XP</span> <span class="nt">--noheader</span>
Found 1016 <span class="nb">jobs</span>: 36094475,36123253,36160821,36160826,36160831,36160945,36160947,36160950,36161025 <span class="o">(</span>and more<span class="o">)</span>
569 COMPLETED <span class="nb">jobs
</span>0 FAILED <span class="nb">jobs
</span>15 RUNNING <span class="nb">jobs
</span>1 PENDING <span class="nb">jobs</span>
</code></pre></div></div>
<p>Now we can try the command and look specifically for our “sleepy-job” by name.
Let’s run the command again, so we get two sets of sleepy jobs.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>sbatch sleepy.job
<span class="nv">$ </span>smanage report <span class="nt">--sacct</span> <span class="nt">--name</span><span class="o">=</span>sleepy-job
Finding <span class="nb">jobs </span>using: /usr/bin/sacct <span class="nt">-XP</span> <span class="nt">--noheader</span> <span class="nt">--name</span><span class="o">=</span>sleepy-job
Found <span class="nb">jobs</span>: 35386489,35386615
100 COMPLETED <span class="nb">jobs
</span>0 FAILED <span class="nb">jobs
</span>0 TIMEOUT <span class="nb">jobs
</span>0 RUNNING <span class="nb">jobs
</span>51 PENDING <span class="nb">jobs</span>
</code></pre></div></div>
<p>Now we see the first 100 completed, and the second batch are started up!
Under the hood, this is using ` ‘sacct –array’ named “sleepy-job”`.
If you are familiar with sacct, any sacct commands can be added to
this command. For example, to report the jobs ran on a specific date,
we might do this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ smanage report --sacct --name=sleepy-job --starttime=2019-01-07
</code></pre></div></div>
<p>You can also add the ‘–verbose’ flag to add more useful information about the jobs.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ smanage --verbose report --sacct --name=sleepy-job --verbose --state=COMPLETED
</code></pre></div></div>
<h2 id="reporting-errors-and-submitting-jobs">Reporting Errors and Submitting Jobs</h2>
<p>The examples above outline the simple reporting feature of smanage, which is
likely what you will find most useful. For advanced users, know that smanage also
handles reporting errors and customized job submission. For these advanced examples,
or to ask a question or get help with the tool, see <a href="https://www.github.com/esurface/smanage">the smanage repository</a>.</p>
<p>Do you have questions or want to see another tutorial? Please <a href="https://www.github.com/vsoch/lessons/issues">reach out</a>!</p>vsochToday I want to introduce you to a script, smanage.sh created by @esurface from Harvard Research Computing. smanage manages jobs running on a slurm compute cluster. It was developed in bash to take advantage of the automatic output from the slurm programs available on the command line, namely sacct and sbatch.The Cognitive Atlas and NeuroSynth2018-09-23T00:00:00+00:002018-09-23T00:00:00+00:00https://vsoch.github.io/lessons/lessons/cogat-neurosynth<h2 id="summary">Summary</h2>
<p>This dinosaur lesson will cover a high level introduction to NeuroSynth and the
Cognitive Atlas. The lesson content and associated practice exercises (via jupyter
notebooks) are provided.</p>
<h2 id="what-will-i-learn-today">What will I learn today?</h2>
<ol>
<li>What are these tools?</li>
<li>Why would I want to use them?</li>
<li>How do I get started learning?</li>
</ol>
<h2 id="links">Links</h2>
<ul>
<li><a href="https://youtu.be/c07MYFA5xPU">Video Lesson</a> This is the current video lesson (YouTube). Make sure to watch in HD</li>
<li><a href="https://github.com/vsoch/dinosaur-lessons/blob/master/neuro/notebook/cognitive-atlas-intro.ipynb">Notebook Lesson</a> A quick walk through of using the cognitiveatlas api.</li>
<li><a href="https://github.com/vsoch/dinosaur-lessons/blob/master/neuro/terms.md">Terms</a> that are important to remember from the lesson.</li>
</ul>vsochSummarySingularity Quick Start2018-08-27T00:00:00+00:002018-08-27T00:00:00+00:00https://vsoch.github.io/lessons/lessons/singularity-quickstart<p>This is a quick start to using Singularity on the Sherlock cluster. You should have familiarity
with how to log in, and generally use a command line. If you run into trouble, please
<a href="https://www.github.com/vsoch/lessons/issues" target="_blank">ask us for help</a>.</p>
<h2 id="login">Login</h2>
<p><a href="https://www.sherlock.stanford.edu/docs/getting-started/connecting/" target="_blank">Here are</a>
the official “How to Connect to Sherlock” docs. Generally you
<a href="https://vsoch.github.io/lessons/kerberos/" target="_blank">set up kerberos</a> and do this:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ssh <username>@login.sherlock.stanford.edu
</code></pre></div></div>
<p>I’m lazy, so my preference is to define this setup in my <code class="language-plaintext highlighter-rouge">~/.ssh/config</code> file. By setting
the login node to be a specific one I can maintain a session over time. Let’ say my
username is “tacos”</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
Host sherlock
User tacos
Hostname sh-ln05.stanford.edu
GSSAPIDelegateCredentials <span class="nb">yes
</span>GSSAPIAuthentication <span class="nb">yes
</span>ControlMaster auto
ControlPersist <span class="nb">yes
</span>ControlPath ~/.ssh/%l%r@%h:%p
</code></pre></div></div>
<p>Then to login I don’t need to type the longer string, I can just type:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ssh sherlock
</code></pre></div></div>
<h2 id="interactive-node">Interactive Node</h2>
<p>You generally shouldn’t run anything computationally intensive on the login nodes! It’s not
just to be courteous, it’s because the processes will be killed and you have to start over.
Don’t waste your time doing this, grab an interactive node to start off:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># interactive node</span>
sdev
<span class="c"># same thing, ask for different memory or time</span>
srun <span class="nt">--time</span> 8:00:00 <span class="nt">--mem</span> 32000 pty bash
</code></pre></div></div>
<p>If your PI has <a href="https://srcc.stanford.edu/sherlock-high-performance-computing-cluster" target="_blank">purchased nodes</a>
and you have a partition for your lab,
this means that your jobs will run a lot faster (you get priority as a member of the group!) and you
should use that advantage:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>srun --time 8:00:00 --mem 32000 --partition mygroup --pty bash
</code></pre></div></div>
<p>If you don’t have any specific partition, the implied one is <code class="language-plaintext highlighter-rouge">--partition normal</code>.</p>
<h2 id="load-singularity">Load Singularity</h2>
<p>Let’s get Singularity set up! You may want to add this to your bash profile (in HOME)
under <code class="language-plaintext highlighter-rouge">$HOME/.bashrc</code> (or similar depending on your shell) so that you don’t need to manually
do it.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
module use system
module load singularity
</code></pre></div></div>
<p>A very import variable to export is the <code class="language-plaintext highlighter-rouge">SINGULARITY_CACHEDIR</code>. This is where Singularity stores
pulled images, built images, and image layers (e.g., when you pull a Docker image it first pulls
the <code class="language-plaintext highlighter-rouge">.tar.gz</code> layers before assembling into a container binary). What happens if you <strong>don’t</strong> export
this variable? The cache defaults to your HOME, your HOME quickly gets filled up (containers
are large files!) and then you are locked out.</p>
<blockquote>
<blockquote>
<p>don’t do that.</p>
</blockquote>
</blockquote>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="nb">export </span><span class="nv">SINGULARITY_CACHEDIR</span><span class="o">=</span><span class="nv">$SCRATCH</span>/.singularity
<span class="nb">mkdir</span> <span class="nt">-p</span> <span class="nv">$SINGULARITY_CACHEDIR</span>
</code></pre></div></div>
<p>Also note for the above that we are creating the directory, which is something that needs to be
done once (and then never again!).</p>
<h1 id="examples">Examples</h1>
<p>Now let’s jump into examples. I’m just going to show you, because many of the commands
speak for themselves, and I’ll add further note if needed. What you should know that we
are going to reference a container with a “uri” (unique resource identifier) and it can
be in reference to (the most common forms):</p>
<ul>
<li><strong>docker://</strong>: a container on Docker Hub</li>
<li><strong>shub://</strong>: a container on Singularity Hub</li>
<li><strong>container.simg</strong>: a container image file</li>
</ul>
<h2 id="pull">Pull</h2>
<p>You <strong>could</strong> just run or execute a command to a container reference, but I recommend
that you pull the container first.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
singularity pull shub://vsoch/hello-world
singularity run <span class="nv">$SCRATCH</span>/.singularity/vsoch-hello-world-master-latest.simg
</code></pre></div></div>
<p>It’s common that you might want to name a container based on it’s Github commit (for Singularity Hub),
it’s image file hash, or a custom name that you really like.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
singularity pull <span class="nt">--name</span> meatballs.simg shub://vsoch/hello-world
singularity pull <span class="nt">--hash</span> shub://vsoch/hello-world
singularity pull <span class="nt">--commit</span> shub://vsoch/hello-world
</code></pre></div></div>
<h2 id="exec">Exec</h2>
<p>The “exec” command will execute a command to a container.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
singularity pull docker://vanessa/salad
singularity <span class="nb">exec</span> <span class="nv">$SCRATCH</span>/.singularity/salad.simg /code/salad spoon
singularity <span class="nb">exec</span> <span class="nv">$SCRATCH</span>/.singularity/salad.simg /code/salad fork
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
# Other options for what you can do (without sudo)
singularity build container.simg docker://ubuntu
singularity exec container.simg echo "Custom commands"
</code></pre></div></div>
<h2 id="run">Run</h2>
<p>The run command will run the container’s runscript, which is the executable (or <code class="language-plaintext highlighter-rouge">ENTRYPOINT</code>/<code class="language-plaintext highlighter-rouge">CMD</code>
in Docker speak) that the creator intended to be used.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
singularity pull docker://vanessa/pokemon
singularity run <span class="nv">$SCRATCH</span>/.singularity/pokemon.simg run catch
</code></pre></div></div>
<h2 id="options">Options</h2>
<p>And here is a quick listing of other useful commands! This is by no means
an exaustive list, but I’ve found it helpful to debug and understand a container.
First, inspect things!</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
singularity inspect <span class="nt">-l</span> <span class="nv">$SCRATCH</span>/.singularity/pokemon.simg <span class="c"># labels</span>
singularity inspect <span class="nt">-e</span> <span class="nv">$SCRATCH</span>/.singularity/pokemon.simg <span class="c"># environment</span>
singularity inspect <span class="nt">-r</span> <span class="nv">$SCRATCH</span>/.singularity/pokemon.simg <span class="c"># runscript</span>
singularity inspect <span class="nt">-d</span> <span class="nv">$SCRATCH</span>/.singularity/pokemon.simg <span class="c"># Singularity recipe definition</span>
</code></pre></div></div>
<p>Using <code class="language-plaintext highlighter-rouge">--pwd</code> below might be necessary if the working directory is important.
Singularity doesn’t respect Docker’s <code class="language-plaintext highlighter-rouge">WORKDIR</code> Dockerfile
command, as we usually use the present working directory.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="c"># Set the present working directory with --pwd when you run the container</span>
singularity run <span class="nt">--pwd</span> /code container.simg
</code></pre></div></div>
<p>Singularity is great because most of the time, you don’t need to think about
binds. The paths that you use most often (e.g., your home and scratch and tmp)
are “inside” the container. I hesitate to use that word because the
boundry really is seamless. Thus, if your host supports
<a href="https://en.wikipedia.org/wiki/OverlayFS" target="_blank">overlayfs</a> and the configuration allows it, your container
will by default see all the bind mounts on the host. You can specify a custom mount
(again if the administrative configuration allows it) with <code class="language-plaintext highlighter-rouge">-B</code> or <code class="language-plaintext highlighter-rouge">--bind</code>.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="c"># scratch, home, tmp, are already mounted :) But use --bind/-B to do custom</span>
singularity run <span class="nt">--bind</span> <span class="nv">$HOME</span>/pancakes:/opt/pancakes container.simg
</code></pre></div></div>
<p>I many times have conflicts with my PYTHONPATH and need to unset it,
or even just clean the environment entirely.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="nv">PYTHONPATH</span><span class="o">=</span> singularity run docker://continuumio/miniconda3 python
singularity run <span class="nt">--cleanenv</span> docker://continuumio/miniconda3 python
</code></pre></div></div>
<p>What does writing a script look like? You are going to treat singularity as
you would any other executable! It’s good practice to load it in your
<a href="https://researchapps.github.io/job-maker/" target="_blank">SBATCH</a>
scripts too.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
module use system
module load singularity
singularity run <span class="nt">--cleanenv</span> docker://continuumio/miniconda3 python <span class="nv">$SCRATCH</span>/analysis/script.py
</code></pre></div></div>
<h2 id="build">Build</h2>
<p>If you build your own container, you have a few options!</p>
<ul>
<li>Created an <a href="https://docs.docker.com/docker-hub/builds/">automated build for Docker Hub</a> then pull the container via the <code class="language-plaintext highlighter-rouge">docker://</code> uri.</li>
<li>add a <code class="language-plaintext highlighter-rouge">Singularity</code> recipe file to a Github repository and build it automatically on <a href="https://www.singularity-hub.org">Singularity Hub</a>.</li>
<li><a href="https://www.github.com/researchapps/sherlock">Ask for @vsoch help</a> to build a custom container!</li>
<li>Check out all the <a href="https://github.com/researchapps/sherlock">example</a> containers to get you started.</li>
<li><a href="https://vsoch.github.io/containershare/">Browse the Containershare</a> and use the <a href="https://www.github.com/vsoch/forward/">Forward tool</a> for interactive jupyter (and other) notebooks.</li>
<li><a href="https://www.sylabs.io/guides/2.6/user-guide/quick_start.html#build-images-from-scratch">Build your container</a> locally, and then use <a href="http://www.hypexr.org/linux_scp_help.php">scp</a> to copy it to Sherlock:</li>
</ul>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>scp container.simg <username>@login.sherlock.stanford.edu:/scratch/users/<username>/container.simg
</code></pre></div></div>
<p>Have a question? Or want to make these notes better? Please <a href="https://www.github.com/vsoch/lessons/issues">open an issue</a>.</p>vsochThis is a quick start to using Singularity on the Sherlock cluster. You should have familiarity with how to log in, and generally use a command line. If you run into trouble, please ask us for help.Containershare: Cluster Notebooks2018-08-09T00:00:00+00:002018-08-09T00:00:00+00:00https://vsoch.github.io/lessons/lessons/containershare<p>Let’s set up our story. You are a graduate student, a postdoc, or staff in lab, and you want
to use Sherlock, Farmshare, or one of the other Stanford clusters to run an interactive notebook.
This might mean Jupyter notebook, or even a notebook with Julia, or R. This tutorial will help teach
you how to do that.</p>
<h2 id="what-are-we-doing-today">What are we doing today?</h2>
<p>Today we are going to do the following:</p>
<ol>
<li>find a container that we like on the <a href="https://vsoch.github.io/containershare">containershare</a>.
<ul>
<li>if you don’t find the right one, you can <a href="https://www.github.com/vsoch/containershare/issues">ask @vsoch to build it</a> or <a href="https://github.com/vsoch/containershare#contribute-a-container">build it yourself</a> from a template.</li>
</ul>
</li>
<li>configure the <a href="https://github.com/vsoch/forward">forward</a> tool for your cluster</li>
<li>use the <a href="https://www.github.com/vsoch/forward">forward tool</a> to start an interactive (or regular ssh) session</li>
<li>access the notebook in a browser on your local machine, or shell into the node.</li>
</ol>
<p>Watch the above in action in the video below, and keep reading to try it out!</p>
<script src="https://asciinema.org/a/195889.js" id="asciicast-195889" async=""></script>
<hr />
<h2 id="step-1-find-a-container">Step 1: Find a Container</h2>
<p>The purpose of the <a href="https://vsoch.github.io/containershare">containershare</a> is so that @vsoch can easily provide version controlled, pre-built containers that are ready to go for you! In fact, the containershare is not only a <a href="https://vsoch.github.io/containershare">table on the web</a>, it’s (on Stanford servers)
an actual folder of the shared containers (hence the container share!) that is looked at before other avenues.</p>
<blockquote>
<p>How does it work?</p>
</blockquote>
<p>The containers in that table are provided on the Stanford clusters defined for the tool under @vsoch scratch, named by their hash. Here is a peek:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="nv">$ </span><span class="nb">ls</span> /scratch/users/vsochat/share <span class="nt">-1</span>
-
010188b95ce3e1b29e133901729f88b5.simg
053d59c24daa2aa73ad3512a007dd408.simg
0b44a706bda7a87b89013ad1e2760190.simg
11a772667fd0712bdf17d8cb52f89cd4.simg
12591da722912b13dfd32da0afe3b1ae.simg
...
</code></pre></div></div>
<p>When you ask for a container, we use the name of the hash to find the file in the containershare! This way, if you
are using a container provided in the table in the web interface (or custom built for you by @vsoch) you don’t need to
pull or download or build anything. This is helpful because in some cases, the pull doesn’t work as expected on Sherlock.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="nv">$ CONTAINER</span><span class="o">=</span>docker://vanessa/repo2docker-julia:9ea83b39fd
<span class="nv">$ CONTAINER_NAME</span><span class="o">=</span><span class="si">$(</span><span class="nb">echo</span> <span class="nt">-n</span> <span class="s2">"</span><span class="k">${</span><span class="nv">CONTAINER</span><span class="k">}</span><span class="s2">"</span> | <span class="nb">md5sum</span> | <span class="nb">awk</span> <span class="s1">'{ print $1 }'</span><span class="si">)</span>.simg
<span class="nv">$ </span><span class="nb">echo</span> <span class="nv">$CONTAINER_NAME</span>
c92c3c3f3f9045bcd460466c952c1484.simg
</code></pre></div></div>
<p>In the above, the container on Docker Hub called <code class="language-plaintext highlighter-rouge">vanessa/repo2docker-julia</code> with tag <code class="language-plaintext highlighter-rouge">9ea83b39fd</code> is named <code class="language-plaintext highlighter-rouge">c92c3c3f3f9045bcd460466c952c1484.simg</code> on the cluster containershare. We’ll see how this is found later in this tutorial.</p>
<blockquote>
<p>How do we find the container we want?</p>
</blockquote>
<p>For this example today, we are going to be using the container in the table called <code class="language-plaintext highlighter-rouge">repo2docker-julia</code>. It’s linked from the containershare table and has <a href="https://vsoch.github.io/repo2docker-julia/" target="_blank">its own table here</a>. If you wanted to learn about the guts of the container:</p>
<ul>
<li>the “Manifest” column will show you image manifests with the commands used to build it</li>
<li>the “Inspect” column will show you packages installed, along with a complete file listing.</li>
</ul>
<p>See the <a href="https://www.github.com/vsoch/repo2docker-julia" target="_blank">the container’s Github repository</a> if you
want to learn about specific usage and instructions for the container, or <a href="https://vsoch.github.io/2018/build-deploy-docs/" target="_blank">this post</a> on the containershare if you want more detail on how the container is built, and the metadata and interface generated.</p>
<blockquote>
<p>How do we reference the container we want?</p>
</blockquote>
<p>We want to grab the “unique resource identifier” (called the “uri”) of the container, which is <code class="language-plaintext highlighter-rouge">docker://vanessa/repo2docker-julia:9ea83b39fd</code>. This is in reference to an
<a href="https://hub.docker.com/r/vanessa/repo2docker-julia/" target="_blank">image on Docker Hub</a> that has a particular tag, and the tag also corresponds to a Github commit.</p>
<blockquote>
<p>What’s wrong with “latest”?</p>
</blockquote>
<p>The tag “latest” is easy to remember, but in that it <em>always</em> corresponds to the latest build, the “latest” that you get today may not be the same container that is tagged with “latest” tomorrow. For the sake of reproducibility, do <strong>not</strong> use “latest.”</p>
<blockquote>
<p>The chosen one!</p>
</blockquote>
<p>Let’s choose <code class="language-plaintext highlighter-rouge">vanessa/repo2docker-julia:9ea83b39fd</code>. This is a vanilla jupyter+julia container that you could use as a template for your
own Jupyter+julia notebook! Want to skip Sherlock all together? You can do this too :) Here is how to run the container with
Docker, on your local machine:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker run <span class="nt">--rm</span> <span class="nt">-it</span> <span class="nt">-p</span> 8888:8888 vanessa/repo2docker-julia jupyter notebook <span class="nt">--ip</span> 0.0.0.0
</code></pre></div></div>
<p>Honestly? If I am using a notebook, this is my preference. The harsh reality is that setting up ssh forwarding, on an interactive
node that you have to wait for, with a large number of dependencies with respect to libraries that are needed, is really hard.
We can only do our best to provide tools that abstract this away. But I understand that this is needed, and so until
we have a different kind of cluster that makes this easy, I (and your research computing team) are here to help you, to the best that we can. Let’s continue!</p>
<h2 id="step-2-download-the-repository">Step 2: Download the Repository</h2>
<p>Next, we need the <a href="https://github.com/vsoch/forward" target="_blank">forward tool</a> to run our container! Clone the repository, and remember the location. You will generate a parameters flie <code class="language-plaintext highlighter-rouge">params.sh</code> that you can go back to and use again.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
git clone https://github.com/vsoch/forward
<span class="nb">cd </span>forward
</code></pre></div></div>
<p>Take a look at the files we have to work with (I’ve removed documentation markdown ones):</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="nv">$ </span>tree <span class="nt">-L</span> 1
<span class="nb">.</span>
├── end.sh <span class="o">(</span><span class="nt">--</span> end a session
├── resume.sh <span class="o">(</span><span class="nt">--</span> resume a started session
├── sbatches <span class="o">(</span><span class="nt">--</span> batch scripts you can run to forward a port to, organized by cluster
├── helpers.sh <span class="o">(</span><span class="nt">--</span> just helper functions
├── hosts <span class="o">(</span><span class="nt">--</span> generate configurations <span class="k">for </span>your ~/.ssh/config
├── setup.sh <span class="o">(</span><span class="nt">--</span> run once to <span class="nb">set </span>up the tool
├── start-node.sh <span class="o">(</span><span class="nt">--</span> start a node <span class="o">(</span>but without ssh forwarding / interactive notebook <span class="o">)</span>
└── start.sh <span class="o">(</span><span class="nt">--</span> how you start a connection
</code></pre></div></div>
<p>For Jupyter, since we want an interactive session, we are first going to generate a file called <code class="language-plaintext highlighter-rouge">params.sh</code>
that is specific to a cluster we are using (e.g., sherlock). We are then going to use <code class="language-plaintext highlighter-rouge">start.sh</code> with the uri of the
container we found above and a batch script for slurm in the <code class="language-plaintext highlighter-rouge">sbatches/sherlock</code> folder to launch our notebook.
Let’s do that.</p>
<h2 id="step-3-generate-your-parameters">Step 3: Generate your Parameters</h2>
<p>Run the <code class="language-plaintext highlighter-rouge">setup.sh</code> file to generate your parameters, which will ask you questions about the cluster
configuration that you want.</p>
<p><strong>partition</strong></p>
<p>If you are on Sherlock and want your libraries to have access to a gpu,
then make sure to select “gpu” for a partition. Otherwise, your PI’s parition or the default of “normal”
is ok. In the example below, I choose the “russpold” partition that corresponds to my lab
in graduate school, just as an example.</p>
<p><strong>resource</strong></p>
<p>This is the resource identifier. Notice that I’m just pushing enter through the
first prompt to get the default “sherlock.”</p>
<p><strong>containershare</strong></p>
<p>This is a new addition added to the forward tool. If you have a <a href="https://github.com/vsoch/containershare#deploy-a-containershare">containershare
installed on your cluster</a>,
meaning a folder of shared containers (either by you or a cluster administrator)
you can use a <code class="language-plaintext highlighter-rouge">containershare-notebook.sbatch</code> (or similar) script to
get direct access to these containers. If you don’t care about containershare, just press enter through
this setting.</p>
<p>We are going to do that in the tutorial today, and will show more detail. The <code class="language-plaintext highlighter-rouge">CONTAINERSHARE</code> variable basically
refers to this table! If you are on Sherlock at Stanford,
leave it to the default (<code class="language-plaintext highlighter-rouge">/scratch/users/vsochat/share</code>) and you’ll get the containers
<a href="https://vsoch.github.io/containershare">in the table</a>. You can also
<a href="https://github.com/vsoch/containershare/tree/master/scripts">create your own</a>, whether that means
downloading an existing one, or deploying a new interface and containers. But let’s
not get sidetracked!</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="nv">$ </span>bash setup.sh
First, choose the resource identifier that specifies your cluster resoure. We
will <span class="nb">set </span>up this name <span class="k">in </span>your ssh configuration, and use it to reference the resource <span class="o">(</span>sherlock<span class="o">)</span><span class="nb">.</span>
Resource identifier <span class="o">(</span>default: sherlock<span class="o">)</span> <span class="o">></span>
sherlock username <span class="o">></span> vsochat
Next, pick a port to use. If someone <span class="k">else </span>is port forwarding using that
port already, this script will not work. If you pick a random number <span class="k">in </span>the
range 49152-65335, you should be good.
Port to use <span class="o">></span> 56845
Next, pick the sherlock partition on which you will be running your
notebooks. If your PI has purchased dedicated hardware on sherlock, you can use
that partition. Otherwise, leave blank to use the default partition <span class="o">(</span>normal<span class="o">)</span><span class="nb">.</span>
sherlock partition <span class="o">(</span>default: normal<span class="o">)</span> <span class="o">></span> russpold
A containershare <span class="o">(</span>https://vsoch.github.io/containershare is a library of
containers that are prebuilt <span class="k">for </span>you, and provided on your cluster resource. <span class="k">if </span>you
are at Stanford, leave this to be the default. If not, ask your HPC administrator
about setting one up, and direct them to https://www.github.com/vsoch/containershare.
container shared folder <span class="o">(</span>default <span class="k">for </span>Stanford: /scratch/users/vsochat/share<span class="o">)</span> <span class="o">></span>
</code></pre></div></div>
<p>That’s it! The above produced the following details in “params.sh”</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="nv">$ </span><span class="nb">cat </span>params.sh
<span class="nv">USERNAME</span><span class="o">=</span><span class="s2">"vsochat"</span>
<span class="nv">PORT</span><span class="o">=</span><span class="s2">"56845"</span>
<span class="nv">PARTITION</span><span class="o">=</span><span class="s2">"russpold"</span>
<span class="nv">RESOURCE</span><span class="o">=</span><span class="s2">"sherlock"</span>
<span class="nv">MEM</span><span class="o">=</span><span class="s2">"20G"</span>
<span class="nv">TIME</span><span class="o">=</span><span class="s2">"8:00:00"</span>
<span class="nv">CONTAINERSHARE</span><span class="o">=</span><span class="s2">"/scratch/users/vsochat/share"</span>
</code></pre></div></div>
<p>This is just a text file with variables that you are free to tweak.</p>
<h2 id="step-4-ssh-credentials">Step 4: SSH Credentials</h2>
<p>SSH credentials are defined in the file <code class="language-plaintext highlighter-rouge">~/.ssh/config</code>, and will make it easy to connect to the cluster(s) of
your choosing. You can look at this file on your computer (if it exists) to see what is set up already.
Since I regularly use farmshare and sherlock, I have them defined:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="nv">$ </span><span class="nb">cat</span> ~/.ssh/config
Host sherlock
User vsochat
Hostname sh-ln06.stanford.edu
GSSAPIDelegateCredentials <span class="nb">yes
</span>GSSAPIAuthentication <span class="nb">yes
</span>ControlMaster auto
ControlPersist <span class="nb">yes
</span>ControlPath ~/.ssh/%l%r@%h:%p
Host farmshare
User vsochat
Hostname rice.stanford.edu
GSSAPIDelegateCredentials <span class="nb">yes
</span>GSSAPIAuthentication <span class="nb">yes
</span>ControlMaster auto
ControlPersist <span class="nb">yes
</span>ControlPath ~/.ssh/%l%r@%h:%p
</code></pre></div></div>
<p>If you don’t have the file, or want to update your configuration, the forward tools provides
a set of scripts that will again prompt you for your username, and then spit out a configuration.
Take a look in the <code class="language-plaintext highlighter-rouge">hosts</code> folder:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="nb">ls </span>hosts/
sherlock_ssh.sh
farmshare_ssh.sh
</code></pre></div></div>
<p>You can run either of these scripts to print a configuration to your screen, and the configuration you can
then add to your <code class="language-plaintext highlighter-rouge">~/.ssh/config</code>.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="nv">$ </span>bash hosts/sherlock_ssh.sh
Sherlock username <span class="o">></span> tacocat
Host sherlock
User tacocat
Hostname sh-ln03.stanford.edu
GSSAPIDelegateCredentials <span class="nb">yes
</span>GSSAPIAuthentication <span class="nb">yes
</span>ControlMaster auto
ControlPersist <span class="nb">yes
</span>ControlPath ~/.ssh/%l%r@%h:%p
</code></pre></div></div>
<p>This comes down to opening the file in a text editor, and then copy pasting,
and you are allowed to have more than one configuration in the file, as I do above. Once they are set, you can
shell into either cluster very easily with commands like:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
ssh sherlock
ssh farmshare
</code></pre></div></div>
<p>The first time will prompt you for credentials, but subsequent sessions will not (see how <code class="language-plaintext highlighter-rouge">ControlPersist</code> is set to yes?). If
you don’t have a file, period, then you can generate the entire thing programatically like this:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bash hosts/sherlock_ssh.sh <span class="o">>></span> ~/.ssh/config
</code></pre></div></div>
<h2 id="step-5-usage">Step 5. Usage</h2>
<p>That’s it! We are now ready to grab and use our container from containershare to launch a notebook!
As a reminder, in the tutorial below, we are going to walk through using Jupyter notebook with Julia.</p>
<h3 id="the-command">The Command</h3>
<p>The basic command format looks like this:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bash start.sh <sbatch> <container>
</code></pre></div></div>
<p>Let’s break it down, like a bad disco dance.</p>
<h3 id="the-job-submission-script-sbatch">The Job Submission Script, “sbatch”</h3>
<p>The <code class="language-plaintext highlighter-rouge"><sbatch></code> in the example above refers to an sbatch script in our “sbatches” folder, let’s take a look at what we have (at the writing of
this tutorial).</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="nv">$ </span>tree sbatches/
sbatches/
├── farmshare <span class="o">(</span><span class="nt">--</span> scripts intended <span class="k">for</span> <span class="s2">"farmshare"</span>
│ ├── singularity-exec.sbatch
│ ├── singularity-jupyter.sbatch
│ └── singularity-run.sbatch
├── sherlock <span class="o">(</span><span class="nt">--</span> scripts intended <span class="k">for </span>sherlock
│ ├── containershare-notebook.sbatch <span class="o">(</span><span class="nt">--</span> the same as singularity-notebook, but checks containershare folder first
│ ├── jupyter.sbatch
│ ├── py2-jupyter.sbatch
│ ├── py2-tensorflow.sbatch
│ ├── py3-jupyter.sbatch
│ ├── py3-tensorflow.sbatch
│ ├── singularity-exec.sbatch
│ ├── singularity-jupyter.sbatch
│ ├── singularity-notebook.sbatch
│ ├── singularity-run.sbatch
│ └── tensorboard.sbatch
├── singularity-exec.sbatch <span class="o">(</span><span class="nt">--</span> scripts that work across clusters
└── singularity-run.sbatch
</code></pre></div></div>
<p>Woweee, many more than with the original release of the tool! Since we are interested in Sherlock,
and we are going to use a containershare container, we are going to use this one:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>│ ├── containershare-notebook.sbatch
</code></pre></div></div>
<p>It’s actually the same as <code class="language-plaintext highlighter-rouge">singularity-notebook</code> but does a check first in our cluster shared folder for
the image binary. If it’s already downloaded (because @vsoch has provided it for you!) we don’t need to wait
for a pull. If you want to use a general notebook (that isn’t in the containershare) but is accessible via
a docker uri (e.g., <code class="language-plaintext highlighter-rouge">docker://<username>/<repository></code> then you would want to look at once of these scripts:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="nv">$ </span><span class="nb">ls </span>sbatches/sherlock/sing<span class="k">*</span> <span class="nt">-1</span>
sbatches/sherlock/singularity-exec.sbatch
sbatches/sherlock/singularity-jupyter.sbatch
sbatches/sherlock/singularity-notebook.sbatch
sbatches/sherlock/singularity-run.sbatch
</code></pre></div></div>
<p>These are the names that we need <sup><a href="#myfootnote1">1</a></sup>. In fact, the names above <strong>without</strong> the extension of sbatch is how we are going to call the command! The script we want is going to work for a containershare notebook <code class="language-plaintext highlighter-rouge">sbatches/sherlock/containershare-notebook.sbatch</code> and we refer to it like this:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bash start.sh sherlock/containershare-notebook <container>
</code></pre></div></div>
<p>By the way, if you ever want to look at the script, usage, or other, just look at the script you are about to
run! You should do this anyway :)</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cat </span>sbatches/sherlock/containershare-notebook.sbatch | less
</code></pre></div></div>
<p>The “less” ensures that you can start at the top of the file, and press ENTER to move
down it. If you are using vim, you can type q to quit.</p>
<h3 id="the-container">The Container</h3>
<p>We are almost there! Let’s just plug in our container now. The final detail is that since this command is expecting a Singularity container (it could reference a full path to a Singularity container on your scratch, for example) we need to tell the software that
it’s a Docker container, and use a docker uri (<code class="language-plaintext highlighter-rouge">docker://</code>)</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>bash start.sh sherlock/singularity-notebook docker://vanessa/repo2docker-julia:9ea83b39fd
</code></pre></div></div>
<p>Let’s walk through the output you see on the screen after issuing the command.</p>
<h3 id="the-submission">The Submission</h3>
<p>At this point you’ve used <code class="language-plaintext highlighter-rouge">setup.sh</code> to create <code class="language-plaintext highlighter-rouge">params.sh</code> and you are launching the container
to create an interactive notebook on the Sherlock cluster. Let’s walk you through the commands
that are printed to the screen, because they are added intentionally to help you.</p>
<h4 id="1-finding-the-slurm-batch-script">1. Finding the SLURM batch script</h4>
<p>The first thing that happens is finding the script that you want, on your local machine!
Since there are several different levels for a script <sup><a href="#myfootnote2">2</a></sup>
the first few lines tell you the script being looked for, and when a match is found,
you are told via the line that starts with <code class="language-plaintext highlighter-rouge">Script</code>:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="nv">$ </span>bash start.sh sherlock/containershare-notebook docker://vanessa/repo2docker-julia:9ea83b39fd
<span class="o">==</span> Finding Script <span class="o">==</span>
Looking <span class="k">for </span>sbatches/sherlock/sherlock/containershare-notebook.sbatch
Looking <span class="k">for </span>sbatches/sherlock/containershare-notebook.sbatch
Script sbatches/sherlock/containershare-notebook.sbatch
</code></pre></div></div>
<p>The script <code class="language-plaintext highlighter-rouge">sbatches/sherlock/containershare-notebook.sbatch</code> will be used.</p>
<h4 id="2-check-for-running-jobs">2. Check for running jobs</h4>
<p>Next, we would want to check for an already existing notebook. If one is found, it would
tell you to end it first.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="o">==</span> Checking <span class="k">for </span>previous notebook <span class="o">==</span>
No existing sherlock/containershare-notebook <span class="nb">jobs </span>found, continuing...
</code></pre></div></div>
<h4 id="3-transfer-the-script-to-the-cluster">3. Transfer the script to the cluster</h4>
<p>The destination directory is a folder in your $HOME called <code class="language-plaintext highlighter-rouge">forward-util</code> where
the sbatch script will be transferred. It needs to be on the cluster before it can
be launched for use.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="o">==</span> Getting destination directory <span class="o">==</span>
<span class="o">==</span> Uploading sbatch script <span class="o">==</span>
containershare-notebook.sbatch 100% 2907 2.8KB/s 00:00
</code></pre></div></div>
<p>If you hadn’t authenticated yet, it would happen during the above step.</p>
<h4 id="4-submission-command">4. Submission Command</h4>
<p>Next (and this is very important
for your learning, or future you that is submitting jobs on the cluster) the tool <strong>shows</strong> you
the command that it uses to submit your script:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">==</span> Submitting sbatch <span class="o">==</span>
sbatch <span class="nt">--job-name</span><span class="o">=</span>sherlock/containershare-notebook <span class="nt">--partition</span><span class="o">=</span>russpold <span class="nt">--output</span><span class="o">=</span>/home/users/vsochat/forward-util/containershare-notebook.sbatch.out <span class="nt">--error</span><span class="o">=</span>/home/users/vsochat/forward-util/containershare-notebook.sbatch.err <span class="nt">--mem</span><span class="o">=</span>20G <span class="nt">--time</span><span class="o">=</span>8:00:00 /home/users/vsochat/forward-util/containershare-notebook.sbatch 56845 <span class="s2">"docker://vanessa/repo2docker-julia:9ea83b39fd"</span>
Submitted batch job 23366437
</code></pre></div></div>
<p><strong>Understand the Command</strong></p>
<p>This is so hugely useful for your learning. Let’s look closely at this command, and understand every piece of it.</p>
<ul>
<li><strong>sbatch</strong> is the executable that comes with <strong>S</strong>LURM to submit a job to the <strong>batch</strong> manager.</li>
<li><strong>job-name</strong> refers to the name of your job so you can refer to it for future commands. We use the name of the script (<code class="language-plaintext highlighter-rouge">sherlock/containershare-notebook</code>) so it’s always linkable, and in fact, this is how we check to see if you have a notebook running already!</li>
<li><strong>partition</strong> is a reference to a named group of machines. If your PI has some funding, he or she can invest in nodes and they get named as a “partition” of the cluster. If you want a gpu, you would specify the parition as “gpu.” The default is “normal.” This setting comes from the <code class="language-plaintext highlighter-rouge">PARITION</code> variable in your <code class="language-plaintext highlighter-rouge">params.sh</code></li>
<li><strong>output</strong> and <strong>error</strong> are exactly what they sound like - full paths to output and error files that you can look at if something goes wrong (or seems to be wrong). This should be one of the first places you look to debug.</li>
<li><strong>mem</strong> The amount of memory requested for the job. This setting comes from the <code class="language-plaintext highlighter-rouge">MEMORY</code> variable in your <code class="language-plaintext highlighter-rouge">params.sh</code></li>
<li><strong>time</strong>: The <code class="language-plaintext highlighter-rouge">HH:MM:SS</code> requested for the job. The default is 8 hours, shown in the command above.</li>
</ul>
<p>There are more details on <a href="https://slurm.schedmd.com/sbatch.html">SLURM’s documentation page</a> if you are interested. The next part of the submission command is the script followed by arguments:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/home/users/vsochat/forward-util/containershare-notebook.sbatch 56845 <span class="s2">"docker://vanessa/repo2docker-julia:9ea83b39fd"</span>
</code></pre></div></div>
<p>This shows us the full path to the same sbatch file that was uploaded (now on the cluster!) and the inputs that we provided.
What’s that weird number in the middle? That’s your <code class="language-plaintext highlighter-rouge">PORT</code>, a variable fed into the script that is also defined in your <code class="language-plaintext highlighter-rouge">params.sh</code>.</p>
<h4 id="5-logs">5. Logs</h4>
<p>Without being empowered to look at logs, you are pretty helpless if something goes wrong. This
is why the tool next prints for you a command you can copy paste (in another terminal from your local
machine) to look at logs. You could also log in to the cluster and view the files, but this might be faster:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="o">==</span> View logs <span class="k">in </span>separate terminal <span class="o">==</span>
ssh sherlock <span class="nb">cat</span> /home/users/vsochat/forward-util/containershare-notebook.sbatch.out
ssh sherlock <span class="nb">cat</span> /home/users/vsochat/forward-util/containershare-notebook.sbatch.err
</code></pre></div></div>
<h4 id="6-waiting-for-allocation">6. Waiting for allocation</h4>
<p>Depending on the load of the cluster and your partition, the allocation of the job to
a node can be speedy, or take a long time. We use a method called <a href="https://en.wikipedia.org/wiki/Exponential_backoff" target="_blank">exponential backoff</a> to keep checking if the node has been allocated.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="o">==</span> Waiting <span class="k">for </span>job to start, using exponential backoff <span class="o">==</span>
Attempt 0: not ready yet... retrying <span class="k">in </span>1..
Attempt 1: not ready yet... retrying <span class="k">in </span>2..
Attempt 2: not ready yet... retrying <span class="k">in </span>4..
Attempt 3: not ready yet... retrying <span class="k">in </span>8..
Attempt 4: not ready yet... retrying <span class="k">in </span>16..
Attempt 5: not ready yet... retrying <span class="k">in </span>32..
Attempt 6: resources allocated to sh-02-21!..
sh-02-21
sh-02-21
notebook running on sh-02-21
</code></pre></div></div>
<h4 id="7-preparing-the-container">7. Preparing the Container</h4>
<p>Woot! We got the node! After the node is allocated, the container is identified, and the tool tells you the steps it is taking
to find it. You’ll see that we start with a URI, the working directory for our notebook will be <code class="language-plaintext highlighter-rouge">$SCRATCH</code>,
and for a first attempt we are going to look for the container in the containershare, which is located
at <code class="language-plaintext highlighter-rouge">/scratch/users/vsochat/share</code>. The hash value is what we get when we do an md5sum of <code class="language-plaintext highlighter-rouge">docker://vanessa/repo2docker-julia:9ea83b39fd</code>.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="o">==</span> Connecting to notebook <span class="o">==</span>
Container is docker://vanessa/repo2docker-julia:9ea83b39fd
Notebook directory is /scratch/users/vsochat
Attempt 1: Looking <span class="k">for</span> /scratch/users/vsochat/share/c92c3c3f3f9045bcd460466c952c1484.simg
</code></pre></div></div>
<p>What happens if we don’t find it in the containershare? Here is the order of operations for our search:</p>
<ol>
<li>is the identifier that the user provided a direct path to a file?</li>
<li>is it a file in the containershare?</li>
<li>is it a file already in your <code class="language-plaintext highlighter-rouge">SINGULARITY_CACHE</code>?</li>
</ol>
<p>If the identifier isn’t found with any of the above avenues, it is pulled to your <code class="language-plaintext highlighter-rouge">SINGULARITY_CACHEDIR</code>. if the pull fails and we still don’t have a container, then we exit because something is up. If the container needed to be pulled, you might see the pull output print to the console after a pause.</p>
<h4 id="8-port-forwarding">8. Port Forwarding</h4>
<p>Given we have a container, the script is going to set up port forwarding to ensure that the display from the node on Sherlock is forwarded to your computer.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="o">==</span> Setting up port forwarding <span class="o">==</span>
ssh <span class="nt">-L</span> 43453:localhost:43453 sherlock ssh <span class="nt">-L</span> 43453:localhost:43453 <span class="nt">-N</span> sh-02-21 &
...
</code></pre></div></div>
<h4 id="9-connecting-to-the-notebook">9. Connecting to the Notebook</h4>
<p>Connecting means that we have the node, and the job has been issued that will obtain and start the container.</p>
<p><strong>Important</strong> if you need to pull the container, it might not be ready when the port forwarding starts, so you should look
at the logs to determine when it is, and then go to the port.</p>
<p>At this point the script is conservative, and (again) assumes that something might go wrong. In a perfect world, things wouldn’t.
But as researchers we know that bugs happen, and instead of pretending that they don’t, we should be prepared by knowing how
to debug. Singularity isn’t perfect and has issues
with pulling and running sometimes. For this case, the tool (again) tells you exactly how to look at error and output logs,
from a terminal on your local machine:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="o">==</span> View logs <span class="k">in </span>separate terminal <span class="o">==</span>
ssh sherlock <span class="nb">cat</span> /home/users/vsochat/forward-util/singularity-notebook.sbatch.out
ssh sherlock <span class="nb">cat</span> /home/users/vsochat/forward-util/singularity-notebook.sbatch.err
</code></pre></div></div>
<blockquote>
<p>This is how you should debug if there are issues, and send the output to your research computing support for help!</p>
</blockquote>
<p>Finally, you can follow the instructions to open your browser!</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="o">==</span> Instructions <span class="o">==</span>
1. Password, output, and error printed to this terminal? Look at logs <span class="o">(</span>see instruction above<span class="o">)</span>
2. Browser: http://sh-02-21.int:56845/ -> http://localhost:56845/...
3. To end session: bash end.sh sherlock/singularity-notebook
</code></pre></div></div>
<p>Please read them :) Note that for some containers, you will need to change the node address to <code class="language-plaintext highlighter-rouge">localhost</code> where you
have it forwarded. If you don’t see your password? Issue the <code class="language-plaintext highlighter-rouge">ssh</code> command to see it in the log.</p>
<blockquote>
<p>Read the messages that we print to you, my dear padwan!</p>
</blockquote>
<p>For example, I didn’t see my password, so here is the command I used for this job on Sherlock, and I copy pasted this directly from my terminal:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>ssh sherlock <span class="nb">cat</span> /home/users/vsochat/forward-util/containershare-notebook.sbatch.err
<span class="o">[</span>I 14:53:10.546 NotebookApp] Writing notebook server cookie secret to /tmp/jupyter/notebook_cookie_secret
<span class="o">[</span>I 14:53:12.008 NotebookApp] JupyterLab alpha preview extension loaded from /srv/conda/lib/python3.6/site-packages/jupyterlab
<span class="o">[</span>I 14:53:12.008 NotebookApp] JupyterLab application directory is /srv/conda/share/jupyter/lab
<span class="o">[</span>I 14:53:12.017 NotebookApp] nteract extension loaded from /srv/conda/lib/python3.6/site-packages/nteract_on_jupyter
<span class="o">[</span>I 14:53:12.024 NotebookApp] Serving notebooks from <span class="nb">local </span>directory: /scratch/users/vsochat
<span class="o">[</span>I 14:53:12.024 NotebookApp] 0 active kernels
<span class="o">[</span>I 14:53:12.025 NotebookApp] The Jupyter Notebook is running at:
<span class="o">[</span>I 14:53:12.025 NotebookApp] http://sh-02-21.int:56845/?token<span class="o">=</span>4f37bad27eb8015254e00f922774d0af8f3e5f570223bdb0
<span class="o">[</span>I 14:53:12.025 NotebookApp] Use Control-C to stop this server and shut down all kernels <span class="o">(</span>twice to skip confirmation<span class="o">)</span><span class="nb">.</span>
<span class="o">[</span>C 14:53:12.025 NotebookApp]
Copy/paste this URL into your browser when you connect <span class="k">for </span>the first <span class="nb">time</span>,
to login with a token:
http://sh-02-21.int:56845/?token<span class="o">=</span>4f37bad27eb8015254e00f922774d0af8f3e5f570223bdb0&token<span class="o">=</span>4f37bad27eb8015254e00f922774d0af8f3e5f570223bdb0
</code></pre></div></div>
<p>And tada! There it is at the bottom. I would want to copy paste this into my browser <strong>and change the sh-02-21 to be localhost</strong> as instructed by the “Instructions.” That’s it! Then you have your notebook ready to go :) Note that when I click “new” I can
start a new Julia notebook!</p>
<p><img src="/lessons/assets/img/tutorials/julia.png" alt="/lessons/assets/img/tutorials/julia.png" /></p>
<h3 id="different-kinds-of-containers">Different kinds of Containers</h3>
<p>Let’s say that you built your own Docker container that runs some kind of notebook. It’s not on the containershare.
How do you use it? You can use a slight variation of the <code class="language-plaintext highlighter-rouge">containershare-notebook</code> script, a script called
<code class="language-plaintext highlighter-rouge">singularity-notebook</code>.</p>
<p><strong>Docker Hub</strong></p>
<p>An easy solution is to create an automated build (or just push) directly to <a href="https://docs.docker.com/docker-hub/" target="_blank">Docker Hub</a>, and then use an equivalent uri to reference your container.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bash start.sh sherlock/singularity-jupyter docker://<username>/<repository>:<tag>
</code></pre></div></div>
<p><strong>Container Binary Image</strong></p>
<p>You might have already done this, and then pulled the container to the cluster, and now it’s a file on your scratch.
Guess what, you can use that too! On Sherlock we did this:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
module use system
module load singularity
<span class="nb">export </span><span class="nv">SINGULARITY_CACHEDIR</span><span class="o">=</span><span class="nv">$SCRATCH</span>/.singularity
singularity pull <span class="nt">--name</span> julia.simg docker://vanessa/repo2docker-julia:9ea83b39fd
</code></pre></div></div>
<p>And now we have the container <code class="language-plaintext highlighter-rouge">julia.simg</code> in our <code class="language-plaintext highlighter-rouge">$SCRATCH</code>. We can use it with the same tool by specifying the
full path of the container:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bash start.sh sherlock/singularity-notebook /scratch/users/vsochat/.singularity/julia.simg
</code></pre></div></div>
<p><strong>Singularity Hub</strong></p>
<p>If we were building a recipe from Github on <a href="https://www.singularity-hub.org" target="_blank">Singularity Hub</a> we
could also use the <code class="language-plaintext highlighter-rouge">shub://</code> uri:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bash start.sh sherlock/singularity-notebook shub://<username>/<repository>:<tag>
</code></pre></div></div>
<p>Remember that when you are done, you can end the job.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bash end.sh sherlock/singularity-notebook
</code></pre></div></div>
<p>This is all under development, and guided by your needs! If you have an issue, I’ll help you fix it. If you have an idea for a new container, I’ll build it for you. If you otherwise have questions or want another tutorial? Please <a href="https://www.github.com/vsoch/lessons/issues">reach out</a>!</p>
<p><a name="myfootnote1">1</a>: Note that “singularity-jupyter” is mostly the same as “singularity-notebook” but it has a pre-built container on @vsoch scratch space, whereas singularity-notebook expects you to provide the path or unique resource identifier for the container.
<br />
<a name="myfootnote2">2</a>: A script called “singularity-notebook” might exist for the sherlock and farmshare clusters, and have a general version, and we would look under “sherlock” first given that it’s the resource you’ve defined in your <code class="language-plaintext highlighter-rouge">params.sh</code>, then look
up one level.</p>vsochLet’s set up our story. You are a graduate student, a postdoc, or staff in lab, and you want to use Sherlock, Farmshare, or one of the other Stanford clusters to run an interactive notebook. This might mean Jupyter notebook, or even a notebook with Julia, or R. This tutorial will help teach you how to do that.Using Pytorch and Singularity on Sherlock2018-08-05T00:00:00+00:002018-08-05T00:00:00+00:00https://vsoch.github.io/lessons/lessons/sherlock-pytorch<p>Here is a quick getting started for using pytorch on the Sherlock cluster! We have pre-built two containers, Docker containers, then we have pulled onto the cluster as Singularity containers that can help you out:</p>
<ul>
<li><a href="https://github.com/researchapps/sherlock/tree/master/pytorch-dev" target="_blank">README</a> with instructions for using one of several pytorch containers provided.</li>
<li>Singularity container with python 2.7, and pytorch [<a href="https://github.com/researchapps/sherlock/blob/master/pytorch-dev/Dockerfile.py2" target="_blank">Dockerfile</a>]</li>
<li>Singularity container with python 3.5, and pytorch [<a href="https://github.com/researchapps/sherlock/blob/master/pytorch-dev/Dockerfile" target="_blank">Dockerfile</a>]</li>
<li>Both containers on <a href="https://hub.docker.com/r/vanessa/pytorch-dev/" target="_blank">Docker Hub</a></li>
</ul>
<p>For tutorials using the containers in detail, see the first link with the README. In the following
example, we will show using the Python 2.7 container.</p>
<hr />
<h2 id="getting-started">Getting Started</h2>
<p>In the getting started snippet, we will show you how to grab an interactive gpu node using
<code class="language-plaintext highlighter-rouge">srun</code>, load the needed libraries and software, and then interact with torch (the module import name
for pytorch) to verify that we have gpu.</p>
<p><strong>Copy the container that you need from @vsoch shared folder</strong></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="c"># Pytorch with python 2.7 (this is used in tutorial below)</span>
<span class="nb">cp</span> /scratch/users/vsochat/share/pytorch-dev-py2.7.simg <span class="nv">$SCRATCH</span>
<span class="c"># Pytorch with python 3</span>
<span class="nb">cp</span> /scratch/users/vsochat/share/pytorch-dev.simg <span class="nv">$SCRATCH</span>
<span class="c"># Pytorch with python 3 (provided by pytorch/pytorch on Docker Hub)</span>
<span class="nb">cp</span> /scratch/users/vsochat/share/pytorch-0.4.1-cuda9-cudnn7-devel.simg <span class="nv">$SCRATCH</span>
</code></pre></div></div>
<p><strong>Grab an interactive node with gpu</strong></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>srun <span class="nt">-p</span> gpu <span class="nt">--gres</span><span class="o">=</span>gpu:1 <span class="nt">--pty</span> bash
</code></pre></div></div>
<p><strong>Load modules Singularity and cuda library</strong></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="nv">$ </span>module use system
<span class="nv">$ </span>module load singularity
<span class="nv">$ </span>module load cuda
</code></pre></div></div>
<p>Don’t forget to copy the container, if you didn’t above!</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="nb">cp</span> /scratch/users/vsochat/share/pytorch-dev-py2.7.simg <span class="nv">$SCRATCH</span>
<span class="nb">cd</span> <span class="nv">$SCRATCH</span>
</code></pre></div></div>
<p><strong>Shell into the container</strong></p>
<p>Now we are going to shell into the Singularity container! Note the <code class="language-plaintext highlighter-rouge">--nv</code> flag here,
it means “nvidia” and will expose the libraries on the host for the container to see.
You can use this flag with “exec” and “run” too!</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="nv">$ </span>singularity shell <span class="nt">--nv</span> pytorch-dev-py2.7.simg
Singularity: Invoking an interactive shell within container...
Singularity pytorch-dev-py2.7.simg:~> python
</code></pre></div></div>
<p>We just launched python! Let’s now import torch, and generate a variable called
“device” that will be “cuda:0” if we are using the gpu, or “cpu” if not.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="n">Python</span> <span class="mf">2.7</span><span class="p">.</span><span class="mi">14</span> <span class="o">|</span><span class="n">Intel</span> <span class="n">Corporation</span><span class="o">|</span> <span class="p">(</span><span class="n">default</span><span class="p">,</span> <span class="n">May</span> <span class="mi">4</span> <span class="mi">2018</span><span class="p">,</span> <span class="mi">04</span><span class="p">:</span><span class="mi">27</span><span class="p">:</span><span class="mi">35</span><span class="p">)</span>
<span class="p">[</span><span class="n">GCC</span> <span class="mf">4.8</span><span class="p">.</span><span class="mi">2</span> <span class="mi">20140120</span> <span class="p">(</span><span class="n">Red</span> <span class="n">Hat</span> <span class="mf">4.8</span><span class="p">.</span><span class="mi">2</span><span class="o">-</span><span class="mi">15</span><span class="p">)]</span> <span class="n">on</span> <span class="n">linux2</span>
<span class="n">Type</span> <span class="s">"help"</span><span class="p">,</span> <span class="s">"copyright"</span><span class="p">,</span> <span class="s">"credits"</span> <span class="ow">or</span> <span class="s">"license"</span> <span class="k">for</span> <span class="n">more</span> <span class="n">information</span><span class="p">.</span>
<span class="n">Intel</span><span class="p">(</span><span class="n">R</span><span class="p">)</span> <span class="n">Distribution</span> <span class="k">for</span> <span class="n">Python</span> <span class="ow">is</span> <span class="n">brought</span> <span class="n">to</span> <span class="n">you</span> <span class="n">by</span> <span class="n">Intel</span> <span class="n">Corporation</span><span class="p">.</span>
<span class="n">Please</span> <span class="n">check</span> <span class="n">out</span><span class="p">:</span> <span class="n">https</span><span class="p">:</span><span class="o">//</span><span class="n">software</span><span class="p">.</span><span class="n">intel</span><span class="p">.</span><span class="n">com</span><span class="o">/</span><span class="n">en</span><span class="o">-</span><span class="n">us</span><span class="o">/</span><span class="n">python</span><span class="o">-</span><span class="n">distribution</span>
<span class="o">>>></span> <span class="kn">import</span> <span class="nn">torch</span>
<span class="o">>>></span> <span class="kn">import</span> <span class="nn">torch.nn</span> <span class="k">as</span> <span class="n">nn</span>
<span class="o">>>></span> <span class="kn">from</span> <span class="nn">torch.autograd</span> <span class="kn">import</span> <span class="n">Variable</span>
<span class="o">>>></span> <span class="n">device</span> <span class="o">=</span> <span class="n">torch</span><span class="p">.</span><span class="n">device</span><span class="p">(</span><span class="s">"cuda:0"</span> <span class="k">if</span> <span class="n">torch</span><span class="p">.</span><span class="n">cuda</span><span class="p">.</span><span class="n">is_available</span><span class="p">()</span> <span class="k">else</span> <span class="s">"cpu"</span><span class="p">)</span>
<span class="o">>>></span> <span class="k">print</span><span class="p">(</span><span class="n">device</span><span class="p">)</span>
<span class="n">cuda</span><span class="p">:</span><span class="mi">0</span>
</code></pre></div></div>
<p>Tada! And from the above we see that we have successfully set up the container to work
with Python 2.7 and GPU. You can now read more about pytorch to get started with machine
learning.</p>
<p>Thanks to one of our awesome users in the <a href="https://cocolab.stanford.edu/" target="_blank">CoCoLab</a> for helping to develop this container and tutorial!</p>
<p>If you need a refresher with job submission, check out <a href="https://vsoch.github.io/lessons/sherlock-jobs/" target="_blank">our post on SLURM</a>.
Do you have questions or want to see another tutorial? Please <a href="https://www.github.com/vsoch/lessons/issues">reach out</a>!</p>vsochHere is a quick getting started for using pytorch on the Sherlock cluster! We have pre-built two containers, Docker containers, then we have pulled onto the cluster as Singularity containers that can help you out:Jupyter with Singularity on Sherlock2018-07-22T00:00:00+00:002018-07-22T00:00:00+00:00https://vsoch.github.io/lessons/lessons/sherlock-singularity<p>This is a followup to our original posts:</p>
<ul>
<li><a href="https://vsoch.github.io/lessons/sherlock-jupyter/">sherlock jupyter</a></li>
<li><a href="https://vsoch.github.io/lessons/jupyter-tensorflow/">sherlock tensorflow</a></li>
</ul>
<p>that described how to get access to a jupyter notebook on Sherlock, with or without
GPU and tensorflow, with port forwarding!</p>
<p>Today we will extend the example to show how to run jupyter via a Singularity container.
We are going to be using a container produced via <a href="https://www.github.com/jupyter/repo2docker">repo2docker</a>
that was pulled to Sherlock from <a href="https://hub.docker.com/r/vanessa/repo2docker/">Docker Hub</a> and
can be used as a general jupyter base, or in a “read only” mode to see the notebook provided by
@vsoch, the author. Why is this cool? It means that you can use the repo2docker tool
to create your own custom container notebooks to share with others. Let’s get started!</p>
<h2 id="what-are-we-doing-today">What are we doing today?</h2>
<p>Today we are going to do the following:</p>
<ul>
<li>configure the <a href="https://github.com/vsoch/forward">forward</a> tool</li>
<li>run a password protected jupyter notebook via a Singularity container on a cluster node</li>
<li>access the notebook in a browsr on your local machine</li>
</ul>
<p>For more background, see the <a href="https://vsoch.github.io/lessons/sherlock-jupyter/">original post</a>.</p>
<h2 id="step-1-download-the-repository">Step 1: Download the Repository</h2>
<p>First, clone the repository, and remember the location. You will generate a parameters flie
<code class="language-plaintext highlighter-rouge">params.sh</code> that you can go back to and use again.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git clone https://github.com/vsoch/forward
<span class="nb">cd </span>forward
</code></pre></div></div>
<p>As a reminder from the previous posts, here are the files that we have to work with.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>├── end.sh <span class="o">(</span><span class="nt">--</span> end a session
├── resume.sh <span class="o">(</span><span class="nt">--</span> resume a started session
├── sbatches <span class="o">(</span><span class="nt">--</span> batch scripts you can run to forward a port to!
├── singularity.sbatch
...
├── jupyter.sbatch
└── tensorboard.sbatch
├── setup.sh <span class="o">(</span><span class="nt">--</span> run once to <span class="nb">set </span>up the tool
└── start.sh <span class="o">(</span><span class="nt">--</span> how you start a connection
</code></pre></div></div>
<h2 id="step-2-generate-your-parameters">Step 2: Generate your Parameters</h2>
<p>You will need to generate a file of parameters to source before using the tool. Importantly, if you
want to use tensorflow with a gpu, you should select the partition to be “gpu” when it asks you.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>bash setup.sh
Sherlock username <span class="o">></span> tacocat
Next, pick a port to use. If someone <span class="k">else </span>is port forwarding using that
port already, this script will not work. If you pick a random number <span class="k">in </span>the
range 49152-65335, you should be good.
Port to use <span class="o">></span> 56143
Next, pick the sherlock partition on which you will be running your
notebooks. If your PI has purchased dedicated hardware on sherlock, you can use
that partition. Otherwise, leave blank to use the default partition <span class="o">(</span>normal<span class="o">)</span><span class="nb">.</span>
Sherlock partition <span class="o">(</span>default: normal<span class="o">)</span> <span class="o">></span> gpu
Next, pick the path to the browser you wish to use. Will default to Safari.
Browser to use <span class="o">(</span>default: /Applications/Safari.app/<span class="o">)</span> <span class="o">></span> /usr/bin/firefox
</code></pre></div></div>
<p>Notice that I’ve changed the default browser (Safari on a Mac) to firefox on Ubuntu.
Also note that I entered “gpu” for my partition. The resulting file is a simple text file:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">cat </span>params.sh
<span class="nv">USERNAME</span><span class="o">=</span><span class="s2">"tacocat"</span>
<span class="nv">PORT</span><span class="o">=</span><span class="s2">"56143"</span>
<span class="nv">PARTITION</span><span class="o">=</span><span class="s2">"gpu"</span>
<span class="nv">BROWSER</span><span class="o">=</span><span class="s2">"/usr/bin/firefox"</span>
<span class="nv">MEM</span><span class="o">=</span><span class="s2">"20G"</span>
<span class="nv">TIME</span><span class="o">=</span><span class="s2">"8:00:00"</span>
</code></pre></div></div>
<h2 id="step-3-ssh-credentials">Step 3: SSH Credentials</h2>
<p>Follow the <a href="https://www.sherlock.stanford.edu/docs/getting-started/connecting/" target="_blank">instructions here</a> to set up your Sherlock credentials. You
can see what you have by looking at your <code class="language-plaintext highlighter-rouge">~/.ssh/config</code>:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cat</span> ~/.ssh/config
</code></pre></div></div>
<p>Don’t have the file? You can use a helper script in the repository to generate it.
There is a helper script for Sherlock:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">ls </span>hosts/
sherlock_ssh.sh
</code></pre></div></div>
<p>and when you run it, the configuration is printed to the screen:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>bash hosts/sherlock_ssh.sh
Sherlock username <span class="o">></span> tacocat
Host sherlock
User tacocat
Hostname sh-ln03.stanford.edu
GSSAPIDelegateCredentials <span class="nb">yes
</span>GSSAPIAuthentication <span class="nb">yes
</span>ControlMaster auto
ControlPersist <span class="nb">yes
</span>ControlPath ~/.ssh/%l%r@%h:%p
</code></pre></div></div>
<p>or just do the entire thing programatically:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bash hosts/sherlock_ssh.sh <span class="o">>></span> ~/.ssh/config
</code></pre></div></div>
<h2 id="step-4-singularity-usage">Step 4. Singularity Usage</h2>
<p>Did you notice we skipped the step to set up jupyter passwords on Sherlock? We don’t
need to worry about software dependencies because we are using containers. Thank goodness!
This makes the entire workflow less error prone - the dependencies for jupyter are
packaged in the container! For the example container I’m showing you, the image is provided on Sherlock</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bash start.sh <software> <path>
</code></pre></div></div>
<p>This notebook will serve a basic container, rooted in your directory of choice!</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>bash start.sh singularity-jupyter /path/to/dir
</code></pre></div></div>
<p>The default will point to your scratch:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>bash start.sh singularity-jupyter
<span class="c"># equivalent to</span>
<span class="nv">$ </span>bash start.sh singularity-jupyter /scratch/users/<username>
</code></pre></div></div>
<p>The above command will submit the job, forward the port, and show you the log that has your
token password to enter into the url. It’s a token generated
on the fly. If it doesn’t, you can find it (or some other bug that was not revealed!)
in the error or output files:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">SHERLOCK_HOME</span><span class="o">=</span>/home/users/vsochat
<span class="nv">$ </span>ssh sherlock <span class="nb">cat</span> <span class="k">${</span><span class="nv">SHERLOCK_HOME</span><span class="k">}</span>/forward-util/singularity-jupyter.err
<span class="nv">$ </span>ssh sherlock <span class="nb">cat</span> <span class="k">${</span><span class="nv">SHERLOCK_HOME</span><span class="k">}</span>/forward-util/singularity-jupyter.out
</code></pre></div></div>
<p>When you are ready to be done, or you’ve left and want to come back later:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Resume a session</span>
bash resume.sh <name>
bash resume.sh singularity-jupyter
<span class="c"># End a session</span>
bash end.sh <name>
bash end.sh singularity-jupyter
</code></pre></div></div>
<p>If you want custom software in your container, give <a href="https://repo2docker.readthedocs.io">repo2docker</a>
a try! For this tutorial, we used the <a href="https://github.com/binder-examples/continuous-build/">continuous builder</a> template to have the image deployed automatically to Docker Hub.</p>
<h2 id="step-6-notebook">Step 6: Notebook</h2>
<p>What highly important python modules did I provide for you in this example? Open up
a Python 3 new Notebook, and try this out:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">pokemon.skills</span> <span class="kn">import</span> <span class="n">get_avatar</span>
<span class="n">avatar</span> <span class="o">=</span> <span class="n">get_avatar</span><span class="p">(</span><span class="s">'vanessa'</span><span class="p">)</span>
</code></pre></div></div>
<p><img src="/lessons/assets/img/tutorials/jupyter-pokemon.png" alt="/lessons/assets/img/tutorials/jupyter-pokemon.png" /></p>
<p>Check out the <a href="https://www.github.com/vsoch/pokemon-ascii">pokemon module</a> for other fun things you can do :)</p>
<h2 id="i-want-to-run-my-own-container">I want to run my own container!</h2>
<p>This is really easy for you to do! You can use the general Singularity container script,
provided as <code class="language-plaintext highlighter-rouge">singularity.sbatch</code> in the forward repository to get started. The basic idea is that you would want to:</p>
<ol>
<li>Create a Github repository with a <code class="language-plaintext highlighter-rouge">requirements.txt</code> file of things to install</li>
<li>Optionally add jupyter notebooks</li>
<li>Push the image to Docker Hub</li>
</ol>
<p>And then either pull it to Sherlock with Singularity, or provide the docker uri to the container (and it will be pulled on the fly). The first is generallly recommended. If you need help, or hey, if you
have a request for a container, please reach out to <a href="https://www.twitter.com/vsoch">@vsoch</a>. I will
be happy to build and provide a container for you from my shared folder!</p>
<h2 id="debugging">Debugging</h2>
<p>Here are some common pitfalls that I hit during development. If you still need help, please
reach out to our group!</p>
<p><strong>There is a missing module!</strong></p>
<p>This actually happened for me - I got an error message that there wasn’t a module called google protobuf.
This is a bug with the installation of tensorflow on sherlock, because it should be loaded if it’s required.
No fear! You can fix it yourself, and even from your terminal. First, do this:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">os</span>
<span class="n">os</span><span class="p">.</span><span class="n">system</span><span class="p">(</span><span class="s">"pip install --user <module>"</span><span class="p">)</span>
</code></pre></div></div>
<p>And then restart the kernal from the interface to have changes take effect (you will
get some glorious error messages if you don’t).</p>
<p><strong>The terminal is hanging!</strong></p>
<p>To make this automated, we issue commands from the <code class="language-plaintext highlighter-rouge">start.sh</code> script to capture output
from sherlock, using <code class="language-plaintext highlighter-rouge">ssh sherlock</code> to send the command. If it’s the case that your login
session has expired (or you got the wrong password), then you might have a password
prompt (that you can’t see) that looks like the terminal is hanging. If this seems to be
the case, try opening a new terminal window, and authenticating with sherlock again (<code class="language-plaintext highlighter-rouge">ssh sherlock pwd</code> should trigger the login authentication flow.)</p>
<p><strong>The job is never allocated</strong></p>
<p>This usually means there is an error on job submission, and you can look at the error<br />
and output files to see them. For example, let’s say that I want to do this for this run:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ssh sherlock <span class="nb">cat</span> /home/users/vsochat/forward-util/singularity-jupyter.err
ssh sherlock <span class="nb">cat</span> /home/users/vsochat/forward-util/singularity-jupyter.out
</code></pre></div></div>
<p><strong>Failed to setup local forwarding</strong></p>
<p>If you have a hanging process (if you killed a session and now can’t recreate it) you might get an error
message about not being able to set up the port forwarding! The easiest thing to do here is to use
end.sh and start.sh again.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bash end.sh singularity-jupyter
</code></pre></div></div>
<p><strong>There is some other problem, arggg!</strong></p>
<p>Well what are you waiting for? Ask for help by <a href="https://github.com/vsoch/forward" target="_blank">opening an issue</a>! or submitting a pull request.</p>
<p>Do you have questions or want to see another tutorial? Please <a href="https://www.github.com/vsoch/lessons/issues">reach out</a>!</p>vsochThis is a followup to our original posts:Jupyter Notebooks on Sherlock2018-07-09T00:00:00+00:002018-07-09T00:00:00+00:00https://vsoch.github.io/lessons/lessons/sherlock-jupyter<p>Today we are going to walk through using a <a href="https://github.com/vsoch/forward">tool</a> provided by one of our users to set up port forwarding on Sherlock. What does this mean? It means you…</p>
<ul>
<li>will run a password protected jupyter notebook on a cluster node</li>
<li>can access the notebook in a browsr on your local machine</li>
</ul>
<p>It also means that <em>any tool that is served on a port</em> that you can run on an interactive node via an sbatch job could
be accessed on your computer. Very cool!</p>
<h2 id="why-should-i-use-this">Why should I use this?</h2>
<p>Likely some of you are doing this on your own with basic bash commands, and this repository nicely packages those commands.</p>
<h3 id="collaborative-scripts">Collaborative Scripts</h3>
<p>The repository has a folder of general sbatch scripts that expose ports that can be forwarded. If you add a new script that you find useful and want to share with others, we encourage you to <a href="https://github.com/vsoch/forward" target="_blank">tell us about it</a>! If it’s not perfect or finished, don’t be afraid to share! We will provide any additional help needed to finish up your script.</p>
<h3 id="community-support">Community Support</h3>
<p>This tool is brought to you by one of our community <a href="https://github.com/raphtown" target="_blank">talented users</a> that reached out to the user list. It’s an amazing example of the goodness that can come out of sharing your code because others can use it too.</p>
<h2 id="1-on-your-computer">1. On Your Computer</h2>
<p>The repository contains a set of scripts for setting up automatic port forwarding on sherlock with jupyter notebooks.
There are a set of commands you will need to run just once to configure the tool, and then general “start”, “end” and “resume” operations for interacting with notebook jobs. The repository README has instructions for setup and usage, and we will also walk through them here.</p>
<h3 id="step-1-download-the-repository">Step 1: Download the Repository</h3>
<p>Make sure you put the folder somewhere meaningful. A subfolder of <code class="language-plaintext highlighter-rouge">$HOME</code> is a suggestion. This will be your working location for future use of the tool, as it holds scripts and a parameter file, <code class="language-plaintext highlighter-rouge">params.sh</code></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git clone https://github.com/vsoch/forward
<span class="nb">cd </span>forward
</code></pre></div></div>
<p>Let’s take a look at what we have here!</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>├── end.sh <span class="o">(</span><span class="nt">--</span> end a session
├── resume.sh <span class="o">(</span><span class="nt">--</span> resume a started session
├── sbatches <span class="o">(</span><span class="nt">--</span> batch scripts you can run to forward a port to!
├── jupyter.sbatch
└── tensorboard.sbatch
├── setup.sh <span class="o">(</span><span class="nt">--</span> run once to <span class="nb">set </span>up the tool
└── start.sh <span class="o">(</span><span class="nt">--</span> how you start a connection
</code></pre></div></div>
<h3 id="step-2-generate-your-parameters">Step 2: Generate your Parameters</h3>
<p>This is a one time generation script that will create a parameter text file with a port, username, and cluster resource. You can generate it by running the script <a href="https://github.com/vsoch/forward/blob/master/setup.sh">setup.sh</a>:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>bash setup.sh
Sherlock username <span class="o">></span> tacocat
Next, pick a port to use. If someone <span class="k">else </span>is port forwarding using that
port already, this script will not work. If you pick a random number <span class="k">in </span>the
range 49152-65335, you should be good.
Port to use <span class="o">></span> 56143
Next, pick the sherlock partition on which you will be running your
notebooks. If your PI has purchased dedicated hardware on sherlock, you can use
that partition. Otherwise, leave blank to use the default partition <span class="o">(</span>normal<span class="o">)</span><span class="nb">.</span>
Sherlock partition <span class="o">(</span>default: normal<span class="o">)</span> <span class="o">></span>
Next, pick the path to the browser you wish to use. Will default to Safari.
Browser to use <span class="o">(</span>default: /Applications/Safari.app/<span class="o">)</span> <span class="o">></span> /usr/bin/firefox
</code></pre></div></div>
<p>Notice that I’ve changed the default browser (Safari on a Mac) to firefox on Ubuntu. Also note that I pressed enter
to use the default queue. The resulting file is a simple text file:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">cat </span>params.sh
<span class="nv">USERNAME</span><span class="o">=</span><span class="s2">"tacocat"</span>
<span class="nv">PORT</span><span class="o">=</span><span class="s2">"56143"</span>
<span class="nv">PARTITION</span><span class="o">=</span><span class="s2">"normal"</span>
<span class="nv">BROWSER</span><span class="o">=</span><span class="s2">"/usr/bin/firefox"</span>
<span class="nv">MEM</span><span class="o">=</span><span class="s2">"20G"</span>
<span class="nv">TIME</span><span class="o">=</span><span class="s2">"8:00:00"</span>
</code></pre></div></div>
<h3 id="step-3-ssh-credentials">Step 3: SSH Credentials</h3>
<p>It wouldn’t be so great if anyone with your username and a port could access your notebook! For this reason we need to create ssh credentials. You might have this already set up if you’ve followed the <a href="https://www.sherlock.stanford.edu/docs/getting-started/connecting/" target="_blank">instructions here</a> for Sherlock. You
can see what you have by looking at your <code class="language-plaintext highlighter-rouge">~/.ssh/config</code>:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cat</span> ~/.ssh/config
</code></pre></div></div>
<p>What we want to do is have this configuration. Add this to your <code class="language-plaintext highlighter-rouge">~/.ssh/config</code>. The repository also has a quick way to generate your ssh config, and print it to the screen. See the hosts folder? In there are small helper scripts to print the configurations to the screen! We have one for Sherlock:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">ls </span>hosts/
sherlock_ssh.sh
</code></pre></div></div>
<p>Running the script will print the configuration to the screen:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>bash hosts/sherlock_ssh.sh
Sherlock username <span class="o">></span> tacocat
Host sherlock
User tacocat
Hostname login.sherlock.stanford.edu
GSSAPIDelegateCredentials <span class="nb">yes
</span>GSSAPIAuthentication <span class="nb">yes
</span>ControlMaster auto
ControlPersist <span class="nb">yes
</span>ControlPath ~/.ssh/%l%r@%h:%p
</code></pre></div></div>
<p>If you don’t have a file existing at <code class="language-plaintext highlighter-rouge">~/.ssh/config</code> then you will need to create it, and you
can do the entire step above programatically:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bash hosts/sherlock_ssh.sh <span class="o">>></span> ~/.ssh/config
</code></pre></div></div>
<p>Your computer is ready to go! Let’s move onto Sherlock. We will come back here when it’s time to start a notebook.</p>
<h2 id="2-on-sherlock">2. On Sherlock</h2>
<p><br /></p>
<h3 id="21-jupyter-notebook">2.1. Jupyter notebook</h3>
<p>We will want to set up a password for our Jupyter notebook, and we can do this programatically
before starting it. First, let’s load the module for Python, and install jupyter notebook! The version of
Python that you choose is up to you.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">-ln07</span> login! ~/]<span class="nv">$ </span>module spider jupyter
<span class="nt">----------------------------------------------------------------------------</span>
py-jupyter:
<span class="nt">----------------------------------------------------------------------------</span>
Description:
Jupyter is a browser-based interactive notebook <span class="k">for </span>programming,
mathematics, and data science. It supports a number of languages via
plugins.
Versions:
py-jupyter/1.0.0_py27
py-jupyter/1.0.0_py36
<span class="nt">----------------------------------------------------------------------------</span>
For detailed information about a specific <span class="s2">"py-jupyter"</span> module <span class="o">(</span>including how to load the modules<span class="o">)</span> use the module<span class="s1">'s full name.
For example:
$ module spider py-jupyter/1.0.0_py36
----------------------------------------------------------------------------
</span></code></pre></div></div>
<p>Let’s load the Python 3.6 jupyter.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>module load py-jupyter/1.0.0_py36
<span class="nv">$ </span>which jupyter
/share/software/user/open/py-jupyter/1.0.0_py36/bin/jupyter
</code></pre></div></div>
<p>We will need to secure our notebooks with a password. Pick a strong one!</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>jupyter notebook password
</code></pre></div></div>
<p>It will be associated with your username, saved to a local file:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Enter password:
Verify password:
<span class="o">[</span>NotebookPasswordApp] Wrote hashed password to /home/users/vsochat/.jupyter/jupyter_notebook_config.json
</code></pre></div></div>
<p>The above might pause or hang a little bit, at least it did when I did (and I suspect someone
was running a resource intensive process on the node…)</p>
<h2 id="3-usage">3. Usage</h2>
<p>We have just set up a password on Sherlock, and are now back on our <em>local machine</em>. Here are the general commands to start and stop sessions. In the tutorial below, we will walk through using Jupyter notebook.</p>
<h3 id="31-start-a-session">3.1. Start a Session</h3>
<p>From the directory where we cloned, we can start a session using the <a href="https://github.com/vsoch/forward/blob/master/start.sh">start.sh</a> script. This is a general script to start any kind of session, and here we will show how to start a jupyter notebook in a specific directory:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bash start.sh <software> <path>
bash start.sh jupyter /path/to/dir
</code></pre></div></div>
<p>What’s going on? It will look in the folder of <a href="https://github.com/vsoch/forward/blob/master/sbatches">sbatch scripts</a> and find one named correpondingly to the command we issued. This means that there is a file called <a href="https://github.com/vsoch/forward/blob/master/sbatches/jupyter.sbatch">jupyter.sbatch</a> in that folder. Note that, to
make this simple, we also have sbatch scripts for using jupyter with different Python kernels:</p>
<p>Let’s fire up a notebook with Python 3!</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>bash start.sh py3-jupyter /scratch/users/vsochat
<span class="o">==</span> Checking <span class="k">for </span>previous notebook <span class="o">==</span>
No existing py3-jupyter <span class="nb">jobs </span>found, continuing...
<span class="o">==</span> Getting destination directory <span class="o">==</span>
<span class="o">==</span> Uploading sbatch script <span class="o">==</span>
py3-jupyter.sbatch 100% 132 0.1KB/s 00:00
<span class="o">==</span> Submitting sbatch <span class="o">==</span>
Submitted batch job 21659278
<span class="o">==</span> Waiting <span class="k">for </span>job to start, using exponential backoff <span class="o">==</span>
Attempt 0: not ready yet... retrying <span class="k">in </span>1..
Attempt 1: not ready yet... retrying <span class="k">in </span>2..
Attempt 2: not ready yet... retrying <span class="k">in </span>4..
Attempt 3: not ready yet... retrying <span class="k">in </span>8..
Attempt 4: not ready yet... retrying <span class="k">in </span>16..
Attempt 5: not ready yet... retrying <span class="k">in </span>32..
Attempt 6: not ready yet... retrying <span class="k">in </span>64..
...
Attempt 6: resources allocated to sh-101-21!..
sh-101-21
notebook running on sh-101-21
<span class="o">==</span> Setting up port forwarding <span class="o">==</span>
ssh <span class="nt">-L</span> 56143:localhost:56143 sherlock ssh <span class="nt">-L</span> 56143:localhost:56143 <span class="nt">-N</span> sh-101-21 &
<span class="o">==</span> Connecting to notebook <span class="o">==</span>
Open your browser to http://localhost:56143
</code></pre></div></div>
<p>When you open your browser to the address, you will see a prompt for the password that you
created previously:</p>
<p><img src="/lessons/assets/img/tutorials/jupyter-password.png" alt="/lessons/assets/img/tutorials/jupyter-password.png" /></p>
<p>If you have an already running session, you will see this message:</p>
<p>and then log in to see your scratch!</p>
<p><img src="/lessons/assets/img/tutorials/jupyter-scratch.png" alt="/lessons/assets/img/tutorials/jupyter-scratch.png" /></p>
<h3 id="32-resume-a-session">3.2. Resume a Session</h3>
<p>Sometimes the job can still be running, but the port forwarding has stopped (has your computer
ever gone to sleep?) In this case, you can resume the session using the equivalent command, but
use resume.sh:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bash resume.sh py3-jupyter<span class="sb">`</span>
</code></pre></div></div>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>bash start.sh py3-jupyter /scratch/users/vsochat
<span class="o">==</span> Checking <span class="k">for </span>previous notebook <span class="o">==</span>
Found existing job <span class="k">for </span>py3-jupyter, sh-101-04 end.sh or resume.sh
</code></pre></div></div>
<h3 id="33-stop-a-session">3.3. Stop a Session</h3>
<p>When you want to stop the session (killing the job) run the equivalent command, but use the end.sh script.
The command is based on the name of the job (the sbatch script name) so to kill the previous job we created,
we would do:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>bash end.sh py3-jupyter /scratch/users/vsochat
Killing py3-jupyter slurm job on sherlock
Killing listeners on sherlock
</code></pre></div></div>
<p>There are additional details and debugging tips in the <a href="https://github.com/vsoch/forward" target="_blank">main repository</a>! Happy Jupyter-ing!</p>
<h2 id="contribute">Contribute!</h2>
<p>The cool thing about small efforts like these is that they provide very useful tools for you
and the larger Stanford community. We can continue to work on them together to make them even better,
and add support for more functions and kinds of sessions. This is the awesome thing about open source!
Here are some suggestions for how you can contribute:</p>
<h3 id="adding-new-sbatch-scripts">Adding new sbatch scripts</h3>
<p>You can add more sbatch scripts by putting them in the sbatches directory. You should assume that an SBATCH script
will be run on a job node, and should take a port as the first argument, and directory to work from as the second. Other additional arguments are up to you! For example, here is what the jupyter batch job looks like:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/bin/bash</span>
<span class="nv">PORT</span><span class="o">=</span><span class="nv">$1</span>
<span class="nv">NOTEBOOK_DIR</span><span class="o">=</span><span class="nv">$2</span>
<span class="nb">cd</span> <span class="nv">$NOTEBOOK_DIR</span>
module load py-jupyter/1.0.0_py27
jupyter notebook <span class="nt">--no-browser</span> <span class="nt">--port</span><span class="o">=</span><span class="nv">$PORT</span>
</code></pre></div></div>
<p>Contributing can also be as simple as suggesting a feature or reporting a bug on the <a href="https://github.com/vsoch/forward/issues" target="_blank">issues board</a>.
See the <a href="https://github.com/vsoch/forward" target="_blank">main repository</a> for how to contribute.</p>
<h3 id="add-functionality">Add Functionality</h3>
<p>Right now we can start, stop (end), and resume. Wouldn’t it be cool to also have a script to print a status?</p>
<p>Do you have questions or want to see another tutorial? Please <a href="https://www.github.com/vsoch/lessons/issues">reach out</a>!</p>vsochToday we are going to walk through using a tool provided by one of our users to set up port forwarding on Sherlock. What does this mean? It means you…Jupyter with Tensorflow (GPU) on Sherlock2018-07-09T00:00:00+00:002018-07-09T00:00:00+00:00https://vsoch.github.io/lessons/lessons/jupyter-tensorflow<p>This is a followup to our <a href="https://vsoch.github.io/lessons/sherlock-jupyter/">original post</a>
that described how to get access to a jupyter notebook on Sherlock with port forwarding!
Today we will extend the example to a new set of
<a href="https://github.com/vsoch/forward/tree/master/sbatches">sbatch scripts</a>
that will start up a jupyter notebook with tensorflow. Want a GPU? We can do that too.
If you want to container-based verison of this tutorial (yes, you can deploy jupyter
from a container too!) then see <a href="https://vsoch.github.io/lessons/sherlock-singularity">this post</a>. Let’s get started!</p>
<h2 id="what-are-we-doing-today">What are we doing today?</h2>
<p>Today we are going to do the following:</p>
<ul>
<li>configure the <a href="https://github.com/vsoch/forward">forward</a> tool to use the GPU partition</li>
<li>run a password protected jupyter notebook with tensorflow on a cluster node</li>
<li>access the notebook in a browsr on your local machine</li>
</ul>
<p>For more background, see the <a href="https://vsoch.github.io/lessons/sherlock-jupyter/">original post</a>.
For these instructions, you should be careful about opening and closing your terminal sessions.
If you find a command hanging or failing, you might need to re-authenticate the session with your
password, or you might be entering the wrong password.</p>
<h2 id="step-1-download-the-repository">Step 1: Download the Repository</h2>
<p>First, clone the repository, and remember the location. You will generate a parameters flie
<code class="language-plaintext highlighter-rouge">params.sh</code> that you can go back to and use again.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git clone https://github.com/vsoch/forward
<span class="nb">cd </span>forward
</code></pre></div></div>
<p>As a reminder from the previous post, here are the files that we have to work with.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>├── end.sh <span class="o">(</span><span class="nt">--</span> end a session
├── resume.sh <span class="o">(</span><span class="nt">--</span> resume a started session
├── sbatches <span class="o">(</span><span class="nt">--</span> batch scripts you can run to forward a port to!
├── jupyter.sbatch
└── tensorboard.sbatch
├── setup.sh <span class="o">(</span><span class="nt">--</span> run once to <span class="nb">set </span>up the tool
└── start.sh <span class="o">(</span><span class="nt">--</span> how you start a connection
</code></pre></div></div>
<h2 id="step-2-generate-your-parameters">Step 2: Generate your Parameters</h2>
<p>You will need to generate a file of parameters to source before using the tool. Importantly, if you
want to use tensorflow with a gpu, you should select the partition to be “gpu” when it asks you.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>bash setup.sh
Sherlock username <span class="o">></span> tacocat
Next, pick a port to use. If someone <span class="k">else </span>is port forwarding using that
port already, this script will not work. If you pick a random number <span class="k">in </span>the
range 49152-65335, you should be good.
Port to use <span class="o">></span> 56143
Next, pick the sherlock partition on which you will be running your
notebooks. If your PI has purchased dedicated hardware on sherlock, you can use
that partition. Otherwise, leave blank to use the default partition <span class="o">(</span>normal<span class="o">)</span><span class="nb">.</span>
Sherlock partition <span class="o">(</span>default: normal<span class="o">)</span> <span class="o">></span> gpu
Next, pick the path to the browser you wish to use. Will default to Safari.
Browser to use <span class="o">(</span>default: /Applications/Safari.app/<span class="o">)</span> <span class="o">></span> /usr/bin/firefox
</code></pre></div></div>
<p>Notice that I’ve changed the default browser (Safari on a Mac) to firefox on Ubuntu.
Also note that I entered “gpu” for my partition. The resulting file is a simple text file:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">cat </span>params.sh
<span class="nv">USERNAME</span><span class="o">=</span><span class="s2">"tacocat"</span>
<span class="nv">PORT</span><span class="o">=</span><span class="s2">"56143"</span>
<span class="nv">PARTITION</span><span class="o">=</span><span class="s2">"gpu"</span>
<span class="nv">BROWSER</span><span class="o">=</span><span class="s2">"/usr/bin/firefox"</span>
<span class="nv">MEM</span><span class="o">=</span><span class="s2">"20G"</span>
<span class="nv">TIME</span><span class="o">=</span><span class="s2">"8:00:00"</span>
</code></pre></div></div>
<h2 id="step-3-ssh-credentials">Step 3: SSH Credentials</h2>
<p>Follow the <a href="https://www.sherlock.stanford.edu/docs/getting-started/connecting/" target="_blank">instructions here</a> to set up your Sherlock credentials. You
can see what you have by looking at your <code class="language-plaintext highlighter-rouge">~/.ssh/config</code>:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cat</span> ~/.ssh/config
</code></pre></div></div>
<p>Don’t have the file? You can use a helper script in the repository to generate it.
There is a helper script for Sherlock:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">ls </span>hosts/
sherlock_ssh.sh
</code></pre></div></div>
<p>and when you run it, the configuration is printed to the screen:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>bash hosts/sherlock_ssh.sh
Sherlock username <span class="o">></span> tacocat
Host sherlock
User tacocat
Hostname sh-ln03.stanford.edu
GSSAPIDelegateCredentials <span class="nb">yes
</span>GSSAPIAuthentication <span class="nb">yes
</span>ControlMaster auto
ControlPersist <span class="nb">yes
</span>ControlPath ~/.ssh/%l%r@%h:%p
</code></pre></div></div>
<p>or just do the entire thing programatically:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bash hosts/sherlock_ssh.sh <span class="o">>></span> ~/.ssh/config
</code></pre></div></div>
<h2 id="step-4-on-sherlock">Step 4. On Sherlock</h2>
<p><br /></p>
<p>For local jupyter usage, set up your jupyter notebook password, for either version for Python 2 and/or 3.
I wound up loading both modules and setting the same password, because I knew I’d forget.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>module load py-jupyter/1.0.0_py36
<span class="c"># and do the same for py-jupyter/1.0.0_py27</span>
<span class="nv">$ </span>which jupyter
/share/software/user/open/py-jupyter/1.0.0_py36/bin/jupyter
</code></pre></div></div>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Set the password</span>
jupyter notebook password
</code></pre></div></div>
<p>Also make sure you have google and protobuf installed. The Python2 scripts loads
protobuf as a module, but I did this anyway just in case.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>pip <span class="nb">install </span>protobuf <span class="nt">--user</span>
<span class="nv">$ </span>pip3 <span class="nb">install </span>protobuf <span class="nt">--user</span>
<span class="nv">$ </span>pip <span class="nb">install </span>google <span class="nt">--user</span>
<span class="nv">$ </span>pip3 <span class="nb">install </span>google <span class="nt">--user</span>
</code></pre></div></div>
<h2 id="step-5-usage">Step 5. Usage</h2>
<p>Now we are back on our <em>local machine</em>. Here are the general commands to start and stop sessions. In the tutorial below, we will walk through using Jupyter notebook.</p>
<p>bash start.sh singularity-jupyte</p>
<p>We’ve already reviewed how to start a session in the previous post, now we will just go over how to start
the tensorflow jupyter notebook, using the password from above. If you want more verbosity, see the previous post. The general command for start.sh looks like this:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bash start.sh <software> <path>
</code></pre></div></div>
<p>The above command will submit the job, forward the port, and show you the log that has your
token password to enter into the url. The command to run jupyter notebook with tensorflow
(using the modules provided on Sherlock) looks like this:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>bash start.sh py2-tensorflow /path/to/dir
<span class="o">==</span> Checking <span class="k">for </span>previous notebook <span class="o">==</span>
No existing py2-tensorflow <span class="nb">jobs </span>found, continuing...
<span class="o">==</span> Getting destination directory <span class="o">==</span>
<span class="o">==</span> Uploading sbatch script <span class="o">==</span>
py2-tensorflow.sbatch 100% 169 0.2KB/s 00:00
<span class="o">==</span> Requesting GPU <span class="o">==</span>
<span class="o">==</span> Submitting sbatch <span class="o">==</span>
sherlock sbatch <span class="nt">--job-name</span><span class="o">=</span>py2-tensorflow <span class="nt">--partition</span><span class="o">=</span>gpu <span class="nt">--gres</span> gpu:1 <span class="nt">--output</span><span class="o">=</span>/home/users/vsochat/forward-util/py2-tensorflow.out <span class="nt">--error</span><span class="o">=</span>/home/users/vsochat/forward-util/py2-tensorflow.err <span class="nt">--mem</span><span class="o">=</span>20G <span class="nt">--time</span><span class="o">=</span>8:00:00 /home/users/vsochat/forward-util/py2-tensorflow.sbatch 56143 <span class="s2">"/scratch/users/vsochat"</span>
Submitted batch job 22351236
<span class="o">==</span> Waiting <span class="k">for </span>job to start, using exponential backoff <span class="o">==</span>
Attempt 0: not ready yet... retrying <span class="k">in </span>1..
Attempt 1: not ready yet... retrying <span class="k">in </span>2..
Attempt 2: not ready yet... retrying <span class="k">in </span>4..
Attempt 3: not ready yet... retrying <span class="k">in </span>8..
Attempt 4: not ready yet... retrying <span class="k">in </span>16..
Attempt 5: not ready yet... retrying <span class="k">in </span>32..
Attempt 6: resources allocated to sh-17-30!..
sh-17-30
notebook running on sh-17-30
<span class="o">==</span> Setting up port forwarding <span class="o">==</span>
ssh <span class="nt">-L</span> 56143:localhost:56143 sherlock ssh <span class="nt">-L</span> 56143:localhost:56143 <span class="nt">-N</span> sh-17-30 &
<span class="o">==</span> Connecting to notebook <span class="o">==</span>
Open your browser to http://localhost:56143
</code></pre></div></div>
<p>When you open your browser to the address, you will see a prompt for the password that you
created previously. And when you are ready to be done, or you’ve left and want to come
back later:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Resume a session</span>
bash resume.sh <name>
bash resume.sh py2-tensorflow<span class="sb">`</span>
<span class="c"># End a session</span>
bash end.sh <name>
bash end.sh py2-tensorflow<span class="sb">`</span>
</code></pre></div></div>
<h2 id="step-6-notebook">Step 6. Notebook</h2>
<p>Once you have jupyter running, you want to <strong>make sure you select the python kernel corresponding to
the one for your job!</strong> For our tutorial, we loaded python 2, so we would want to create a Python 2 notebook.
Once in the notebook, here is a command that will let you check the devices available:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">tensorflow</span>
<span class="kn">from</span> <span class="nn">tensorflow.python.client</span> <span class="kn">import</span> <span class="n">device_lib</span>
<span class="k">print</span><span class="p">(</span><span class="n">device_lib</span><span class="p">.</span><span class="n">list_local_devices</span><span class="p">())</span>
</code></pre></div></div>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">[</span><span class="n">name</span><span class="p">:</span> <span class="s">"/device:CPU:0"</span>
<span class="n">device_type</span><span class="p">:</span> <span class="s">"CPU"</span>
<span class="n">memory_limit</span><span class="p">:</span> <span class="mi">268435456</span>
<span class="n">locality</span> <span class="p">{</span>
<span class="p">}</span>
<span class="n">incarnation</span><span class="p">:</span> <span class="mi">7029902632617210308</span>
<span class="p">,</span> <span class="n">name</span><span class="p">:</span> <span class="s">"/device:GPU:0"</span>
<span class="n">device_type</span><span class="p">:</span> <span class="s">"GPU"</span>
<span class="n">memory_limit</span><span class="p">:</span> <span class="mi">12050019124</span>
<span class="n">locality</span> <span class="p">{</span>
<span class="n">bus_id</span><span class="p">:</span> <span class="mi">1</span>
<span class="n">links</span> <span class="p">{</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="n">incarnation</span><span class="p">:</span> <span class="mi">2152078726800988611</span>
<span class="n">physical_device_desc</span><span class="p">:</span> <span class="s">"device: 0, name: Tesla K80, pci bus id: 0000:09:00.0, compute capability: 3.7"</span>
<span class="p">]</span>
</code></pre></div></div>
<p><img src="/lessons/assets/img/tutorials/tensorflow-sherlock.png" alt="/lessons/assets/img/tutorials/tensorflow-sherlock.png" /></p>
<h2 id="debugging">Debugging</h2>
<p>Tensorflow is really buggy, and this solution is a bit hacky, but when you get it up and running,
it really is a great workflow! That said, I want to help you to resolve issues that you might run into,
so I’ve written up some common pitfalls that I hit during development. If you still need help, please
reach out to our group!</p>
<p><strong>There is a missing module!</strong></p>
<p>This actually happened for me - I got an error message that there wasn’t a module called google protobuf.
This is a bug with the installation of tensorflow on sherlock, because it should be loaded if it’s required.
No fear! You can fix it yourself, and even from your terminal. First, do this:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">os</span>
<span class="n">os</span><span class="p">.</span><span class="n">system</span><span class="p">(</span><span class="s">"pip install --user protobuf"</span><span class="p">)</span>
</code></pre></div></div>
<p>And then restart the kernal from the interface to have changes take effect (you will
get some glorious error messages if you don’t).</p>
<p><strong>I get error code 256 when I try to install a module</strong></p>
<p>This usually means you are trying to uninstall or install to somewhere on the system where you don’t
have permission. The best thing to do to get around this is to, in an interactive shell outside
of the notebook, unset your python path, and use a version of pip that is under your control (and
after added to your PYTHONPATH) to install the needed module. E.g.:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">unset </span>PYTHONPATH
wget https://bootstrap.pypa.io/get-pip.py
<span class="nv">$HOME</span>/anaconda3/bin/python get-pip.py
<span class="nv">$HOME</span>/anaconda3/bin/python <span class="nt">-m</span> pip <span class="nb">install </span>google
</code></pre></div></div>
<p>and then in your <code class="language-plaintext highlighter-rouge">.bashrc</code> make sure this path is hit first (note you may need to adjust the location
and version of python, this was for Anaconda3 installed in the default location)!</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># in .bashrc</span>
<span class="nb">export </span><span class="nv">PYTHONPATH</span><span class="o">=</span><span class="nv">$HOME</span>/anaconda3/lib/python3.6/site-packages:<span class="nv">$PYTHONPATH</span>
<span class="nb">export </span><span class="nv">PATH</span><span class="o">=</span><span class="nv">$HOME</span>/anaconda3/bin:<span class="nv">$PATH</span>
</code></pre></div></div>
<p><strong>My password doesn’t work!</strong></p>
<p>I ran into this issue with my password, and it seemed to be because I had set the password
with one version of jupyter, and then was using the second. To resolve this, I shut down
the server (and killed the job), and then deleted the <code class="language-plaintext highlighter-rouge">$HOME/.jupyter</code> folder and ran
<code class="language-plaintext highlighter-rouge">jupyter notebook password</code> again, both for Python 2 and Python 3, and set the same password
for both. I then started a new terminal and session on my local machine, and the changes
were picked up.</p>
<p><strong>The terminal is hanging!</strong></p>
<p>To make this automated, we issue commands from the <code class="language-plaintext highlighter-rouge">start.sh</code> script to capture output
from sherlock, using <code class="language-plaintext highlighter-rouge">ssh sherlock</code> to send the command. If it’s the case that your login
session has expired (or you got the wrong password), then you might have a password
prompt (that you can’t see) that looks like the terminal is hanging. If this seems to be
the case, try opening a new terminal window, and authenticating with sherlock again (<code class="language-plaintext highlighter-rouge">ssh sherlock pwd</code> should trigger the login authentication flow.)</p>
<p><strong>Failed to setup local forwarding</strong></p>
<p>If you have a hanging process (if you killed a session and now can’t recreate it) you might get an error
message about not being able to set up the port forwarding! What you want to do is use <code class="language-plaintext highlighter-rouge">ps</code> to list
processes with ssh, find the process id, and kill it. Here is an example.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Here I'm searching for processes with ssh</span>
<span class="nv">$ </span>ps aux | <span class="nb">grep </span>ssh
vanessa 749 0.0 0.0 44792 5216 pts/18 S 02:02 0:00 ssh <span class="nt">-L</span> 56143:localhost:56143 sherlock ssh <span class="nt">-L</span> 56143:localhost:56143 <span class="nt">-N</span> sh-113-14
vanessa 909 0.0 0.0 14228 984 pts/18 S+ 02:04 0:00 <span class="nb">grep</span> <span class="nt">--color</span><span class="o">=</span>auto ssh
vanessa 19987 0.0 0.0 49056 6608 pts/19 S+ Jul19 0:07 ssh <span class="nt">-XY</span> vsochat@login.sherlock.stanford.edu
vanessa 32442 0.0 0.0 44792 5260 pts/18 S 01:48 0:00 ssh <span class="nt">-L</span> 56143:localhost:56143 sherlock ssh <span class="nt">-L</span> 56143:localhost:56143 <span class="nt">-N</span> sh-ln01
</code></pre></div></div>
<p>See the last one, with pid <code class="language-plaintext highlighter-rouge">32442</code>? That’s the one I want to kill:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">kill </span>32442
<span class="c"># Is it gone?</span>
<span class="nv">$ </span>ps aux| <span class="nb">grep </span>ssh
vanessa 749 0.0 0.0 44792 5216 pts/18 S 02:02 0:00 ssh <span class="nt">-L</span> 56143:localhost:56143 sherlock ssh <span class="nt">-L</span> 56143:localhost:56143 <span class="nt">-N</span> sh-113-14
vanessa 922 0.0 0.0 14228 988 pts/18 R+ 02:04 0:00 <span class="nb">grep</span> <span class="nt">--color</span><span class="o">=</span>auto ssh
vanessa 19987 0.0 0.0 49056 6608 pts/19 S+ Jul19 0:07 ssh <span class="nt">-XY</span> vsochat@login.sherlock.stanford.edu
<span class="o">[</span>1]+ Exit 255 ssh <span class="nt">-L</span> 56143:localhost:56143 sherlock ssh <span class="nt">-L</span> 56143:localhost:56143 <span class="nt">-N</span> sh-ln01
</code></pre></div></div>
<p>It just exited! At this point, it’s easiest to run end.sh and then start.sh again.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bash end.sh singularity-jupyter
</code></pre></div></div>
<p><strong>My notebook doesn’t have GPU</strong></p>
<p>We have a simple method that checks for the <code class="language-plaintext highlighter-rouge">--partition</code> to be <code class="language-plaintext highlighter-rouge">gpu</code>, and if this is the case,
it appends <code class="language-plaintext highlighter-rouge">--gres gpu:1</code>. If you want to customize this further, just do so by changing
the entire <code class="language-plaintext highlighter-rouge">PARTITION</code> environment variable to include both the <code class="language-plaintext highlighter-rouge">--partition</code> and <code class="language-plaintext highlighter-rouge">--gres</code> (and
any other flags) of your liking. Take a look <a href="https://www.sherlock.stanford.edu/docs/user-guide/gpu/">here</a>
to see rules for usage, and your options.</p>
<p><strong>There is some other problem, arggg!</strong></p>
<p>Well what are you waiting for? Ask for help by <a href="https://github.com/vsoch/forward" target="_blank">opening an issue</a>! or submitting a pull request.</p>
<h2 id="wait-why-not-containers">Wait, Why not Containers?</h2>
<p>It occurs to me, it’s pretty annoying to need to load modules, and still have broken dependencies.
Would you be interested in a function like this to start an interactive container? And then allow it to
select any container of your choosing? While this isn’t a container cluster proper, this is likely
not too hard to do! Please let me know (@vsoch on <a href="https://www.github.com/vsoch">Github</a>) or <a href="https://www.twitter.com/vsoch">Twitter</a>) if this would be of interest to you, and consider it done!</p>
<p>Do you have questions or want to see another tutorial? Please <a href="https://www.github.com/vsoch/lessons/issues">reach out</a>!</p>vsochThis is a followup to our original post that described how to get access to a jupyter notebook on Sherlock with port forwarding! Today we will extend the example to a new set of sbatch scripts that will start up a jupyter notebook with tensorflow. Want a GPU? We can do that too. If you want to container-based verison of this tutorial (yes, you can deploy jupyter from a container too!) then see this post. Let’s get started!Jupyter (R Kernel) Notebooks on Sherlock2018-07-09T00:00:00+00:002018-07-09T00:00:00+00:00https://vsoch.github.io/lessons/lessons/sherlock-juputer-r<p>Today we are going to use the forward <a href="https://github.com/vsoch/forward">tool</a> to set up a Jupyter notebook
with an R kernel and port forwarding to our local machine. Today we are going to:</p>
<ul>
<li>set up the <a href="IRKernel">https://github.com/IRkernel/IRkernel</a> package for Jupyter</li>
<li>run a password protected jupyter notebook on a cluster node</li>
<li>access the notebook in a browser on our local machine</li>
</ul>
<h2 id="why-should-i-use-this">Why should I use this?</h2>
<p>Since we want to harness the local module installation
of R and jupyter, this will be run via environment modules and not containers. The use case for this is if you
want to explore large datasets, but not necessary run the notebook at scale or share your steps.
You just want to mess around and are okay to be working non-reproducibly. If you
want to do either of these things, you are <em>strongly</em> recommended to build a container that can more reliably
reproduce and thus share your environment.</p>
<h2 id="1-on-your-computer">1. On Your Computer</h2>
<p>The repository contains a set of scripts for setting up automatic port forwarding on sherlock with jupyter notebooks.
There are a set of commands you will need to run just once to configure the tool, and then general “start”, “end” and “resume” operations for interacting with notebook jobs. The repository README has instructions for setup and usage, and we will also walk through them here.</p>
<h3 id="step-1-download-the-repository">Step 1: Download the Repository</h3>
<p>Make sure you put the folder somewhere meaningful. A subfolder of <code class="language-plaintext highlighter-rouge">$HOME</code> is a suggestion. This will be your working location for future use of the tool, as it holds scripts and a parameter file, <code class="language-plaintext highlighter-rouge">params.sh</code></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git clone https://github.com/vsoch/forward
<span class="nb">cd </span>forward
</code></pre></div></div>
<p>Let’s take a look at what we have here!</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>├── end.sh <span class="o">(</span><span class="nt">--</span> end a session
├── resume.sh <span class="o">(</span><span class="nt">--</span> resume a started session
├── sbatches <span class="o">(</span><span class="nt">--</span> batch scripts you can run to forward a port to!
├── sherlock
└── farmshare
├── setup.sh <span class="o">(</span><span class="nt">--</span> run once to <span class="nb">set </span>up the tool
└── start.sh <span class="o">(</span><span class="nt">--</span> how you start a connection
</code></pre></div></div>
<h3 id="step-2-generate-your-parameters">Step 2: Generate your Parameters</h3>
<p>This is a one time generation script that will create a parameter text file with a port, username, and cluster resource. You can generate it by running the script <a href="https://github.com/vsoch/forward/blob/master/setup.sh">setup.sh</a>:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>bash setup.sh
Sherlock username <span class="o">></span> tacocat
Next, pick a port to use. If someone <span class="k">else </span>is port forwarding using that
port already, this script will not work. If you pick a random number <span class="k">in </span>the
range 49152-65335, you should be good.
Port to use <span class="o">></span> 56143
Next, pick the sherlock partition on which you will be running your
notebooks. If your PI has purchased dedicated hardware on sherlock, you can use
that partition. Otherwise, leave blank to use the default partition <span class="o">(</span>normal<span class="o">)</span><span class="nb">.</span>
Sherlock partition <span class="o">(</span>default: normal<span class="o">)</span> <span class="o">></span>
</code></pre></div></div>
<p>Notice that I’ve changed the default browser (Safari on a Mac) to firefox on Ubuntu. Also note that I pressed enter
to use the default queue. The resulting file is a simple text file:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">cat </span>params.sh
<span class="nv">USERNAME</span><span class="o">=</span><span class="s2">"tacocat"</span>
<span class="nv">PORT</span><span class="o">=</span><span class="s2">"56143"</span>
<span class="nv">PARTITION</span><span class="o">=</span><span class="s2">"normal"</span>
<span class="nv">RESOURCE</span><span class="o">=</span><span class="s2">"sherlock"</span>
<span class="nv">MEM</span><span class="o">=</span><span class="s2">"20G"</span>
<span class="nv">TIME</span><span class="o">=</span><span class="s2">"8:00:00"</span>
<span class="nv">CONTAINERSHARE</span><span class="o">=</span><span class="s2">"/scratch/users/vsochat/share"</span>
</code></pre></div></div>
<h3 id="step-3-ssh-credentials">Step 3: SSH Credentials</h3>
<p>It wouldn’t be so great if anyone with your username and a port could access your notebook! For this reason we need to create ssh credentials. You might have this already set up if you’ve followed the <a href="https://www.sherlock.stanford.edu/docs/getting-started/connecting/" target="_blank">instructions here</a> for Sherlock. You
can see what you have by looking at your <code class="language-plaintext highlighter-rouge">~/.ssh/config</code>:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cat</span> ~/.ssh/config
</code></pre></div></div>
<p>What we want to do is have this configuration. Add this to your <code class="language-plaintext highlighter-rouge">~/.ssh/config</code>. The repository also has a quick way to generate your ssh config, and print it to the screen. See the hosts folder? In there are small helper scripts to print the configurations to the screen! We have one for Sherlock:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">ls </span>hosts/
sherlock_ssh.sh
farmshare_ssh.sh
</code></pre></div></div>
<p>Running the script will print the configuration to the screen:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>bash hosts/sherlock_ssh.sh
Sherlock username <span class="o">></span> tacocat
Host sherlock
User tacocat
Hostname sh-ln03.stanford.edu
GSSAPIDelegateCredentials <span class="nb">yes
</span>GSSAPIAuthentication <span class="nb">yes
</span>ControlMaster auto
ControlPersist <span class="nb">yes
</span>ControlPath ~/.ssh/%l%r@%h:%p
</code></pre></div></div>
<p>If you don’t have a file existing at <code class="language-plaintext highlighter-rouge">~/.ssh/config</code> then you will need to create it, and you
can do the entire step above programatically:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bash hosts/sherlock_ssh.sh <span class="o">>></span> ~/.ssh/config
</code></pre></div></div>
<p>Remember that this is a suggested configuration. The login node is generated randomly above, so you could
change it to be a different node (e.g., <code class="language-plaintext highlighter-rouge">sh-ln03.sherlock.stanford.edu</code>) or select from all of them (<code class="language-plaintext highlighter-rouge">login.sherlock.stanford.edu</code>)`. Your computer is ready to go! Let’s move onto Sherlock. We will come back here when it’s time to start a notebook.</p>
<h2 id="2-on-sherlock">2. On Sherlock</h2>
<p><br /></p>
<h3 id="21-install-r-kernel">2.1 Install R Kernel</h3>
<p>The script will run this for you, but since devtools takes a while to compile, I recommend
that you log into sherlock, grab a development node with sdev, and install these first
to verify it works okay. First, grab the node and load the modules that have jupyter
and R.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sdev
module load py-jupyter/1.0.0_py36
module load R/3.5.1
</code></pre></div></div>
<p>Next, install devtools and IRKernel from Github. These commands must end in success!”</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Install devtools and IRkernel</span>
Rscript <span class="nt">-e</span> <span class="s2">"install.packages('devtools', repos='http://cran.us.r-project.org');"</span>
Rscript <span class="nt">-e</span> <span class="s2">"library('devtools'); devtools::install_github('IRkernel/IRkernel')"</span>
<span class="c"># register the kernel in the current R installation</span>
Rscript <span class="nt">-e</span> <span class="s2">"IRkernel::installspec()"</span>
</code></pre></div></div>
<p>That’s it! Now let’s make sure we set up the jupyter notebook to have a headless password.
This is important so that if your port is discovered, a malicious user can’t delete or mess
with your files.</p>
<h3 id="22-jupyter-notebook">2.2. Jupyter notebook</h3>
<p>We will want to set up a password for our Jupyter notebook, and we can do this programatically
before starting it. First, let’s load the module for Python, and install jupyter notebook! The version of
Python that you choose is up to you.</p>
<p>As a reminder, we’ve already loaded jupyter</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
module load py-jupyter/1.0.0_py36
<span class="nv">$ </span>which jupyter
/share/software/user/open/py-jupyter/1.0.0_py36/bin/jupyter
</code></pre></div></div>
<p>We will need to secure our notebooks with a password. Pick a strong one!</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>jupyter notebook password
</code></pre></div></div>
<p>It will be associated with your username, saved to a local file:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Enter password:
Verify password:
<span class="o">[</span>NotebookPasswordApp] Wrote hashed password to /home/users/vsochat/.jupyter/jupyter_notebook_config.json
</code></pre></div></div>
<p>The above might pause or hang a little bit, at least it did when I did.</p>
<h2 id="3-usage">3. Usage</h2>
<p>We have just set up a password on Sherlock, and are now back on our <em>local machine</em>. Here are the general commands to start and stop sessions. In the tutorial below, we will walk through using Jupyter notebook.</p>
<h3 id="31-start-a-session">3.1. Start a Session</h3>
<p>From the directory where we cloned, we can start a session using the <a href="https://github.com/vsoch/forward/blob/master/start.sh">start.sh</a> script. This is a general script to start any kind of session, and here we will show how to start a jupyter notebook in a specific directory:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bash start.sh <host>/<software> <path>
bash start.sh sherlock/r-jupyter /path/to/dir
</code></pre></div></div>
<p>or in a nutshell, you can do this and use defaults:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bash start.sh sherlock/r-jupyter
</code></pre></div></div>
<p>What’s going on? It will look in the folder of <a href="https://github.com/vsoch/forward/blob/master/sbatches">sbatch scripts</a> and find one named correpondingly to the command we issued, a script named <code class="language-plaintext highlighter-rouge">r-jupyter.sbatch</code> in the <code class="language-plaintext highlighter-rouge">sbatches</code> folder in a subfolder named “sherlock”. At this point, you should see expected output, and a brief set
of steps to do the same install we already did.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>bash start.sh sherlock/r-jupyter
<span class="o">==</span> Finding Script <span class="o">==</span>
Looking <span class="k">for </span>sbatches/sherlock/sherlock/r-jupyter.sbatch
Looking <span class="k">for </span>sbatches/sherlock/r-jupyter.sbatch
Script sbatches/sherlock/r-jupyter.sbatch
<span class="o">==</span> Checking <span class="k">for </span>previous notebook <span class="o">==</span>
No existing sherlock/r-jupyter <span class="nb">jobs </span>found, continuing...
<span class="o">==</span> Getting destination directory <span class="o">==</span>
<span class="o">==</span> Uploading sbatch script <span class="o">==</span>
r-jupyter.sbatch 100% 587 0.6KB/s 00:00
<span class="o">==</span> Submitting sbatch <span class="o">==</span>
sbatch <span class="nt">--job-name</span><span class="o">=</span>sherlock/r-jupyter <span class="nt">--partition</span><span class="o">=</span>russpold <span class="nt">--output</span><span class="o">=</span>/home/users/vsochat/forward-util/r-jupyter.sbatch.out <span class="nt">--error</span><span class="o">=</span>/home/users/vsochat/forward-util/r-jupyter.sbatch.err <span class="nt">--mem</span><span class="o">=</span>12G <span class="nt">--time</span><span class="o">=</span>8:00:00 /home/users/vsochat/forward-util/r-jupyter.sbatch 56143 <span class="s2">""</span>
Submitted batch job 30407076
<span class="o">==</span> View logs <span class="k">in </span>separate terminal <span class="o">==</span>
ssh sherlock <span class="nb">cat</span> /home/users/vsochat/forward-util/r-jupyter.sbatch.out
ssh sherlock <span class="nb">cat</span> /home/users/vsochat/forward-util/r-jupyter.sbatch.err
<span class="o">==</span> Waiting <span class="k">for </span>job to start, using exponential backoff <span class="o">==</span>
Attempt 0: not ready yet... retrying <span class="k">in </span>1..
Attempt 1: not ready yet... retrying <span class="k">in </span>2..
Attempt 2: not ready yet... retrying <span class="k">in </span>4..
Attempt 3: not ready yet... retrying <span class="k">in </span>8..
Attempt 4: not ready yet... retrying <span class="k">in </span>16..
Attempt 5: resources allocated to sh-106-04!..
sh-106-04
sh-106-04
notebook running on sh-106-04
<span class="o">==</span> Setting up port forwarding <span class="o">==</span>
ssh <span class="nt">-L</span> 43453:localhost:43453 sherlock ssh <span class="nt">-L</span> 43453:localhost:43453 <span class="nt">-N</span> sh-106-04 &
<span class="o">==</span> Connecting to notebook <span class="o">==</span>
Installing package into ‘/home/users/vsochat/R/x86_64-pc-linux-gnu-library/3.5’
<span class="o">(</span>as ‘lib’ is unspecified<span class="o">)</span>
trying URL <span class="s1">'http://cran.us.r-project.org/src/contrib/devtools_2.0.1.tar.gz'</span>
Content <span class="nb">type</span> <span class="s1">'application/x-gzip'</span> length 388953 bytes <span class="o">(</span>379 KB<span class="o">)</span>
<span class="o">==================================================</span>
downloaded 379 KB
<span class="k">*</span> installing <span class="k">*</span><span class="nb">source</span><span class="k">*</span> package ‘devtools’ ...
<span class="k">**</span> package ‘devtools’ successfully unpacked and MD5 sums checked
<span class="k">**</span> R
<span class="k">**</span> inst
<span class="k">**</span> byte-compile and prepare package <span class="k">for </span>lazy loading
<span class="k">**</span> <span class="nb">help</span>
<span class="k">***</span> installing <span class="nb">help </span>indices
<span class="k">***</span> copying figures
<span class="k">**</span> building package indices
<span class="k">**</span> installing vignettes
<span class="k">**</span> testing <span class="k">if </span>installed package can be loaded
<span class="k">*</span> DONE <span class="o">(</span>devtools<span class="o">)</span>
The downloaded <span class="nb">source </span>packages are <span class="k">in</span>
‘/tmp/Rtmp3JNqt9/downloaded_packages’
Skipping <span class="nb">install </span>of <span class="s1">'IRkernel'</span> from a github remote, the SHA1 <span class="o">(</span>97c492b2<span class="o">)</span> has not changed since last install.
Use <span class="sb">`</span>force <span class="o">=</span> TRUE<span class="sb">`</span> to force installation
<span class="o">[</span>InstallKernelSpec] Removing existing kernelspec <span class="k">in</span> /home/users/vsochat/.local/share/jupyter/kernels/ir
<span class="o">[</span>InstallKernelSpec] Installed kernelspec ir <span class="k">in</span> /home/users/vsochat/.local/share/jupyter/kernels/ir
<span class="o">[</span>I 22:40:41.752 NotebookApp] Writing notebook server cookie secret to /tmp/jupyter/notebook_cookie_secret
<span class="o">[</span>I 22:40:43.430 NotebookApp] Serving notebooks from <span class="nb">local </span>directory: /home/users/vsochat
<span class="o">[</span>I 22:40:43.430 NotebookApp] 0 active kernels
<span class="o">[</span>I 22:40:43.430 NotebookApp] The Jupyter Notebook is running at: http://localhost:56143/
<span class="o">[</span>I 22:40:43.430 NotebookApp] Use Control-C to stop this server and shut down all kernels <span class="o">(</span>twice to skip confirmation<span class="o">)</span><span class="nb">.</span>
<span class="o">==</span> View logs <span class="k">in </span>separate terminal <span class="o">==</span>
ssh sherlock <span class="nb">cat</span> /home/users/vsochat/forward-util/r-jupyter.sbatch.out
ssh sherlock <span class="nb">cat</span> /home/users/vsochat/forward-util/r-jupyter.sbatch.err
<span class="o">==</span> Instructions <span class="o">==</span>
1. Password, output, and error printed to this terminal? Look at logs <span class="o">(</span>see instruction above<span class="o">)</span>
2. Browser: http://sh-02-21.int:56143/ -> http://localhost:56143/...
3. To end session: bash end.sh sherlock/r-jupyter
</code></pre></div></div>
<p>When you open your browser to the address, you will see a prompt for the password that you
created previously:</p>
<p><img src="/lessons/assets/img/tutorials/jupyter-password.png" alt="/lessons/assets/img/tutorials/jupyter-password.png" /></p>
<p>If you have an already running session, you will see this message. When you open the interface,
you can find that R is now an option for a Kernel!</p>
<p><img src="/lessons/assets/img/tutorials/jupyter-r.png" alt="/lessons/assets/img/tutorials/jupyter-r.png" /></p>
<p>And now you have an R session.</p>
<p><img src="/lessons/assets/img/tutorials/jupyter-r2.png" alt="/lessons/assets/img/tutorials/jupyter-r2.png" /></p>
<h3 id="32-resume-a-session">3.2. Resume a Session</h3>
<p>Sometimes the job can still be running, but the port forwarding has stopped (has your computer
ever gone to sleep?) In this case, you can resume the session using the equivalent command, but
use resume.sh:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bash resume.sh sherlock/r-jupyter<span class="sb">`</span>
</code></pre></div></div>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>bash start.sh sherlock/r-jupyter
<span class="o">==</span> Checking <span class="k">for </span>previous notebook <span class="o">==</span>
Found existing job <span class="k">for </span>sherlock/r-jupyter, sh-101-04 end.sh or resume.sh
</code></pre></div></div>
<h3 id="33-stop-a-session">3.3. Stop a Session</h3>
<p>When you want to stop the session (killing the job) run the equivalent command, but use the end.sh script.
The command is based on the name of the job (the sbatch script name) so to kill the previous job we created,
we would do:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>bash end.sh sherlock/r-jupyter
Killing sherlock/r-jupyter slurm job on sherlock
Killing listeners on sherlock
</code></pre></div></div>
<p>There are additional details and debugging tips in the <a href="https://github.com/vsoch/forward" target="_blank">main repository</a>! Happy Jupyter-ing!</p>
<p>Do you have questions or want to see another tutorial? Please <a href="https://www.github.com/vsoch/lessons/issues">reach out</a>!</p>vsochToday we are going to use the forward tool to set up a Jupyter notebook with an R kernel and port forwarding to our local machine. Today we are going to: