Changes between Version 4 and Version 5 of SOPs/creatingVirtualEnvs

Timestamp:

10/09/25 13:33:49 (8 weeks ago)

Author:

xinlei.gao

Comment:

--

SOPs/creatingVirtualEnvs

@@ -66 +66 @@
 Afterward, when you create a new notebook or change the kernel of an existing one, you will find "Python (RNAseq_2024a)" as an available kernel.
 
+You can also use the virtual environment from a jupyter notebook on the fry cluster. In this way, you can require more computational resources (like multiple cpus/gpus or big memory).
+
+To do it, first you need to install jupyterlab in your virtual environment:
+
+{{{
+conda activate RNAseq_2024a
+conda install jupyterlab
+conda deactivate
+}}}
+
+Next, create a slurm script as below -- here we will save it as 'jupyterOnCluster.sbatch'
+
+{{{
+#!/bin/bash
+# Configuration values for a SLURM batch job.
+# One leading hash(#) before the word SBATCH is not a comment, but two are.
+#SBATCH --job-name=jobName
+#SBATCH --nodes=1                 # Ensure that all cores are on one machine
+#SBATCH --ntasks=1                # Run a single task
+#SBATCH --ntasks-per-core=8       # Enter number of cores/threads you wish to request
+#SBATCH --time 02:00:00           # max time that the job/jupyter instance actually needs to be running (in hh:mm:ss format) 
+#SBATCH --mem=16gb                # Enter amount of memory you wish to request
+#SBATCH --partition=20            # partition (queue) to use
+#SBATCH --output %x-%j.out        # name of output file.  %x is the job-name %j is jobid
+#SBATCH --mail-type=ALL
+#SBATCH --mail-user=your.name@wi.mit.edu
+
+# If using a conda environment 
+# (if not, delete or comment out next 3 lines):
+# Activate conda itself
+eval "$(/nfs/BaRC/USER/conda/bin/conda shell.bash hook)"
+# Activate your specific conda environment
+conda activate RNAseq_2024a
+# If using a python virtual environment 
+# (if not, delete or comment out next line):
+#source /path/to/your/python/virtualenv/directory/bin/activate
+
+
+# Workaround for jupyter bug
+unset XDG_RUNTIME_DIR
+
+
+jupyter-lab \
+--no-browser \
+--port-retries=0 \
+--ip=0.0.0.0 \
+--port=`shuf -i 8900-10000 -n 1` \
+--notebook-dir=/ \
+--LabApp.default_url="/lab/tree/home/$(whoami)"
+
+
+
+
+#/usr/bin/mail -s "$SLURM_JOB_NAME $SLURM_JOB_ID" yourwhiteheadusername@wi.mit.edu < %x-%j.out  #uncomment (remove hash at beginning of this line) if you want your job output emailed to you
+
+}}}
+
+Then submit the slurm job:
+
+{{{
+sbatch jupyterOnCluster.sbatch
+}}}
+
+After the slurm job gets running on the cluster, an output file named 'jobName-%j.out' will be created.
+
+Open the output file and you will find a URL pointing to the jupyter notebook. Copy and paste the URL into your web browser, you will get access to the jupyter notebook with your virual environment.
+
+Make sure to set a reasonable time limit to your slurm job or cancel it after you finish working on it, to avoid occupying computational resources unnecessarily.
+
+Reference to IT's instructions on this topic: https://docs.google.com/document/d/1eYGVn5M402n2b9pueWdHoeLx84Ue-IGQG8t6e2QK7To
+
 Save the environment