Tutorial on Linux
Papercompanion for APL Center of Geospace Storms
Pokeballs capture monsters in Containers, and we capture a massive piece of high-performance compute code named Kaiju and Gamera in a Container
This is a demo lab with content from Johns-Hopkins APL
Here, we showcase, how a researcher may share their paper-companions with an interactive part. Remember, the free-tier here is 1 hr (for readers) per day. So, we will keep the tutorial short.
docker run -it adlsregistrysbx.azurecr.io/pokeball/devel:latest /bin/bash -c "cmake -DALLOW_INVALID_COMPILERS=ON /app/. && make gamera -j9 && /app/quickstart/loop2d/prepare_loop2d.py && ./bin/gamera.x loop2d.xml"
First, let's configure rclone to connect to the Zenodo record using a heredoc. This command creates the necessary directory and writes the configuration file in one step.
mkdir -p ~/.config/rclone/
cat <<EOF > ~/.config/rclone/rclone.conf
[remote]
type = doi
doi = 10.5281/zenodo.8178574
EOF
mkdir -p ~/zenodo_mount
rclone mount remote: ~/zenodo_mount --vfs-cache-mode full &
cat <<EOF > list_zenodo_files.py
import os
import sys
# The path where you mounted the Zenodo remote
mount_path = os.path.expanduser('~/zenodo_mount')
print(f"--- Accessing Zenodo files at: {mount_path} ---")
# Check if the mount point exists and is accessible
if not os.path.isdir(mount_path) or not os.listdir(mount_path):
print(f"Error: Mount point '{mount_path}' is not accessible or is empty.")
print("Please ensure the 'rclone mount' command is running correctly.")
sys.exit(1)
# List all files and directories at the root of the mount
print("Files found in the Zenodo record:")
try:
for item in sorted(os.listdir(mount_path)):
item_path = os.path.join(mount_path, item)
if os.path.isfile(item_path):
size = os.path.getsize(item_path)
print(f"- {item} ({size / 1024:.2f} KB)")
except OSError as e:
print(f"An error occurred while listing files: {e}")
print("-------------------------------------------------")
EOF
python3 list_zenodo_files.py
Accessing Zenodo files at /home/laborant/zenodo_mount 💡
laborant@docker-01:~$ python3 list_zenodo_files.py
--- Accessing Zenodo files at: /home/laborant/zenodo_mount ---
Files found in the Zenodo record:
- msphere.deltab.h5 (268479.28 KB)
- msphere.deltab.xmf (95.62 KB)
- msphere.mhdrcm.h5 (108968.14 KB)
- msphere.mix.h5 (222519.42 KB)
- msphere.mix.xmf (105.57 KB)
- msphere.rcm.h5 (831710.90 KB)
- msphere.volt.h5 (190778.08 KB)
- msphere_0008_0008_0001_0000_0000_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0000_0001_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0000_0002_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0000_0003_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0000_0004_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0000_0005_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0000_0006_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0000_0007_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0001_0000_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0001_0001_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0001_0002_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0001_0003_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0001_0004_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0001_0005_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0001_0006_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0001_0007_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0002_0000_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0002_0001_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0002_0002_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0002_0003_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0002_0004_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0002_0005_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0002_0006_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0002_0007_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0003_0000_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0003_0001_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0003_0002_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0003_0003_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0003_0004_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0003_0005_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0003_0006_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0003_0007_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0004_0000_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0004_0001_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0004_0002_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0004_0003_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0004_0004_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0004_0005_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0004_0006_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0004_0007_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0005_0000_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0005_0001_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0005_0002_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0005_0003_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0005_0004_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0005_0005_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0005_0006_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0005_0007_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0006_0000_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0006_0001_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0006_0002_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0006_0003_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0006_0004_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0006_0005_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0006_0006_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0006_0007_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0007_0000_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0007_0001_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0007_0002_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0007_0003_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0007_0004_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0007_0005_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0007_0006_0000.gam.h5 (140803.00 KB)
- msphere_0008_0008_0001_0007_0007_0000.gam.h5 (140803.00 KB)
- sim.000000.xmf (533.36 KB)
- sim.000001.xmf (533.36 KB)
- sim.000002.xmf (533.36 KB)
- sim.000003.xmf (533.36 KB)
- sim.000004.xmf (533.36 KB)
- sim.000005.xmf (533.36 KB)
- sim.000006.xmf (533.36 KB)
- sim.000007.xmf (533.36 KB)
- sim.000008.xmf (533.36 KB)
- sim.000009.xmf (533.36 KB)
- sim.000010.xmf (534.48 KB)
- sim.000011.xmf (534.48 KB)
- sim.000012.xmf (534.48 KB)
-------------------------------------------------
Create a Plot on the official HPC data (remotely mounted)
As instructed in the paper-companion, install required packages
pip3 install kaipy h5py numpy matplotlib --break-system-packages
Now, let's create and run the Python script (see publication and companion ) to plot the data.
cat <<EOF > plot_graph.py
import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt
import matplotlib.gridspec as gridspec
import kaipy.kaiViz as kv
import kaipy.kaiH5 as kh5
import kaipy.gamera.magsphere as msph
import kaipy.kaiTools as ktools
import kaipy.remix.remix as remix
import datetime
import os
import sys
# Point to data
fdir = os.path.expanduser('~/zenodo_mount')
ftag = "msphere"
if not os.path.isdir(fdir):
print(f"Error: Data directory '{fdir}' not found. Is rclone mounted?", file=sys.stderr)
sys.exit(1)
# Load data from magnetosphere
gsph = msph.GamsphPipe(fdir,ftag,doFast=False)
# Get data from ionosphere (mix)
mixFile = os.path.join(fdir, ftag + ".mix.h5") #Ionospheric data (remix)
dbFile = os.path.join(fdir, ftag + ".deltab.h5") #Ground DB data
# Data is loaded, now choose which time slice to display
nSteps = len(gsph.MJDs)
print(f"\tFound {nSteps} steps")
print(f"\tSteps = [{gsph.s0} ,{gsph.sFin} ]")
MJDMin = gsph.MJDs.min()
MJDMax = gsph.MJDs.max()
print(f"\tRange = {ktools.MJD2UT(MJDMin)} to {ktools.MJD2UT(MJDMax)} ")
# Now pick a step to plot in the range above
nStp = 6 #Picking interesting value
MJD = gsph.MJDs[nStp-gsph.s0]
print(f"Using Step {nStp} , {ktools.MJD2UT(MJD)} ")
# Read necessary data for this time slice
ion = remix.remix(mixFile, nStp)
dBz = gsph.DelBz(nStp)
P = gsph.EggSlice("P" ,nStp,doEq=True)
# Get synthetic SMRs/SMLs from ground db data
SMR = kh5.PullAtt(dbFile,"SMR" ,nStp)
SMR06 = kh5.PullAtt(dbFile,"SMR_06",nStp)
SMR12 = kh5.PullAtt(dbFile,"SMR_12",nStp)
SMR18 = kh5.PullAtt(dbFile,"SMR_18",nStp)
SMR00 = kh5.PullAtt(dbFile,"SMR_00",nStp)
SML = kh5.PullAtt(dbFile,"SML" ,nStp)
SMU = kh5.PullAtt(dbFile,"SMU" ,nStp)
SME = kh5.PullAtt(dbFile,"SME" ,nStp)
# Now make a figure
figSz = (10,11)
xyBds = [-12.5,10.0,-10,10]
dbMax = 50.0
vDB = kv.genNorm(dbMax)
cbDB = "RdGy_r"
vP = kv.genNorm(1.0e-1,1.0e+2,doLog=True)
pVals = np.linspace(1.0,1.0e+2,25)
cmP = "viridis"
remix.facMax = 2.0
plt.close('all')
fig = plt.figure(figsize=figSz)
gs = gridspec.GridSpec(4,3,height_ratios=[1,20,1,1])
Ax = fig.add_subplot (gs[ 1,:])
AxCB1 = fig.add_subplot(gs[-1,0])
AxCB2 = fig.add_subplot(gs[-1,1])
AxCB3 = fig.add_subplot(gs[-1,2])
cbb1 = kv.genCB(AxCB1,vDB,r"Residual Field [nT]",cM=cbDB,Ntk=5)
cbb1.set_ticks(np.linspace(-dbMax,dbMax,5))
cbb2 = kv.genCB(AxCB2,vP ,r"Pressure [nPa]" ,cM=cmP)
cbb3 = kv.genCB(AxCB3,kv.genNorm(remix.facMax),"FAC",cM=remix.facCM)
Ax.pcolormesh(gsph.xxi,gsph.yyi,dBz,norm=vDB,cmap=cbDB)
Ax.contour(kv.reWrap(gsph.xxc),kv.reWrap(gsph.yyc),kv.reWrap(P),pVals,norm=vP,cmap=cmP,linewidths=0.5)
gsRM = gs[1,:].subgridspec(20,20)
ion.init_vars('NORTH')
aA = ion.plot('current' ,gs=gsRM[1:7,1:7],doInset=True)
kv.addEarth2D(ax=Ax)
kv.SetAx(xyBds,Ax)
kv.SetAxLabs(Ax,'SM-X [Re]','SM-Y [Re]')
MJD = kh5.tStep(dbFile,nStp,aID="MJD")
utS = ktools.MJD2UT([MJD])
utDT= utS[0]
tStr = utDT.strftime("%m/%d /%Y\\n%H:%M:%S")
aStr = "Auroral Indices [nT]\\nSME = %8.2f \\nSML = %8.2f \\nSMU = %8.2f "%(SME,SML,SMU)
rStr = "Ring Current Indices [nT]\\nSMR = %8.2f \\nDawn-Dusk = %8.2f \\nDay-Night = %8.2f "%(SMR,SMR06-SMR18,SMR12-SMR00)
Ax.set_title(tStr,fontsize="x-large",loc="center")
Ax.set_title(aStr,fontsize="medium" ,loc="right",fontname="monospace")
Ax.set_title(rStr,fontsize="medium" ,loc="left" ,fontname="monospace")
fOut = "ds1pic.png"
kv.savePic(fOut)
print(f"Plot saved to {fOut}")
EOF
python3 plot_graph.py
/home/laborant/zenodo_mount/msphere.h5 not found, looking for MPI database
Found 64 = (8,8,1) ranks
#-Steps |∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙| 23/
#-Steps |∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙| 13
Found 13 timesteps
Time = [36000.006462,39600.003305]
Steps = [0,12]
#-Steps |∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙| 13
Grid size = (192,192,256)
Cells = 9.437184e+06
Variables (Root/Step) = (7,18)
Root: ['Bx0', 'BxD', 'By0', 'ByD', 'Bz0', 'BzD', 'dV']
Step: ['Bx', 'By', 'Bz', 'Cs', 'D', 'Jx', 'Jy', 'Jz', 'P', 'Pb', 'SrcD', 'SrcDT', 'SrcP', 'SrcX1', 'SrcX2', 'Vx', 'Vy', 'Vz']
GameraPipe: b'EARTH'
setting UnitsID
Units Type = EARTH
msphere/Grid |██████████████████████████████| 64/64 [100%] in 1:35.
Found MJD data
Time (Min/Max) = 56368.416667/56368.458333
Found ReMIX data, reading ...
#-Steps |∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙| 15/
#-Steps |∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙| 13
#-Steps |∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙| 13
GameraPipe: b'ReMIX'
setting UnitsID
|██████████████████████████████| 1/1 [100%] in 0.1s (9.74/s)
#-Steps |∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙| 13
#-Steps |∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙| 13
Found 13 steps
Steps = [0 ,12 ]
Range = 2013-03-17 10:00:00.006000 to 2013-03-17 11:00:00.003000
Using Step 6 , 2013-03-17 10:30:00.011000
msphere/Step#6/Bz |█████████████████████████▎ | ▆█▆ 54/64 [84%] in 2:25
Now, you should wait until the task below is green to indicate your plot is ready
Head to the other IDE tab and look at the plot
Other things you can do
➡ There is a full tutorial on authoring content
Shared debugging session
Let s assume, you wanted to run the actual code and show to a colleague the output
First, we create a problem: run the docker container
docker run -it adlsregistrysbx.azurecr.io/pokeball/devel:latest /bin/bash -c "cmake -DALLOW_INVALID_COMPILERS=ON /app/. && make gamera -j9 && /app/quickstart/loop2d/prepare_loop2d.py && ./bin/gamera.x loop2d.xml"
You get an code":"UNAUTHORIZED"
Lets assume, you need my help to debug this. So: go to the right top corner
Share a terminal session with another person
You could now share this URL with me and ask me debug the AUTHORIZATION problem.
Expose ports on HTTP/S: example Jupyter
mkdir -p data
cp *.png data
sudo chown -R 1000:100 data
docker run -it --rm \
-p 8888:8888 \
-v /home/laborant/data:/home/jovyan/data \
jupyter/base-notebook \
start-notebook.sh \
--NotebookApp.ip='0.0.0.0' \
--NotebookApp.open_browser=False \
--NotebookApp.password='' \
--NotebookApp.allow_root=True
Now, open the tab (top right again), but first grab the token that jupyter prints into the shell and input the 8888 as port number (no HTTPS).
Click on the generated URL and a new BrowserTab will open, here paste the Token
and Voila !! 🎉
Explore, test and try out to your ♥️ content. Give feedback on LinkedIn, Slack, X, Bluesky or email constanze.roedig@tuwien.ac.at
Level up your Server Side game — Join 11,000 engineers who receive insightful learning materials straight to their inbox
