Real-life scenario

Let’s imagine you receive the following email and decide to take on this task.

From: eric.gablehauser@caltech.edu
To: sheldon@caltech.edu
Date: Mon, 01 Nov 2007 06:10:30 -0000

Hello, Sheldon

I have prepared the new version of the simulations for the Super Nova study,
but I am going to the Super Nova South congress in Atlanta this week, so could
you run the new simulation to get new results for next week?
I put the new version on my server gbl-srv, you have an account on it.
To launch the simulation use the new server supernova.caltech.edu, you connect
with the login testing.
By the way, it’s important that you put your ssh key on it (I got a mail from
IT-crowd that they will forbid normal password acces from next week).
To launch simulation as usual : runme.py

Best wishes,
Eric Gablehauser
Head of the Physics Department
California Institute of Technology

P.S.: I don’t know if there is enough space on supernova server, in case you could
delete the dat files in the super_nova_test folder, but be careful not to delete
the *.bin files (or others)

P.P.S.: Pay attention that there is no other simulation running: you remember lock problem
that we never solved.

P.P.P.S.: I used some new features of Python 3.9 so it should be run with at least 3.9 python.
I hope we have a good version on supernova server. If it’s not… hum, you should ask somebody
of IT-crowd how to install it.

1. Connection to gbl-srv (server1) to see the file

[sheldon@localhost ~]$ ssh sheldon@gbl-srv.caltech.edu
(sheldon@glb-srv:/home/sheldon)$> ls /group/snova/simulations/
super_nova_simulations.tgz
(sheldon@glb-srv:/home/sheldon)$> exit
[sheldon@localhost ~]$

Tip

ssh : remote login program

2. Getting the file from gbl-srv (server1) (rsync)

[sheldon@localhost ~]$ rsync -aurv sheldon@gbl-srv.caltech.edu:/group/snova/simulations/super_nova_simulations.tgz .
bash: rsync: command not found
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(228) [sender=3.2.3]

Tip

rsync : file-copying tool
-aurv : archive, update, recursive, verbose
archive : “preserve almost everything” ( = -rlptgoD)

Hmmm…?

3. Getting the file from gbl-srv (server1) (scp)

[sheldon@localhost ~]$ scp -r sheldon@gbl-srv.caltech.edu:/group/snova/simulations/super_nova_simulations.tgz .
super_nova_simulations.tgz                                                                 100% 3163KB  11.1MB/s   00:00

Tip

scp : secure copy (remote file copy program)
-r : Recursively copy entire directories

4. Create ssh-key on Sheldon’s computer

[sheldon@localhost ~]$ ssh-keygen -t rsa -f .ssh/id_rsa_supernova
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in .ssh/id_rsa_supernova
Your public key has been saved in .ssh/id_rsa_supernova.pub
The key fingerprint is:
SHA256:oSZygbCt1Z3FEfQkofm/zvnGulTMf/rppAcPEu2eHcI sheldon@localhost.localdomain
The key's randomart image is:
+---[RSA 4096]----+
|.      oB+.      |
| + o . =.+       |
|. + o = . . .    |
| o   . o . + .   |
|. . o o S   B    |
|   o o   . o E . |
|          o.o O.o|
|         o oooo*.|
|         .B=..++.|
+----[SHA256]-----+

Tip

ssh-keygen : ssh-key generation and management
-t type : type of key to create
-f filename : filename of the key file

5. Copy ssh-key to supernova serever (server2)

[sheldon@localhost ~]$ ssh-copy-id -i .ssh/id_rsa_supernova testing@supernova.caltech.edu
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: ".ssh/id_rsa_supernova.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
testing@supernova.caltech.edu's password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'testing@supernova.caltech.edu'"
and check to make sure that only the key(s) you wanted were added.

Tip

ssh-copy-id : install your public key in a remote machine’s authorized_keys
-i identity_file : which key to copy (if not default)

6. Putting the file to supernova serever (server2)

[sheldon@localhost ~]$ rsync -aurv new_simulations.tgz testing@supernova.caltech.edu:~
receiving incremental file list
super_nova_simulations.tgz

sent 43 bytes  received 3,239,389 bytes  6,478,864.00 bytes/sec
total size is 3,238,495  speedup is 1.00

7. Connect to supernova serever (server2) and extract the file (archive)

[sheldon@localhost ~]$ ssh testing@supernova.caltech.edu
[testing@supernova:~]$
[testing@supernova:~]$ tar xvzf super_nova_simulations.tgz
super_nova_simulations/
super_nova_simulations/IEG0ctikNK.dat
super_nova_simulations/xUPb0ybDie.dat
super_nova_simulations/runme.py
super_nova_simulations/create_files.sh
super_nova_simulations/uYLuadok35.py
super_nova_simulations/1jnLpRUfia.dat
super_nova_simulations/qtTVBJ8iWX.bin
super_nova_simulations/6TJvY2GdQP.dat
super_nova_simulations/BBTwbQR42v.py
super_nova_simulations/KGO5xjA2n1.dat
super_nova_simulations/samples.dat
tar: super_nova_simulations/samples.dat: Cannot write: No space left on device
super_nova_simulations/Cojc4YTcDk.bin
tar: super_nova_simulations/Cojc4YTcDk.bin: Cannot write: No space left on device
tar: Exiting with failure status due to previous errors

Tip

tar : secure copy (remote file copy program)
-xvzf : extract, verbose, ungzip, file

8. Check disk filesystems

[testing@supernova:~]$ df -h
Filesystem                        Size  Used Avail Use% Mounted on
/dev/mapper/vg-root              11.9G  6.7G  5.1G  56% /
/dev/mapper/vg-home               6.9G  6.8G     0 100% /home
none                              492K  4.0K  488K   1% /dev
udev                               12G     0   12G   0% /dev/tty
tmpfs                              12G     0   12G   0% /dev/shm
tmpfs                              12G  8.1M   12G   1% /run
tmpfs                             5.0M     0  5.0M   0% /run/lock
tmpfs                              12G     0   12G   0% /sys/fs/cgroup
tmpfs                             2.4G     0  2.4G   0% /run/user/0

Tip

df : report file system disk space usage
-h : human-readable

9. Check disk usage

[testing@supernova:~]$ cd super_nova_test/
[testing@supernova:super_nova_test]$ ls
BBTwbQR42v.py   create_files.sh  QsArAZrgWD.dat  qtTVBJ8iWX.bin  uYLuadok35.py
Cojc4YTcDk.bin  first_testing    Qst1yEn40y.dat  runme.py        ZZZ9871.bin
[testing@supernova:super_nova_test]$ ls -alh
total 216K
drwxrwxr-x 3 testing testing  36K Nov  3 11:58 .
drwxrwxr-x 4 testing testing 4.0K Nov  2 16:39 ..
-rw------- 1 testing testing  17K Oct 15 17:45 BBTwbQR42v.py
-rw------- 1 testing testing  26K Oct 15 17:45 Cojc4YTcDk.bin
-rwxr--r-- 1 testing testing  519 Oct 15 17:50 create_files.sh
drwxrwxr-x 2 testing testing  36K Nov  3 11:58 first_testing
-rw------- 1 testing testing  31K Nov  3 11:56 QsArAZrgWD.dat
-rw------- 1 testing testing  22K Nov  3 11:56 Qst1yEn40y.dat
-rw------- 1 testing testing  16K Oct 15 17:45 qtTVBJ8iWX.bin
-rw------- 1 testing testing 4.1K Oct 15 17:46 runme.py
-rw------- 1 testing testing 5.1K Oct 15 17:45 uYLuadok35.py
-rw-rw-r-- 1 testing testing    0 Oct 15 17:51 ZZZ9871.bin
[testing@supernova:super_nova_test]$ du
3164768	./first_testing
6310676	.
[testing@supernova:~]$ du -h --max-depht=1
du: unrecognized option '--max-depht=1'
Try 'du --help' for more information.

Tip

du : estimate file space usage
-h : human-readable
–max-depth=1 : -d 1

10. History & check disk usage

[testing@supernova:super_nova_test]$
[testing@supernova:super_nova_test]$ history | grep du
 4765  Apr 13 16:34:42 convert test_drawing*.svg dumb1.pdf
 4987  Apr 26 09:10:19 du -h --max-depth=1
 6326  Jun 30 11:55:51 du -h --max-depth=1 | grep G
 6328  Jun 30 11:56:07 du -h --max-depth=1 | grep G
 8265  Nov 03 11:26:01 du -h --max-depth=1
 8266  Nov 03 11:26:16 du
 8268  Nov 03 11:35:01 du -h -d 1
 8272  Nov 03 11:35:54 du
 8275  Nov 03 11:38:22 history | grep du
[testing@supernova:super_nova_test]$ !8268
du -h -d 1
3.1G	./first_testing
6.1G	.
[testing@supernova:super_nova_test]$ ls first_testing/
Q4b7l9lFUE.bin  RAtZQQpRTi.dat  SO97248WCP.dat  TuCWd8jmys.dat  VO4Uglp5kl.dat  WyCw9GGvf2.dat  YJOZ77dTRG.png
Q4qxv2JZuk.dat  RBHahCj8U7.dat  Sp7MTqyPLz.dat  UbLyqQvRlr.dat  VWHibcqSJB.dat  X0n3MJGQ9Q.dat  YL6IdUF6AI.dat
Q5IbbXT5oK.dat  RBRJG77AdM.dat  SP9EXwlHn1.dat  UBrWhXFCgk.dat  VWv5PBQ0Wi.dat  X0P0u3Od51.dat  Yo108KaCWw.dat
Q6bdTY9vmt.dat  RGZjLjArHR.dat  SR37K4NCpj.dat  UCf60tbDDM.dat  Vx4rIllgu7.dat  X1lePdTtvA.dat  Ytv7jK5TJU.dat
QAMs4rBmLo.dat  RJpZVuFhW4.dat  Sr6ymQie56.dat  UcPQC53B2c.dat  VylL0MyuWp.dat  X3bFFZky8B.dat  YXtWbZDxJN.dat
QdxSUbrvaV.dat  Rl6ZR6mkwl.dat  SrMbcaRyus.dat  Uh3JDInZxY.dat  VyQQt2w3fN.dat  XCF0Yo8wht.dat  YXUIzGHjFz.dat
QEHgM7cAaq.tmp  Rne3tqCHl6.dat  SrMPli52Bn.dat  Uhbzs4000k.dat  W0aW9KHVgQ.dat  XDnKHw7yP9.dat  Z2HjgjYMmV.dat
QHhPRsCc2h.dat  RNOFFqhpNW.dat  SSdrpglOOQ.dat  UhxYuOmqq4.dat  W1cP1RWRyf.dat  Xf1YUqVWUv.dat  Z9nwXH0wZb.dat
QifbSD8Gs3.dat  Rpne6gvJlS.dat  SV0sAkSWRM.dat  UJaJhRcPWF.dat  W40E0rqRgp.dat  XkYzvJwyqz.dat  Z9YeFB9Mrp.dat
QIGr7lS98m.dat  RqJQdw3ELK.dat  SwFwpcXxcs.dat  ULIbsE4WvB.dat  W7yltEdKmh.dat  XmSQxf97mb.dat  Zdc7t0ZJ5m.dat
QkH7TpnvXI.dat  RQL2OdyFnb.dat  SY7uJvMFvM.dat  UM2UvKnUKq.dat  W7Z5pKIGAk.dat  XqwCI0XbyK.dat  ZIgJFMTmuQ.dat
QKK62Unb3P.dat  RqOStzy7Eo.dat  SyhZ47VThS.dat  UmzyHEnb6Q.dat  WDtS5cKFbU.dat  XssEs5bAgJ.dat  ZITuFpN38O.dat
QNAg31kTbI.dat  RWI35lvl71.dat  SypFiT8MfY.dat  UnCCcrvLtL.bin  WGPCmw1AfJ.dat  XULISCyTG0.dat  ZknJyhc0lO.dat
QPzAquus4E.dat  RxnYj1G6XZ.dat  SzHs20M7Vf.dat  UqvIZ6otFC.dat  Wj8aSY8nW7.dat  Xz0buAj50h.dat  ZMmbVKYmdi.dat
QwsaBqvM72.dat  RZpmdnXV1j.dat  T66XaaDUb6.dat  VAOU21ZSlz.dat  WjVVEhA0uJ.dat  Xz7EolJbfo.dat  ZnlWEmKV7B.dat
QYqv2EMAkX.dat  S0qmLGAUTv.dat  T7WHUBYMTJ.dat  VFpQhT8AJG.dat  WLDrd3yBcS.dat  Xz7Na7bAOV.dat  ZQ8Rs1iVAt.dat
R0lCNTGqZA.dat  S28bUfXFOM.dat  TJ9oITJ4ck.dat  VGIp3JvA23.dat  WSXN0dDenT.dat  Y0G1cjVEEz.dat  Zqood2cgsH.dat
R3J6Jt51jz.dat  SCCMo0agrW.dat  TJbmXO9pgy.dat  VhpjWgXNGi.dat  WtYnppHcuw.dat  YCAPWhKxmh.dat  Zw8790WLQ7.dat
R4kAQvMEO8.dat  SdRCsxcZJ6.dat  TK3ghdhkkk.dat  VioCORx1MS.dat  WUUbdfeBcg.dat  YCdeI7SUzs.dat  Z999999999.txt
RabccJubgk.dat  SFfHGA9fsX.dat  TKa73NWb5s.dat  VIoG4vnNXY.dat  WwDlpAIsGm.dat  Yfp5EB9l3b.dat
RAL9q40aU3.dat  SmAgSPTgl8.dat  TLNfkXoZVV.dat  VmQkrxzWle.dat  WwZsiHRmdV.dat  YG9B7xPgNV.dat
RaqA98rFLg.dat  SMbo0mLEnE.dat  TmX2o4Izi8.dat  VO3NvLXAbn.dat  WXpeFbNP7G.dat  YiapNHGqAf.dat

Tip

history : show history of commands (see also : HISTSIZE, HISTFILESIZE)
| : pass output of left command to input right command
grep pattern : print lines matching a pattern

11. Find dat files

[testing@supernova:super_nova_test]$ find . -name "*.dat"
./Qst1yEn40y.dat
./QsArAZrgWD.dat
./first_testing/TuCWd8jmys.dat
./first_testing/UJaJhRcPWF.dat
./first_testing/UcPQC53B2c.dat
./first_testing/RxnYj1G6XZ.dat
./first_testing/YiapNHGqAf.dat
[skip]
./first_testing/Zw8790WLQ7.dat
./first_testing/RBRJG77AdM.dat
./first_testing/YCdeI7SUzs.dat
./first_testing/ZIgJFMTmuQ.dat
./first_testing/ZyfJVTAYBc.dat
./first_testing/WwDlpAIsGm.dat
./samples1.dat

Tip

find : search for files in a directory hierarchy
-name pattern : file name (the path with the leading directories removed) matches pattern

12. Find and remove

[testing@supernova:super_nova_test]$ find . -name "*.dat" -exec rm {} \;

Tip

find . -name “pattern” -exec command ; : find and execute
-exec command ; : execute command

13. Recheck file systems

[testing@supernova:super_nova_test]$ df -h
Filesystem                        Size  Used Avail Use% Mounted on
/dev/mapper/vg-root              11.9G  6.7G  5.1G  56% /
/dev/mapper/vg-home               6.9G  0.8G  6.1G  12% /home
none                              492K  4.0K  488K   1% /dev
udev                               12G     0   12G   0% /dev/tty
tmpfs                              12G     0   12G   0% /dev/shm
tmpfs                              12G  8.1M   12G   1% /run
tmpfs                             5.0M     0  5.0M   0% /run/lock
tmpfs                              12G     0   12G   0% /sys/fs/cgroup
tmpfs                             2.4G     0  2.4G   0% /run/user/0
[testing@supernova:super_nova_test]$ cd -

Tip

cd : change the working directory
- : cd “$OLDPWD” && pwd (changes to the previous working directory and then writes its name)

14. Untar

[testing@supernova:~]$ tar xvzf super_nova_simulations.tgz
super_nova_simulations/
super_nova_simulations/IEG0ctikNK.dat
super_nova_simulations/xUPb0ybDie.dat
super_nova_simulations/runme.py
super_nova_simulations/create_files.sh
super_nova_simulations/uYLuadok35.py
super_nova_simulations/1jnLpRUfia.dat
super_nova_simulations/qtTVBJ8iWX.bin
super_nova_simulations/6TJvY2GdQP.dat
super_nova_simulations/BBTwbQR42v.py
super_nova_simulations/KGO5xjA2n1.dat
super_nova_simulations/samples.dat
super_nova_simulations/Cojc4YTcDk.bin
[testing@supernova:~]$ cd super_nova_simulations

15. Run simulations (oups)

[testing@supernova:super_nova_simulations]$ ./runme.py
bash: ./runme.py: Permission denied
[testing@supernova:super_nova_simulations]$ ls -alh runme.py
-rw-rw-r-- 1 testing testing   35 Nov  2 16:42 runme.py
[testing@supernova:super_nova_simulations]$ chmod 760 runme.py
[testing@supernova:super_nova_simulations]$ ls -alh runme.py
-rwxrw-r-- 1 testing testing   35 Nov  2 16:43 runme.py
[testing@supernova:super_nova_simulations]$ ./runme.py
./runme.py: line 5: syntax error near unexpected token "Starting super nova simulations"
./runme.py: line 5: print("Starting super nova simulations")
[testing@supernova:super_nova_simulations]$ head -3 runme.py
import os, sys
import re
import numpy as np

Tip

./myprogram : execute myprogram in current directory (instead of searching in $PATH)
chmod : change file rights – the three values are for the user, then the unix group, then all users r = read | w = write | x = execute 760 : = rwx(7)rw-(6)—(0) (rwxrw—-)
head : output the first part of files
-3 : just 3 first lines

16. Where is python ?

[testing@supernova:super_nova_simulations]$ which python
/usr/bin/python
[testing@supernova:super_nova_simulations]$ python --version
Python 3.9.7
[testing@supernova:super_nova_simulations]$ python runme.py
Starting super nova simulations
Could not lock temporary file, aborting...

Tip

which programme : locate a command
python –version : show python version

17. Kill

[testing@supernova:super_nova_simulations]$ ps -aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  2.1 169340  5580 ?        Ss   Oct15   0:05 /sbin/init
root        47  0.0  3.1  29644  8252 ?        Ss   Oct15   0:02 /lib/systemd/systemd-journald
root        74  0.0  0.6  45080  1676 ?        Ss   Oct15   0:01 /usr/sbin/sssd -i --logger=files
root        75  0.0  0.1 156184   376 ?        Ssl  Oct15   0:00 /usr/sbin/rsyslogd -n -iNONE
message+    76  0.0  1.1  17244  2892 ?        Ss   Oct15   0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --
root       225  0.0  0.7  15848  1996 ?        Ss   Oct15   0:00 /usr/sbin/sshd -D
root       305  0.0  0.5  43516  1428 ?        Ss   Oct15   0:04 /usr/lib/postfix/sbin/master -w
postfix    308  0.0  0.3  54080   864 ?        S    Oct15   0:00 qmgr -l -t unix -u
root       309  0.0  1.5  48460  3980 ?        S    Oct15   0:15 /usr/lib/x86_64-linux-gnu/sssd/sssd_be --domain caltech.edu
root       310  0.0  2.0  62844  5452 ?        S    Oct15   0:12 /usr/lib/x86_64-linux-gnu/sssd/sssd_nss --uid 0 --gid 0 --lo
root       311  0.0  3.3  42204  8864 ?        S    Oct15   0:11 /usr/lib/x86_64-linux-gnu/sssd/sssd_pam --uid 0 --gid 0 --lo
root       317  0.0  1.8  28044  4896 ?        Ss   Oct15   0:02 /lib/systemd/systemd-logind
root       318  0.0  0.3   5508   916 ?        Ss   Oct15   0:02 /usr/sbin/cron -f
root       321  0.0  0.0   2416   104 pts/0    Ss+  Oct15   0:00 /sbin/agetty -o -p -- \u --noclear --keep-baud console 11520
root       322  0.0  0.0   2416   112 pts/0    Ss+  Oct15   0:00 /sbin/agetty -o -p -- \u --noclear --keep-baud tty1 115200,3
root       323  0.0  0.0   2416    52 pts/1    Ss+  Oct15   0:00 /sbin/agetty -o -p -- \u --noclear --keep-baud tty2 115200,3
postfix   6976  0.0  3.0  54028  8048 ?        S    11:20   0:00 pickup -l -t unix -u -c
root      6980  0.0  3.2  35524  8516 ?        Ss   12:15   0:00 sshd: testing [priv]
testing  6983  0.0  3.2  21024  8404 ?        Ss   12:15   0:00 /lib/systemd/systemd --user
testing  6984  0.0  1.0 189200  2736 ?        S    12:15   0:00 (sd-pam)
testing  6992  0.0  1.9  35524  5052 ?        S    12:15   0:00 sshd: testing@pts/2
testing  6993  0.0  1.6  13372  4416 pts/2    Ss   12:15   0:00 -bash
testing  6996  0.0  1.1   3780  2960 pts/2    S+   12:15   0:00 python ./runme.py
root      7008  0.0  3.2  35524  8548 ?        Ss   12:15   0:00 sshd: testing [priv]
testing  7018  0.0  1.9  35524  5060 ?        S    12:15   0:00 sshd: testing@pts/3
testing  7019  0.0  1.6  13372  4332 pts/3    Ss   12:15   0:00 -bash
testing  7204  0.0  0.2   2292   744 pts/2    S+   12:18   0:00 sleep 1
testing  7205  0.0  1.1  16180  2976 pts/3    R+   12:18   0:00 ps -aux
[testing@supernova:super_nova_simulations]$ kill 6996
[testing@supernova:super_nova_simulations]$ kill -s 9 6996

Tip

ps : show current processes
-aux : all processes, effective user ID, Lift the BSD-style “must have a tty” restriction
kill : send a signal to a process (9 = SIGKILL, 15 = SIGTERM (default), 2 = SIGINT (Ctrl-C))

18. Relaunch

[testing@supernova:super_nova_simulations]$ python runme.py
Starting super nova simulations
Simulation is successfully started...