Real-life scenario#
Let’s imagine you receive the following email and decide to take on this task.
From: eric.gablehauser@caltech.edu
To: sheldon@caltech.edu
Date: Mon, 01 Nov 2007 06:10:30 -0000Hello, Sheldon
I have prepared the new version of the simulations for the Super Nova study,
but I am going to the Super Nova South congress in Atlanta this week, so could
you run the new simulation to get new results for next week?
I put the new version on my server gbl-srv, you have an account on it.
To launch the simulation use the new server supernova.caltech.edu, you connect
with the login testing.
By the way, it’s important that you put your ssh key on it (I got a mail from
IT-crowd that they will forbid normal password access from next week).
To launch simulation as usual : runme.pyBest wishes,
Eric Gablehauser
Head of the Physics Department
California Institute of TechnologyP.S.: I don’t know if there is enough space on supernova server, in case you could
delete the dat files in the super_nova_test folder, but be careful not to delete
the *.bin files (or others)P.P.S.: Pay attention that there is no other simulation running: you remember lock problem
that we never solved.P.P.P.S.: I used some new features of Python 3.9 so it should be run with at least 3.9 python.
I hope we have a good version on supernova server. If it’s not… hum, you should ask somebody
of IT-crowd how to install it.
1. Connection to gbl-srv (server1) to see the file#
[sheldon@localhost ~]$ ssh sheldon@gbl-srv.caltech.edu
(sheldon@glb-srv:/home/sheldon)$> ls /group/snova/simulations/
super_nova_simulations.tgz
(sheldon@glb-srv:/home/sheldon)$> exit
[sheldon@localhost ~]$
Tip
ssh : remote login program
2. Getting the file from gbl-srv (server1) (rsync)#
[sheldon@localhost ~]$ rsync -aurv sheldon@gbl-srv.caltech.edu:/group/snova/simulations/super_nova_simulations.tgz .
bash: rsync: command not found
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(228) [sender=3.2.3]
Tip
rsync : file-copying tool
-aurv : archive, update, recursive, verbose
archive : “preserve almost everything” ( = -rlptgoD)
Hmmm…?
3. Getting the file from gbl-srv (server1) (scp)#
[sheldon@localhost ~]$ scp -r sheldon@gbl-srv.caltech.edu:/group/snova/simulations/super_nova_simulations.tgz .
super_nova_simulations.tgz 100% 3163KB 11.1MB/s 00:00
Tip
scp : secure copy (remote file copy program)
-r : Recursively copy entire directories
4. Create ssh-key on Sheldon’s computer#
[sheldon@localhost ~]$ ssh-keygen -t rsa -b 4096 -f .ssh/id_rsa_supernova
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in .ssh/id_rsa_supernova
Your public key has been saved in .ssh/id_rsa_supernova.pub
The key fingerprint is:
SHA256:oSZygbCt1Z3FEfQkofm/zvnGulTMf/rppAcPEu2eHcI sheldon@localhost.localdomain
The key's random art image is:
+---[RSA 4096]----+
|. oB+. |
| + o . =.+ |
|. + o = . . . |
| o . o . + . |
|. . o o S B |
| o o . o E . |
| o.o O.o|
| o oooo*.|
| .B=..++.|
+----[SHA256]-----+
Tip
ssh-keygen : ssh-key generation and management
-t type : type of key to create
-f filename : filename of the key file
5. Copy ssh-key to supernova server (server2)#
[sheldon@localhost ~]$ ssh-copy-id -i .ssh/id_rsa_supernova testing@supernova.caltech.edu
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: ".ssh/id_rsa_supernova.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
testing@supernova.caltech.edu's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'testing@supernova.caltech.edu'"
and check to make sure that only the key(s) you wanted were added.
Tip
ssh-copy-id : install your public key in a remote machine’s authorized_keys
-i identity_file : which key to copy (if not default)
6. Putting the file to supernova server (server2)#
[sheldon@localhost ~]$ rsync -aurv new_simulations.tgz testing@supernova.caltech.edu:~
receiving incremental file list
super_nova_simulations.tgz
sent 43 bytes received 3,239,389 bytes 6,478,864.00 bytes/sec
total size is 3,238,495 speedup is 1.00
7. Connect to supernova server (server2) and extract the file (archive)#
[sheldon@localhost ~]$ ssh testing@supernova.caltech.edu
[testing@supernova:~]$
[testing@supernova:~]$ tar xvzf super_nova_simulations.tgz
super_nova_simulations/
super_nova_simulations/IEG0ctikNK.dat
super_nova_simulations/xUPb0ybDie.dat
super_nova_simulations/runme.py
super_nova_simulations/create_files.sh
super_nova_simulations/uYLuadok35.py
super_nova_simulations/1jnLpRUfia.dat
super_nova_simulations/qtTVBJ8iWX.bin
super_nova_simulations/6TJvY2GdQP.dat
super_nova_simulations/BBTwbQR42v.py
super_nova_simulations/KGO5xjA2n1.dat
super_nova_simulations/samples.dat
tar: super_nova_simulations/samples.dat: Cannot write: No space left on device
super_nova_simulations/Cojc4YTcDk.bin
tar: super_nova_simulations/Cojc4YTcDk.bin: Cannot write: No space left on device
tar: Exiting with failure status due to previous errors
Tip
tar : secure copy (remote file copy program)
-xvzf : extract, verbose, ungzip, file
8. Check disk filesystem#
[testing@supernova:~]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg-root 11.9G 6.7G 5.1G 56% /
/dev/mapper/vg-home 6.9G 6.8G 0 100% /home
none 492K 4.0K 488K 1% /dev
udev 12G 0 12G 0% /dev/tty
tmpfs 12G 0 12G 0% /dev/shm
tmpfs 12G 8.1M 12G 1% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 12G 0 12G 0% /sys/fs/cgroup
tmpfs 2.4G 0 2.4G 0% /run/user/0
Tip
df : report file system disk space usage
-h : human-readable
9. Check disk usage#
[testing@supernova:~]$ cd super_nova_test/
[testing@supernova:super_nova_test]$ ls
BBTwbQR42v.py create_files.sh QsArAZrgWD.dat qtTVBJ8iWX.bin uYLuadok35.py
Cojc4YTcDk.bin first_testing Qst1yEn40y.dat runme.py ZZZ9871.bin
[testing@supernova:super_nova_test]$ ls -alh
total 216K
drwxrwxr-x 3 testing testing 36K Nov 3 11:58 .
drwxrwxr-x 4 testing testing 4.0K Nov 2 16:39 ..
-rw------- 1 testing testing 17K Oct 15 17:45 BBTwbQR42v.py
-rw------- 1 testing testing 26K Oct 15 17:45 Cojc4YTcDk.bin
-rwxr--r-- 1 testing testing 519 Oct 15 17:50 create_files.sh
drwxrwxr-x 2 testing testing 36K Nov 3 11:58 first_testing
-rw------- 1 testing testing 31K Nov 3 11:56 QsArAZrgWD.dat
-rw------- 1 testing testing 22K Nov 3 11:56 Qst1yEn40y.dat
-rw------- 1 testing testing 16K Oct 15 17:45 qtTVBJ8iWX.bin
-rw------- 1 testing testing 4.1K Oct 15 17:46 runme.py
-rw------- 1 testing testing 5.1K Oct 15 17:45 uYLuadok35.py
-rw-rw-r-- 1 testing testing 0 Oct 15 17:51 ZZZ9871.bin
[testing@supernova:super_nova_test]$ du
3164768 ./first_testing
6310676 .
[testing@supernova:~]$ du -h --max-depht=1
du: unrecognized option '--max-depht=1'
Try 'du --help' for more information.
Tip
du : estimate file space usage
-h : human-readable
–max-depth=1 : -d 1
10. History & check disk usage#
[testing@supernova:super_nova_test]$ history | grep du
4765 Apr 13 16:34:42 convert test_drawing*.svg dumb1.pdf
4987 Apr 26 09:10:19 du -h --max-depth=1
6326 Jun 30 11:55:51 du -h --max-depth=1 | grep G
6328 Jun 30 11:56:07 du -h --max-depth=1 | grep G
8265 Nov 03 11:26:01 du -h --max-depth=1
8266 Nov 03 11:26:16 du
8268 Nov 03 11:35:01 du -h -d 1
8272 Nov 03 11:35:54 du
8275 Nov 03 11:38:22 history | grep du
[testing@supernova:super_nova_test]$ !8268
du -hd 1
3.1G ./first_testing
6.1G .
[testing@supernova:super_nova_test]$ ls first_testing/
Q4b7l9lFUE.bin RAtZQQpRTi.dat SO97248WCP.dat TuCWd8jmys.dat VO4Uglp5kl.dat WyCw9GGvf2.dat YJOZ77dTRG.png
Q4qxv2JZuk.dat RBHahCj8U7.dat Sp7MTqyPLz.dat UbLyqQvRlr.dat VWHibcqSJB.dat X0n3MJGQ9Q.dat YL6IdUF6AI.dat
Q5IbbXT5oK.dat RBRJG77AdM.dat SP9EXwlHn1.dat UBrWhXFCgk.dat VWv5PBQ0Wi.dat X0P0u3Od51.dat Yo108KaCWw.dat
Q6bdTY9vmt.dat RGZjLjArHR.dat SR37K4NCpj.dat UCf60tbDDM.dat Vx4rIllgu7.dat X1lePdTtvA.dat Ytv7jK5TJU.dat
QAMs4rBmLo.dat RJpZVuFhW4.dat Sr6ymQie56.dat UcPQC53B2c.dat VylL0MyuWp.dat X3bFFZky8B.dat YXtWbZDxJN.dat
QdxSUbrvaV.dat Rl6ZR6mkwl.dat SrMbcaRyus.dat Uh3JDInZxY.dat VyQQt2w3fN.dat XCF0Yo8wht.dat YXUIzGHjFz.dat
QEHgM7cAaq.tmp Rne3tqCHl6.dat SrMPli52Bn.dat Uhbzs4000k.dat W0aW9KHVgQ.dat XDnKHw7yP9.dat Z2HjgjYMmV.dat
QHhPRsCc2h.dat RNOFFqhpNW.dat SSdrpglOOQ.dat UhxYuOmqq4.dat W1cP1RWRyf.dat Xf1YUqVWUv.dat Z9nwXH0wZb.dat
QifbSD8Gs3.dat Rpne6gvJlS.dat SV0sAkSWRM.dat UJaJhRcPWF.dat W40E0rqRgp.dat XkYzvJwyqz.dat Z9YeFB9Mrp.dat
QIGr7lS98m.dat RqJQdw3ELK.dat SwFwpcXxcs.dat ULIbsE4WvB.dat W7yltEdKmh.dat XmSQxf97mb.dat Zdc7t0ZJ5m.dat
QkH7TpnvXI.dat RQL2OdyFnb.dat SY7uJvMFvM.dat UM2UvKnUKq.dat W7Z5pKIGAk.dat XqwCI0XbyK.dat ZIgJFMTmuQ.dat
QKK62Unb3P.dat RqOStzy7Eo.dat SyhZ47VThS.dat UmzyHEnb6Q.dat WDtS5cKFbU.dat XssEs5bAgJ.dat ZITuFpN38O.dat
QNAg31kTbI.dat RWI35lvl71.dat SypFiT8MfY.dat UnCCcrvLtL.bin WGPCmw1AfJ.dat XULISCyTG0.dat ZknJyhc0lO.dat
QPzAquus4E.dat RxnYj1G6XZ.dat SzHs20M7Vf.dat UqvIZ6otFC.dat Wj8aSY8nW7.dat Xz0buAj50h.dat ZMmbVKYmdi.dat
QwsaBqvM72.dat RZpmdnXV1j.dat T66XaaDUb6.dat VAOU21ZSlz.dat WjVVEhA0uJ.dat Xz7EolJbfo.dat ZnlWEmKV7B.dat
QYqv2EMAkX.dat S0qmLGAUTv.dat T7WHUBYMTJ.dat VFpQhT8AJG.dat WLDrd3yBcS.dat Xz7Na7bAOV.dat ZQ8Rs1iVAt.dat
R0lCNTGqZA.dat S28bUfXFOM.dat TJ9oITJ4ck.dat VGIp3JvA23.dat WSXN0dDenT.dat Y0G1cjVEEz.dat Zqood2cgsH.dat
R3J6Jt51jz.dat SCCMo0agrW.dat TJbmXO9pgy.dat VhpjWgXNGi.dat WtYnppHcuw.dat YCAPWhKxmh.dat Zw8790WLQ7.dat
R4kAQvMEO8.dat SdRCsxcZJ6.dat TK3ghdhkkk.dat VioCORx1MS.dat WUUbdfeBcg.dat YCdeI7SUzs.dat Z999999999.txt
RabccJubgk.dat SFfHGA9fsX.dat TKa73NWb5s.dat VIoG4vnNXY.dat WwDlpAIsGm.dat Yfp5EB9l3b.dat
RAL9q40aU3.dat SmAgSPTgl8.dat TLNfkXoZVV.dat VmQkrxzWle.dat WwZsiHRmdV.dat YG9B7xPgNV.dat
RaqA98rFLg.dat SMbo0mLEnE.dat TmX2o4Izi8.dat VO3NvLXAbn.dat WXpeFbNP7G.dat YiapNHGqAf.dat
Tip
history : show history of commands (see also : HISTSIZE, HISTFILESIZE)
| : pass output of left command to input right command
grep pattern : print lines matching a pattern
11. Find dat files#
[testing@supernova:super_nova_test]$ find . -name "*.dat"
./Qst1yEn40y.dat
./QsArAZrgWD.dat
./first_testing/TuCWd8jmys.dat
./first_testing/UJaJhRcPWF.dat
./first_testing/UcPQC53B2c.dat
./first_testing/RxnYj1G6XZ.dat
./first_testing/YiapNHGqAf.dat
[skip]
./first_testing/Zw8790WLQ7.dat
./first_testing/RBRJG77AdM.dat
./first_testing/YCdeI7SUzs.dat
./first_testing/ZIgJFMTmuQ.dat
./first_testing/ZyfJVTAYBc.dat
./first_testing/WwDlpAIsGm.dat
./samples1.dat
Tip
find : search for files in a directory hierarchy
-name pattern : file name (the path with the leading directories removed) matches pattern
12. Find and remove#
[testing@supernova:super_nova_test]$ find . -name "*.dat" -exec rm {} \;
Tip
find . -name “pattern” -exec command ; : find and execute
-exec command ; : execute command
13. Recheck file systems#
[testing@supernova:super_nova_test]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg-root 11.9G 6.7G 5.1G 56% /
/dev/mapper/vg-home 6.9G 0.8G 6.1G 12% /home
none 492K 4.0K 488K 1% /dev
udev 12G 0 12G 0% /dev/tty
tmpfs 12G 0 12G 0% /dev/shm
tmpfs 12G 8.1M 12G 1% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 12G 0 12G 0% /sys/fs/cgroup
tmpfs 2.4G 0 2.4G 0% /run/user/0
[testing@supernova:super_nova_test]$ cd -
Tip
cd : change the working directory
- : cd “$OLDPWD” && pwd (changes to the previous working directory and then writes its name)
14. Untar#
[testing@supernova:~]$ tar xvzf super_nova_simulations.tgz
super_nova_simulations/
super_nova_simulations/IEG0ctikNK.dat
super_nova_simulations/xUPb0ybDie.dat
super_nova_simulations/runme.py
super_nova_simulations/create_files.sh
super_nova_simulations/uYLuadok35.py
super_nova_simulations/1jnLpRUfia.dat
super_nova_simulations/qtTVBJ8iWX.bin
super_nova_simulations/6TJvY2GdQP.dat
super_nova_simulations/BBTwbQR42v.py
super_nova_simulations/KGO5xjA2n1.dat
super_nova_simulations/samples.dat
super_nova_simulations/Cojc4YTcDk.bin
[testing@supernova:~]$ cd super_nova_simulations
15. Run simulations (oups)#
[testing@supernova:super_nova_simulations]$ ./runme.py
bash: ./runme.py: Permission denied
[testing@supernova:super_nova_simulations]$ ls -alh runme.py
-rw-rw-r-- 1 testing testing 35 Nov 2 16:42 runme.py
[testing@supernova:super_nova_simulations]$ chmod 760 runme.py
[testing@supernova:super_nova_simulations]$ ls -alh runme.py
-rwxrw-r-- 1 testing testing 35 Nov 2 16:43 runme.py
[testing@supernova:super_nova_simulations]$ ./runme.py
./runme.py: line 5: syntax error near unexpected token "Starting super nova simulations"
./runme.py: line 5: print("Starting super nova simulations")
[testing@supernova:super_nova_simulations]$ head -3 runme.py
import os, sys
import re
import numpy as np
Tip
./myprogram : execute myprogram in current directory (instead of searching in $PATH)
chmod : change file rights – the three values are for the user, then the unix group, then all users
r = read | w = write | x = execute
760 : = rwx(7)rw-(6)—(0) (rwxrw—-)
head : output the first part of files
-3 : just 3 first lines
16. Where is python ?#
[testing@supernova:super_nova_simulations]$ which python
/usr/bin/python
[testing@supernova:super_nova_simulations]$ python --version
Python 3.9.7
[testing@supernova:super_nova_simulations]$ python runme.py
Starting super nova simulations
Could not lock temporary file, aborting...
Tip
which program : locate a command
python –version : show python version
17. Kill#
[testing@supernova:super_nova_simulations]$ ps -aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 2.1 169340 5580 ? Ss Oct15 0:05 /sbin/init
root 47 0.0 3.1 29644 8252 ? Ss Oct15 0:02 /lib/systemd/systemd-journald
root 74 0.0 0.6 45080 1676 ? Ss Oct15 0:01 /usr/sbin/sssd -i --logger=files
root 75 0.0 0.1 156184 376 ? Ssl Oct15 0:00 /usr/sbin/rsyslogd -n -iNONE
message+ 76 0.0 1.1 17244 2892 ? Ss Oct15 0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --
root 225 0.0 0.7 15848 1996 ? Ss Oct15 0:00 /usr/sbin/sshd -D
root 305 0.0 0.5 43516 1428 ? Ss Oct15 0:04 /usr/lib/postfix/sbin/master -w
postfix 308 0.0 0.3 54080 864 ? S Oct15 0:00 qmgr -l -t unix -u
root 309 0.0 1.5 48460 3980 ? S Oct15 0:15 /usr/lib/x86_64-linux-gnu/sssd/sssd_be --domain caltech.edu
root 310 0.0 2.0 62844 5452 ? S Oct15 0:12 /usr/lib/x86_64-linux-gnu/sssd/sssd_nss --uid 0 --gid 0 --lo
root 311 0.0 3.3 42204 8864 ? S Oct15 0:11 /usr/lib/x86_64-linux-gnu/sssd/sssd_pam --uid 0 --gid 0 --lo
root 317 0.0 1.8 28044 4896 ? Ss Oct15 0:02 /lib/systemd/systemd-logind
root 318 0.0 0.3 5508 916 ? Ss Oct15 0:02 /usr/sbin/cron -f
root 321 0.0 0.0 2416 104 pts/0 Ss+ Oct15 0:00 /sbin/agetty -o -p -- \u --noclear --keep-baud console 11520
root 322 0.0 0.0 2416 112 pts/0 Ss+ Oct15 0:00 /sbin/agetty -o -p -- \u --noclear --keep-baud tty1 115200,3
root 323 0.0 0.0 2416 52 pts/1 Ss+ Oct15 0:00 /sbin/agetty -o -p -- \u --noclear --keep-baud tty2 115200,3
postfix 6976 0.0 3.0 54028 8048 ? S 11:20 0:00 pickup -l -t unix -u -c
root 6980 0.0 3.2 35524 8516 ? Ss 12:15 0:00 sshd: testing [priv]
testing 6983 0.0 3.2 21024 8404 ? Ss 12:15 0:00 /lib/systemd/systemd --user
testing 6984 0.0 1.0 189200 2736 ? S 12:15 0:00 (sd-pam)
testing 6992 0.0 1.9 35524 5052 ? S 12:15 0:00 sshd: testing@pts/2
testing 6993 0.0 1.6 13372 4416 pts/2 Ss 12:15 0:00 -bash
testing 6996 0.0 1.1 3780 2960 pts/2 S+ 12:15 0:00 python ./runme.py
root 7008 0.0 3.2 35524 8548 ? Ss 12:15 0:00 sshd: testing [priv]
testing 7018 0.0 1.9 35524 5060 ? S 12:15 0:00 sshd: testing@pts/3
testing 7019 0.0 1.6 13372 4332 pts/3 Ss 12:15 0:00 -bash
testing 7204 0.0 0.2 2292 744 pts/2 S+ 12:18 0:00 sleep 1
testing 7205 0.0 1.1 16180 2976 pts/3 R+ 12:18 0:00 ps -aux
[testing@supernova:super_nova_simulations]$ kill 6996
[testing@supernova:super_nova_simulations]$ kill -s 9 6996
Tip
ps : show current processes
-aux : all processes, effective user ID, Lift the BSD-style “must have a tty” restriction
kill : send a signal to a process (9 = SIGKILL, 15 = SIGTERM (default), 2 = SIGINT (Ctrl-C))
18. Relaunch#
[testing@supernova:super_nova_simulations]$ python runme.py
Starting super nova simulations
Simulation is successfully started...