class: center, middle ## Imagine... a small scenario. ![Sheldon](images/sheldon.gif) --- class: mail-page >From: eric.gablehauser@caltech.edu >To: sheldon@caltech.edu >Date: Mon, 01 Nov 2007 06:10:30 -0000 Hello, Sheldon I have prepared the new version of the simulations for the Super Nova study, but I am going to the Super Nova South congress in Atlanta this week, so could you run the new simulation to get new results for next week? I put the new version on my server gbl-srv, you have an account on it. To launch the simulation use the new server supernova.caltech.edu, you connect with the login testing. By the way, it's important that you put your ssh key on it (I got a mail from IT-crowd that they will forbid normal password acces from next week). To launch simulation as usual : runme.py >Best wishes, >Eric Gablehauser >Head of the Physics Department >California Institute of Technology P.S.: I don't know if there is enough space on supernova server, in case you could delete the dat files in the super_nova_test folder, but be careful not to delete the *.bin files (or others) P.P.S.: Pay attention that there is no other simulation running: you remember lock problem that we never solved. P.P.P.S.: I used some new features of Python 3.9 so it should be run with at least 3.9 python. I hope we have a good version on supernova server. If it's not... hum, you should ask somebody of IT-crowd how to install it. --- class: center, middle ![Morning](images/good-morning.gif) --- ### 1. Connection to gbl-srv (server1) to see the file .remark-code.hljs[``` [sheldon@localhost ~]$ ssh sheldon@gbl-srv.caltech.edu (sheldon@glb-srv:/home/sheldon)$> ls /group/snova/simulations/super_nova_simulations.tgz /group/snova/simulations/super_nova_simulations.tgz (sheldon@glb-srv:/home/sheldon)$> exit [sheldon@localhost ~]$ ```] 💡 **NOTE** > **ssh** : remote login program --- ### 2. Getting the file from gbl-srv (server1) (rsync) .remark-code.hljs[``` [sheldon@localhost ~]$ rsync -aurv sheldon@gbl-srv.caltech.edu:/group/snova/simulations/super_nova_simulations.tgz . bash: rsync: command not found rsync: connection unexpectedly closed (0 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(228) [sender=3.2.3] ```] 💡 **NOTE** > **rsync** : file-copying tool > *-aurv* : archive, update, recursive, verbose > *archive* : "preserve almost everything" ( = -rlptgoD) Hmmm...?
--- ### 3. Getting the file from gbl-srv (server1) (scp) .remark-code.hljs[``` [sheldon@localhost ~]$ scp -r sheldon@gbl-srv.caltech.edu:/group/snova/simulations/super_nova_simulations.tgz . super_nova_simulations.tgz 100% 3163KB 11.1MB/s 00:00 ```] 💡 **NOTE** > **scp** : secure copy (remote file copy program) > *-r* : Recursively copy entire directories --- ### 4. Create ssh-key on Sheldon's computer .remark-code.hljs[``` [sheldon@localhost ~]$ ssh-keygen -t rsa -f .ssh/id_rsa_supernova Generating public/private rsa key pair. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in .ssh/id_rsa_supernova Your public key has been saved in .ssh/id_rsa_supernova.pub The key fingerprint is: SHA256:oSZygbCt1Z3FEfQkofm/zvnGulTMf/rppAcPEu2eHcI sheldon@localhost.localdomain The key's randomart image is: +---[RSA 4096]----+ |. oB+. | | + o . =.+ | |. + o = . . . | | o . o . + . | |. . o o S B | | o o . o E . | | o.o O.o| | o oooo*.| | .B=..++.| +----[SHA256]-----+ ```] 💡 **NOTE** > **ssh-keygen** : ssh-key generation and management > *-t
type
* : type of key to create > *-f
filename
* : filename of the key file --- ### 5. Copy ssh-key to supernova serever (server2) .remark-code.hljs[``` [sheldon@localhost ~]$ ssh-copy-id -i .ssh/id_rsa_supernova testing@supernova.caltech.edu /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: ".ssh/id_rsa_supernova.pub" /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys testing@supernova.caltech.edu's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'testing@supernova.caltech.edu'" and check to make sure that only the key(s) you wanted were added. ```] 💡 **NOTE** > **ssh-copy-id** : install your public key in a remote machine's authorized_keys > *-i
identity_file
* : which key to copy (if not default) ### 6. Putting the file to supernova serever (server2) .remark-code.hljs[``` [sheldon@localhost ~]$ rsync -aurv new_simulations.tgz testing@supernova.caltech.edu:~ receiving incremental file list super_nova_simulations.tgz sent 43 bytes received 3,239,389 bytes 6,478,864.00 bytes/sec total size is 3,238,495 speedup is 1.00 ```] --- ### 7. Connect to supernova serever (server2) and extract the file (archive) .remark-code.hljs[``` [sheldon@localhost ~]$ ssh testing@supernova.caltech.edu [testing@supernova:~]$ [testing@supernova:~]$ tar xvzf super_nova_simulations.tgz super_nova_simulations/ super_nova_simulations/IEG0ctikNK.dat super_nova_simulations/xUPb0ybDie.dat super_nova_simulations/runme.py super_nova_simulations/create_files.sh super_nova_simulations/uYLuadok35.py super_nova_simulations/qtTVBJ8iWX.bin super_nova_simulations/6TJvY2GdQP.dat super_nova_simulations/BBTwbQR42v.py super_nova_simulations/KGO5xjA2n1.dat super_nova_simulations/samples.dat tar: super_nova_simulations/samples.dat: Cannot write: No space left on device super_nova_simulations/Cojc4YTcDk.bin tar: super_nova_simulations/Cojc4YTcDk.bin: Cannot write: No space left on device tar: Exiting with failure status due to previous errors ```] 💡 **NOTE** > **tar** : Archiving utility. Often combined with a compression method, such as gzip or bzip2. > *-xvzf* : extract, verbose, ungzip, file
--- ### 8. Check disk filesystems .remark-code.hljs[``` [testing@supernova:~]$ df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg-root 11.9G 6.7G 5.1G 56% / /dev/mapper/vg-home 6.9G 6.8G 0 100% /home none 492K 4.0K 488K 1% /dev udev 12G 0 12G 0% /dev/tty tmpfs 12G 0 12G 0% /dev/shm tmpfs 12G 8.1M 12G 1% /run tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 12G 0 12G 0% /sys/fs/cgroup tmpfs 2.4G 0 2.4G 0% /run/user/0 ```] 💡 **NOTE** > **df** : report file system disk space usage > *-h* : human-readable --- ### 9. Check disk usage .remark-code.hljs[``` [testing@supernova:~]$ cd super_nova_test/ [testing@supernova:super_nova_test]$ ls BBTwbQR42v.py create_files.sh QsArAZrgWD.dat qtTVBJ8iWX.bin uYLuadok35.py Cojc4YTcDk.bin first_testing Qst1yEn40y.dat runme.py ZZZ9871.bin [testing@supernova:super_nova_test]$ ls -alh total 216K drwxrwxr-x 3 testing testing 36K Nov 3 11:58 . drwxrwxr-x 4 testing testing 4.0K Nov 2 16:39 .. -rw------- 1 testing testing 17K Oct 15 17:45 BBTwbQR42v.py -rw------- 1 testing testing 26K Oct 15 17:45 Cojc4YTcDk.bin -rwxr--r-- 1 testing testing 519 Oct 15 17:50 create_files.sh drwxrwxr-x 2 testing testing 36K Nov 3 11:58 first_testing -rw------- 1 testing testing 31K Nov 3 11:56 QsArAZrgWD.dat -rw------- 1 testing testing 22K Nov 3 11:56 Qst1yEn40y.dat -rw------- 1 testing testing 16K Oct 15 17:45 qtTVBJ8iWX.bin -rw------- 1 testing testing 4.1K Oct 15 17:46 runme.py -rw------- 1 testing testing 5.1K Oct 15 17:45 uYLuadok35.py -rw-rw-r-- 1 testing testing 0 Oct 15 17:51 ZZZ9871.bin [testing@supernova:super_nova_test]$ du 6164768 ./first_testing 6310676 . [testing@supernova:~]$ du -h --max-depht=1 du: unrecognized option '--max-depht=1' Try 'du --help' for more information. ```] 💡 **NOTE** > **du** : estimate file space usage > *-h* : human-readable > *--max-depth=1* : *-d 1* --- ### 10. History & check disk usage .remark-code.hljs[``` [testing@supernova:super_nova_test]$ history | grep du 4765 Apr 13 16:34:42 convert test_drawing*.svg dumb1.pdf 4987 Apr 26 09:10:19 du -h --max-depth=1 6326 Jun 30 11:55:51 du -h --max-depth=1 | grep G 6328 Jun 30 11:56:07 du -h --max-depth=1 | grep G 8265 Nov 03 11:26:01 du -h --max-depth=1 8266 Nov 03 11:26:16 du 8268 Nov 03 11:35:01 du -h -d 1 8272 Nov 03 11:35:54 du 8275 Nov 03 11:38:22 history | grep du [testing@supernova:super_nova_test]$ !8268 du -h -d 1 6.1G ./first_testing 6.1G . [testing@supernova:super_nova_test]$ ls first_testing/ Q4b7l9lFUE.bin RAtZQQpRTi.dat SO97248WCP.dat TuCWd8jmys.dat VO4Uglp5kl.dat WyCw9GGvf2.dat YJOZ77dTRG.png Q4qxv2JZuk.dat RBHahCj8U7.dat Sp7MTqyPLz.dat UbLyqQvRlr.dat VWHibcqSJB.dat X0n3MJGQ9Q.dat YL6IdUF6AI.dat Q5IbbXT5oK.dat RBRJG77AdM.dat SP9EXwlHn1.dat UBrWhXFCgk.dat VWv5PBQ0Wi.dat X0P0u3Od51.dat Yo108KaCWw.dat Q6bdTY9vmt.dat RGZjLjArHR.dat SR37K4NCpj.dat UCf60tbDDM.dat Vx4rIllgu7.dat X1lePdTtvA.dat Ytv7jK5TJU.dat QAMs4rBmLo.dat RJpZVuFhW4.dat Sr6ymQie56.dat UcPQC53B2c.dat VylL0MyuWp.dat X3bFFZky8B.dat YXtWbZDxJN.dat QdxSUbrvaV.dat Rl6ZR6mkwl.dat SrMbcaRyus.dat Uh3JDInZxY.dat VyQQt2w3fN.dat XCF0Yo8wht.dat YXUIzGHjFz.dat QEHgM7cAaq.tmp Rne3tqCHl6.dat SrMPli52Bn.dat Uhbzs4000k.dat W0aW9KHVgQ.dat XDnKHw7yP9.dat Z2HjgjYMmV.dat QHhPRsCc2h.dat RNOFFqhpNW.dat SSdrpglOOQ.dat UhxYuOmqq4.dat W1cP1RWRyf.dat Xf1YUqVWUv.dat Z9nwXH0wZb.dat QifbSD8Gs3.dat Rpne6gvJlS.dat SV0sAkSWRM.dat UJaJhRcPWF.dat W40E0rqRgp.dat XkYzvJwyqz.dat Z9YeFB9Mrp.dat QIGr7lS98m.dat RqJQdw3ELK.dat SwFwpcXxcs.dat ULIbsE4WvB.dat W7yltEdKmh.dat XmSQxf97mb.dat Zdc7t0ZJ5m.dat [skip] ```] 💡 **NOTE** > **history** : show history of commands (see also : HISTSIZE, HISTFILESIZE) > **|** : pass output of left command to input right command > **grep**
pattern
: print lines matching a pattern --- ### 11. Find dat files .remark-code.hljs[``` [testing@supernova:super_nova_test]$ find . -name "*.dat" ./Qst1yEn40y.dat ./QsArAZrgWD.dat ./first_testing/TuCWd8jmys.dat ./first_testing/UJaJhRcPWF.dat ./first_testing/UcPQC53B2c.dat ./first_testing/RxnYj1G6XZ.dat ./first_testing/YiapNHGqAf.dat [skip] ./first_testing/Zw8790WLQ7.dat ./first_testing/RBRJG77AdM.dat ./first_testing/YCdeI7SUzs.dat ./first_testing/ZIgJFMTmuQ.dat ./first_testing/ZyfJVTAYBc.dat ./first_testing/WwDlpAIsGm.dat ```] 💡 **NOTE** > **find** : search for files in a directory hierarchy > *-name
pattern
* : file name (the path with the leading directories removed) matches pattern --- ### 12. Find and remove .remark-code.hljs[``` [testing@supernova:super_nova_test]$ find . -name "*.dat" -exec rm {} \; ```] 💡 **NOTE** > **find . -name "pattern" -exec
command
;** : find and execute > *-exec
command
;* : execute command --- ### 13. Recheck file systems .remark-code.hljs[``` [testing@supernova:super_nova_test]$ df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg-root 11.9G 6.7G 5.1G 56% / /dev/mapper/vg-home 6.9G 0.8G 6.1G 12% /home none 492K 4.0K 488K 1% /dev udev 12G 0 12G 0% /dev/tty tmpfs 12G 0 12G 0% /dev/shm tmpfs 12G 8.1M 12G 1% /run tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 12G 0 12G 0% /sys/fs/cgroup tmpfs 2.4G 0 2.4G 0% /run/user/0 [testing@supernova:super_nova_test]$ cd - /home/testing ```] 💡 **NOTE** > **cd** : change the working directory > *-* : cd "$OLDPWD" && pwd (changes to the previous working directory and then writes its name) --- ### 14. Untar .remark-code.hljs[``` [testing@supernova:~]$ tar xvzf super_nova_simulations.tgz super_nova_simulations/ super_nova_simulations/IEG0ctikNK.dat super_nova_simulations/xUPb0ybDie.dat super_nova_simulations/runme.py super_nova_simulations/create_files.sh super_nova_simulations/uYLuadok35.py super_nova_simulations/1jnLpRUfia.dat super_nova_simulations/qtTVBJ8iWX.bin super_nova_simulations/6TJvY2GdQP.dat super_nova_simulations/BBTwbQR42v.py super_nova_simulations/KGO5xjA2n1.dat super_nova_simulations/samples.dat super_nova_simulations/Cojc4YTcDk.bin [testing@supernova:~]$ cd super_nova_simulations ```] --- ### 15. Run simulations (oups) .remark-code.hljs[``` [testing@supernova:super_nova_simulations]$ ./runme.py bash: ./runme.py: Permission denied [testing@supernova:super_nova_simulations]$ ls -alh total 3.1G drwxrwxr-x 2 testing testing 4.0K Nov 2 16:42 . drwxrwxr-x 4 testing testing 4.0K Nov 2 16:39 .. -rw------- 1 testing testing 4.4K Oct 15 17:45 1jnLpRUfia.dat -rw------- 1 testing testing 12K Oct 15 17:45 6TJvY2GdQP.dat -rw------- 1 testing testing 17K Oct 15 17:45 BBTwbQR42v.py -rw------- 1 testing testing 26K Oct 15 17:45 Cojc4YTcDk.bin -rw-r--r-- 1 testing testing 502 Oct 15 17:45 create_files.sh -rw------- 1 testing testing 2.3K Oct 15 17:45 IEG0ctikNK.dat -rw------- 1 testing testing 16K Oct 15 17:45 KGO5xjA2n1.dat -rw------- 1 testing testing 16K Oct 15 17:45 qtTVBJ8iWX.bin -rw-rw-r-- 1 testing testing 35K Nov 2 16:42 runme.py -rw-rw-r-- 1 testing testing 3.0G Oct 15 16:27 samples.dat -rw------- 1 testing testing 5.1K Oct 15 17:45 uYLuadok35.py -rw------- 1 testing testing 8.0K Oct 15 17:45 xUPb0ybDie.dat [testing@supernova:super_nova_simulations]$ chmod 760 runme.py [testing@supernova:super_nova_simulations]$ ./runme.py ./runme.py: line 5: syntax error near unexpected token "Starting super nova simulations" ./runme.py: line 5: print("Starting super nova simulations") [testing@supernova:super_nova_simulations]$ head -3 runme.py import os, sys import re import numpy as np ```] 💡 **NOTE** > **./
programme
** : run programme in current directory (instead of $PATH) > **chmod** *760*: change file rights (760 = rwx(7)rw-(6)---(0)) > **head** *-3* : output the first part of files (here just first 3 lines) --- ### 16. Where is python ? .remark-code.hljs[``` [testing@supernova:super_nova_simulations]$ which python /usr/bin/python [testing@supernova:super_nova_simulations]$ python --version Python 3.9.7 [testing@supernova:super_nova_simulations]$ python runme.py Starting super nova simulations Could not lock temporary file, aborting... ```] 💡 **NOTE** > **which
programme
** : locate a command > **python --version** : show python version --- ### 17. Kill .remark-code.hljs[``` [testing@supernova:super_nova_simulations]$ ps -aux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 2.1 169340 5580 ? Ss Oct15 0:05 /sbin/init root 47 0.0 3.1 29644 8252 ? Ss Oct15 0:02 /lib/systemd/systemd-journald root 74 0.0 0.6 45080 1676 ? Ss Oct15 0:01 /usr/sbin/sssd -i --logger=files root 225 0.0 0.7 15848 1996 ? Ss Oct15 0:00 /usr/sbin/sshd -D root 309 0.0 1.5 48460 3980 ? S Oct15 0:15 /usr/lib/x86_64-linux-gnu/sssd/sssd_be --domain caltech.edu root 310 0.0 2.0 62844 5452 ? S Oct15 0:12 /usr/lib/x86_64-linux-gnu/sssd/sssd_nss --uid 0 --gid 0 --lo root 311 0.0 3.3 42204 8864 ? S Oct15 0:11 /usr/lib/x86_64-linux-gnu/sssd/sssd_pam --uid 0 --gid 0 --lo root 317 0.0 1.8 28044 4896 ? Ss Oct15 0:02 /lib/systemd/systemd-logind root 321 0.0 0.0 2416 104 pts/0 Ss+ Oct15 0:00 /sbin/agetty -o -p -- \u --noclear --keep-baud console 11520 root 322 0.0 0.0 2416 112 pts/0 Ss+ Oct15 0:00 /sbin/agetty -o -p -- \u --noclear --keep-baud tty1 115200,3 root 323 0.0 0.0 2416 52 pts/1 Ss+ Oct15 0:00 /sbin/agetty -o -p -- \u --noclear --keep-baud tty2 115200,3 root 6980 0.0 3.2 35524 8516 ? Ss 02:15 0:00 sshd: testing [priv] testing 6983 0.0 3.2 21024 8404 ? Ss 10:11 0:00 /lib/systemd/systemd --user testing 6984 0.0 1.0 189200 2736 ? S 10:11 0:00 (sd-pam) testing 6992 0.0 1.9 35524 5052 ? S 10:11 0:00 sshd: testing@pts/2 testing 6993 0.0 1.6 13372 4416 pts/2 Ss 10:11 0:00 -bash testing 6996 0.0 1.1 3780 2960 pts/2 S+ 10:11 0:00 python ./runme.py root 7008 0.0 3.2 35524 8548 ? Ss 12:15 0:00 sshd: testing [priv] testing 7018 0.0 1.9 35524 5060 ? S 12:15 0:00 sshd: testing@pts/3 testing 7019 0.0 1.6 13372 4332 pts/3 Ss 12:15 0:00 -bash testing 7205 0.0 1.1 16180 2976 pts/3 R+ 12:18 0:00 ps -aux [testing@supernova:super_nova_simulations]$ kill 6996 [testing@supernova:super_nova_simulations]$ kill -s 9 6996 ```] 💡 **NOTE** > **ps** : show current processes > *-aux* : all processes, effective user ID, Lift the BSD-style "must have a tty" restriction > **kill** : send a signal to a process (9 = SIGKILL, 15 = SIGTERM (default), 2 = SIGINT (Ctrl-C)) --- ### 18. Relaunch .remark-code.hljs[``` [testing@supernova:super_nova_simulations]$ python runme.py Starting super nova simulations Simulation is successfully started... ```] .center[ ![Bzinga](images/bazinga.jpg) ]