       |
| ||
| ||
|
|
Storage
User Guide - IntroductionThe user guide for this resource involves getting a GRID certificate and learning how to use two commands,migftp and globus-url-copy. In order to get access to
the storage area, you must have a valid grid
certificate. This certificate is used instead of regular passwords
as the authentication mechanism when accessing the storage
system. There is no need for a special account/username, everything
is resolved through the unique certificate id. After having received
the certificate you need to send an email to support@nsc.liu.se requesting
your certificate being activated on the storage system.
IMPORTANT
For more information regarding certificates and public key cryptography:
Getting a Grid Certificate
The following steps tells you how to create a certificate request
together with a private key. In order to be useful, this certificate
request (a public key, together with your name, address and
organisation) must be digitally signed by a certificate authority that
we trust. This digital signature binds this public key, to you as a
person.
Renewing a Grid Certificate
The Grid Certificate expires in 1 year and must at some time be
"renewed". Your old certificate will not really be renewed since you
will get a completely new certificate together with a new private key
and the old certificate together with the old private key will be
obsolete. The new certificate and the new private key must both be
distributed again to all the systems that will use them.
IMPORTANT: Do not mix up the old certificate and the old private key with the newly generated ones. The signed certificate from the certificate authority is only valid together with the private key that was created together with the certificate request. If these files does not match together and the correct pair get lost, you will have to renew your certificate again, which may take some time.
Transferring Files
Storage Server: nuffs.nsc.liu.se
Accessible from any computer with the NorduGrid software installed currently available from the following clusters (contact support@nsc.liu.se if you need this to work on a computer not administrated by NSC):
Current directory structure at nuffs: /gridstorage/misu/dblcopy/ /gridstorage/misu/nobackup/ /gridstorage/rossby/dblcopy/ /gridstorage/rossby/nobackup/ /gridstorage/smhid/dblcopy/ /gridstorage/smhid/nobackup/ /gridstorage/smhip/dblcopy/ /gridstorage/smhip/nobackup/
Make sure you have the correct certificate and private key in ~/.globus at the machine from where the file transfer is performed (see 3b and 3c). Also verify the file permission flags on the directory and files (most importantly, userkey.pem should only be readable by you): [perl@dunder ~]$ ls -al ~/.globus/ drwxr-xr-x 2 perl nsc 43 Dec 7 14:23 . drwx------ 21 perl nsc 4096 Dec 7 14:23 .. -rw-r--r-- 1 perl nsc 1155 Dec 7 16:25 usercert.pem -rw-r--r-- 1 perl nsc 1354 Dec 7 16:25 usercert_request.pem -r-------- 1 perl nsc 963 Dec 7 16:25 userkey.pem Every time you want access to the storage area you must begin by authenticating yourself using your grid certificate. Since the private key userkey.pem is encrypted you must decrypt this key using the pass phrase you choose while creating the certificate request. This step will create a proxy certificate which is valid as authentication for up to 12 hours. After 12 hours has passed you must do this step again (ongoing file transfers will not be terminated but you must authenticate yourself again if you start a new file transfer session to nuffs). To create the proxy certificate type: [perl@tornado ~]$ grid-proxy-init Your identity: /O=Grid/O=NorduGrid/OU=nsc.liu.se/CN=Per Lundqvist Enter GRID pass phrase for this identity: Creating proxy .......................................... Done Your proxy is valid until: Thu Dec 22 23:14:21 2005 To see other options such as changing the time the proxy certificate is valid type: [perl@tornado ~]$ grid-proxy-init -help8. Use one of the following clients to transfer your files: migftp globus-url-copy Regardless of which application is used to transfer files it is recommended to archive your data with tar before uploading to nuffs. This has the advantage of preserving the complete data structure and being more efficient since it is more efficient to transfer and store a few large files than many small.
globus-url-copy [options] sourceURL destURLwhere sourceURL and destURL is on the form: file://full_path_to_file or
gsiftp://full_path_to_file
file: specifies the full path on the cluster. NOTE: this will result in 3 forward-slashes as in file:///home/perl/file. gsiftp: specifies the path to the file on nuffs. For example: gsiftp://nuffs/gridstorage/rossby/nobackup/perl. One useful option is the verbose flag ``-vb''. For help, type: globus-url-copy -help
Examples# Generate a proxy-certificate that is valid for 12 hours:[perl@tornado ~]$ grid-proxy-init Your identity: /O=Grid/O=NorduGrid/OU=nsc.liu.se/CN=Per Lundqvist Enter GRID pass phrase for this identity: Creating proxy ................................... Done Your proxy is valid until: Wed Jan 25 01:58:07 2006# Start migftp with current working directory at nuffs set to /gridstorage/rossby/nobackup/perl [perl@tornado ~]$ migftp rossby/nobackup/perl Connecting to: sftp://nuffs.nsc.liu.se:3022 cd ok, cwd=3D/gridstorage/rossby/nobackup/perl [ftp] nuffs.nsc.liu.se:~/rossby/nobackup/perl># Use ``mirror'' to recursively download whole directory trees (but without preserving symbolic links) from nuffs. Use ``mirror -R'' (reverse) to upload recursively. All file transfer commands (put, mput, get, mget, mirror) has the ability to resume previously unfinished file transfers using the option ``-c'' (continue). [ftp] nuffs.nsc.liu.se:/gridstorage/rossby/nobackup/perl> !ls bin example.dir install tmp [ftp] nuffs.nsc.liu.se:/gridstorage/rossby/nobackup/perl> !ls example.dir -l total 2048004 drwxr-xr-x 2 perl nsc 27 Mar 17 10:01 dir1 -rw------- 1 perl nsc 1048576000 Mar 17 09:36 file100 -rw-r--r-- 1 perl nsc 1048576000 Mar 17 09:36 file100.copy.1 lrwxrwxrwx 1 perl nsc 7 Mar 17 11:30 file100.ln -> file100# Recursively upload directory example.dir to nuffs while also ignoring symbolic links: [ftp] nuffs.nsc.liu.se:/gridstorage/rossby/nobackup/perl> mirror -R -c example.dir Total: 2 directories, 3 files, 1 symlink New: 3 files, 0 symlinks [ftp] nuffs.nsc.liu.se:/gridstorage/rossby/nobackup/perl> ls example.dir drwxr-xr-x 3 perl nsc 256 Mar 17 14:18 . drwxr-xr-x 3 perl nsc 256 Mar 17 14:17 .. drwxr-xr-x 2 perl nsc 256 Mar 17 14:18 dir1 -rw------- 1 perl nsc 1048576000 Mar 17 14:18 file100 -rw-r--r-- 1 perl nsc 1048576000 Mar 17 14:18 file100.copy.1# Upload directory example.dir recursively to nuffs recreating symbolic links on nuffs as the regular files they point to (using ``-L''): [ftp] nuffs.nsc.liu.se:/gridstorage/rossby/nobackup/perl> mirror -R -c -L example.dir Total: 2 directories, 4 files, 0 symlinks New: 1 file, 0 symlinks [ftp] nuffs.nsc.liu.se:/gridstorage/rossby/nobackup/perl> ls example.dir drwxr-xr-x 3 perl nsc 256 Mar 17 14:21 . drwxr-xr-x 3 perl nsc 256 Mar 17 14:17 .. drwxr-xr-x 2 perl nsc 256 Mar 17 14:18 dir1 -rw------- 1 perl nsc 1048576000 Mar 17 14:18 file100 -rw-r--r-- 1 perl nsc 1048576000 Mar 17 14:18 file100.copy.1 -rw------- 1 perl nsc 1048576000 Mar 17 14:21 file100.ln# Since symbolic links are not preserved when uploading files, archive the directory using tar and automatically upload the tar-file to nuffs when the archive has been created by using the ``queue'' command:
[ftp] nuffs.nsc.liu.se:/gridstorage/rossby/nobackup/perl> queue !tar cf example.dir.tar example.dir/
[ftp] nuffs.nsc.liu.se:/gridstorage/rossby/nobackup/perl> jobs
[0] queue (sftp://nuffs.nsc.liu.se:3022)
Now executing: [1] ! tar cf example.dir.tar example.dir/
[1] ! tar cf example.dir.tar example.dir/
[ftp] nuffs.nsc.liu.se:/gridstorage/rossby/nobackup/perl> queue put example.dir.tar
[ftp] nuffs.nsc.liu.se:/gridstorage/rossby/nobackup/perl> jobs
[0] queue (sftp://nuffs.nsc.liu.se:3022)
Now executing: [1] ! tar cf example.dir.tar example.dir/
Commands queued:
1. put example.dir.tar
[1] ! tar cf example.dir.tar example.dir/
[ftp] nuffs.nsc.liu.se:/gridstorage/rossby/nobackup/perl> jobs
[0] queue (sftp://nuffs.nsc.liu.se:3022)
Queue is stopped.
[ftp] nuffs.nsc.liu.se:/gridstorage/rossby/nobackup/perl> ls
drwxr-xr-x 2 perl nsc 256 Mar 17 12:40 .
drwxrwxrwt 17 nfsnobody rossby 8192 Mar 16 16:14 ..
-rw-r--r-- 1 perl nsc 3145738240 Mar 17 12:42 example.dir.tar
[ftp] nuffs.nsc.liu.se:/gridstorage/rossby/nobackup/perl> !ls -l example.dir.tar
-rw-r--r-- 1 perl nsc 3145738240 Mar 17 12:40 example.dir.tar
# Example when the proxy certificate must be renewed:
[ftp] nuffs.nsc.liu.se:~/smhid> cd nobackup/ cd: Login failed: Password required [ftp] nuffs.nsc.liu.se:~/smhid> suspend [1]+ Stopped lftp sftp://nuffs.nsc.liu.se:2022 [perl@tornado ~]$ grid-proxy-init Your identity: /O=3DGrid/O=3DNorduGrid/OU=3Dnsc.liu.se/CN=3DPer Lundqvist Enter GRID pass phrase for this identity: Creating proxy ..................................................... Done Your proxy is valid until: Fri Dec 23 02:40:33 2005 [perl@tornado ~]$ fg lftp sftp://nuffs.nsc.liu.se:2022 [ftp] nuffs.nsc.liu.se:~/smhid> cd nobackup/ cd ok, cwd=3D/gridstorage/smhid/nobackup [ftp] nuffs.nsc.liu.se:~/smhid/nobackup># Retrieve all tar files from nuffs using mget and wild-cards while also specifying the directory where the files should be stored (without -O they are stored at the local current working directory): [ftp] nuffs.nsc.liu.se:/gridstorage/rossby/nobackup/perl> ls drwxr-xr-x 3 perl nsc 256 Mar 17 14:48 . drwxrwxrwt 17 nfsnobody rossby 8192 Mar 16 16:14 .. drwxr-xr-x 3 perl nsc 256 Mar 17 14:21 example.dir -rw-rw-r-- 1 perl nsc 3145738240 Mar 17 14:49 example.dir.2.tar -rw-r--r-- 1 perl nsc 3145738240 Mar 17 12:42 example.dir.tar [ftp] nuffs.nsc.liu.se:/gridstorage/rossby/nobackup/perl> mget -c example.dir.*tar -O tmp/ 5373706240 bytes transferred in 168 seconds (30.57M/s) Total 2 files transferred [ftp] nuffs.nsc.liu.se:/gridstorage/rossby/nobackup/perl> !ls -l tmp/example.dir.*tar -rw-r--r-- 1 perl nsc 3145738240 Mar 17 14:55 tmp/example.dir.2.tar -rw-r--r-- 1 perl nsc 3145738240 Mar 17 14:56 tmp/example.dir.tar globus-url-copy examples:# copy /home/perl/example.dir.2.tar from tornado to /gridstorage/rossby/nobackup/perl/example.dir.2.tar at nuffs:globus-url-copy -vb file:///home/perl/example.dir.2.tar \ gsiftp://nuffs/gridstorage/rossby/nobackup/perl/example.dir.2.tar# make a tar archive before uploading, in order to preserve symbolic links: [perl@tornado ~]$ tar cvf example.dir.2.tar example.dir & \ globus-url-copy -vb file:///home/perl/example.dir.2.tar \ gsiftp://nuffs/gridstorage/rossby/nobackup/perl/example.dir.2.tar example.dir/ example.dir/file100 example.dir/file100.copy.1 example.dir/dir1/ example.dir/dir1/file100.copy.2 example.dir/file100.ln 3123707904 bytes 28445.33 KB/sec avg 52224.00 KB/sec inst Using screen
When using migftp or globus-url-copy we recommend you to use it
together with the command screen.
screen examples# Using screen together with migftp on dunder (screen can be used by any application that only displays its output to the terminal):[perl@dunder ~]$ screen [perl@dunder ~]$ grid-proxy-init Your identity: /O=3DGrid/O=3DNorduGrid/OU=3Dnsc.liu.se/CN=3DPer Lundqvist Enter GRID pass phrase for this identity: Creating proxy ................................................................ Done Your proxy is valid until: Tue Mar 21 00:39:08 2006 [perl@dunder ~]$ migftp rossby/nobackup/perl Connecting to: sftp://nuffs.nsc.liu.se:3022 cd ok, cwd=3D/gridstorage/rossby/nobackup/perl [ftp] nuffs.nsc.liu.se:/gridstorage/rossby/nobackup/perl> [ftp] nuffs.nsc.liu.se:/gridstorage/rossby/nobackup/perl> mput example.file `example.file' at 33306202 (0%) 5.38M/s eta:85m [Sending data]# Press ctrl-a followed by d (i.e. "C-a d") to detach this screen and go back to the ordinary shell. The screen session (including migftp and any other processes running in the screen) will continue to live even if I without of detaching from it, kill the corresponding xterm): [detached] [perl@dunder ~]# To see available screen sessions: [perl@dunder ~]$ screen -ls There is a screen on: 10972.pts-0.dunder (Detached) 1 Socket in /tmp/uscreens/S-perl.# To resume the screen above you must specify the shortest string that uniquely specifies a screen session, i.e. in this case all of these will work: [perl@dunder ~]$ screen -r 10972.pts-0.dunder [perl@dunder ~]$ screen -r 10972 [perl@dunder ~]$ screen -r 109 [perl@dunder ~]$ screen -r# When resuming a screen session, all content that has been written to the screen will be displayed: [perl@dunder ~]$ grid-proxy-init Your identity: /O=3DGrid/O=3DNorduGrid/OU=3Dnsc.liu.se/CN=3DPer Lundqvist Enter GRID pass phrase for this identity: Creating proxy ................................................................ Done Your proxy is valid until: Tue Mar 21 00:39:08 2006 [perl@dunder ~]$ migftp rossby/nobackup/perl Connecting to: sftp://nuffs.nsc.liu.se:3022 cd ok, cwd=3D/gridstorage/rossby/nobackup/perl [ftp] nuffs.nsc.liu.se:/gridstorage/rossby/nobackup/perl> [ftp] nuffs.nsc.liu.se:/gridstorage/rossby/nobackup/perl> mput example.file `example.file' at 1187385685 (2%) 35.67M/s eta:18m [Sending data]# Detaching a screen that is already attached to another terminal:
[perl@dunder ~]$ screen -ls
There is a screen on:
10972.pts-0.dunder (Attached)
1 Socket in /tmp/uscreens/S-perl.
[perl@dunder ~]$ screen -d 10972
[10972.pts-0.dunder detached.]
# To terminate a screen you only would have to log out from the shell
you got (from where grid-proxy-init and migftp was started above),
with ``exit'' or using "C-d":
[ftp] nuffs.nsc.liu.se:/gridstorage/rossby/nobackup/perl> quit [perl@dunder ~]$ exit [screen is terminating] [perl@dunder ~]$
|
