Data: Moving Local Files to the TeraGrid

Home > User Info > Data > Moving Local Files to the TeraGrid

This section describes common Unix commands that can be used to transfer data from a local client to the TeraGrid.  For high-performance data transfer between TeraGrid resources please see How to Initiate Transfers.

The SSH protocol provides a file transfer mechanism called Secure Copy (SCP) that uses the same encrypted channel as SSH to send files securely over the network.  This UNIX utility uses public key encryption to authenticate and encrypt communications.  To take advantage of certificate authentication, gsiscp, a version of scp that is based on the Grid Security Infrastructure (GSI), can use your proxy certificate.

While scp and gsiscp are easy to use, they provide very poor performance when transferring multiple files or large files; in addition, scp does not support file transfers above 2 GB.  Data loss may result.  gsiscp is not supported in all locations (e.g., NCSA's mass storage, UniTree). 

As a rule-of-thumb, when transferring file larger than 1 GB on the TeraGrid, GridFTP-based programs should be used for most TeraGrid file transfers.

Within home and scratch directories, mv and cp work as usual.  These commands are not supported at NCSA's mass storage, Unitree.

Please note that scp on most sites of the TeraGrid requires an ssh2 client; if you are trying to copy files from your local workstation using ssh1, you may receive an error message.

Scp/gsicp Syntax

The syntax of scp and gsiscp are the same:

% scp/gsiscp <source> <destination>
scp [user1@][host1]:][file1] [user2@][host2]:[file2]

Directories may be copied with a recursive option (-r).  As with most Unix utilities, more information about scp may be found from its man page (type man scp at the prompt).

For scp, the source and destination host will challenge for passphrases as necessary.  Gsiscp requires that the user have an active proxy certificate session enabled.  See the Proxy Certificate page of the Access section of this guide for more information on initiating a proxy.

Examples

To copy everything in the local directory to SDSC's TeraGrid cluster into the directory ~/run1:

% gsiscp * tg-login.sdsc.teragrid.org:run1

or

% scp * tg-login.sdsc.teragrid.org:run1

Transferring large files with bbftp (SDSC only)

For transferring large data files, typically those bigger than 2GB in size, SDSC has installed bbftp software on DataStar and the IA-64 cluster. This uses multiple transfer streams and compression in order to speed up the transfer process, making transfers up to 10x faster than can be achieved using regular scp. See SDSC's TeraGrid transfer documentation for details.