File Transfer Overview

Secure and efficient data transfer to and from CARC systems can be achieved with a variety of useful tools, the choice of which depends on whether the storage location is a personal computer or an external site (e.g., cloud storage). The choice also depends heavily on the requirement of data sensitivity as well as the familiarity of the user. There are three methods of data transfer that we recommend for use with CARC systems: command-line tools, graphical tools, and the Globus service.

Note: Due to security risks, please be mindful of the type of information being transferred. Where possible, omit all information that may be considered confidential. For examples of confidential information that requires additional consideration, visit http://itservices.usc.edu/security/--sensitive-info.

Data Management on CARC Systems video

Dedicated data transfer nodes

CARC has two dedicated, high-speed, 100 Gbps data transfer nodes at hpc-transfer1.usc.edu and hpc-transfer2.usc.edu. These nodes are especially useful for larger transfers. If needed, you can log in to them using, for example, ssh <username>@hpc-transfer1.usc.edu. The Discovery and Endeavour login nodes have a 40 Gbps connection speed and are adequate for most transfers.

Note: Your transfer speeds are determined by a number of factors, such as the network speed at your location, router and firewall settings, etc. If you experience slower than expected transfers, try to troubleshoot these issues first.

Command-line tools

There are a number of command-line interface (CLI) tools to transfer data to and from CARC storage systems, such as sftp, rsync, and rclone. Each is targeted to specific use cases.

For more information, see the guide for Transferring Files Using the Command Line.

Graphical tools

Applications such as Cyberduck, FileZilla, and WinSCP provide a graphical user interface (GUI) to transfer data between a personal computer and a storage solution that allows SFTP connections, including CARC storage systems. These applications offer drag-and-drop capability, but transfer speeds may be slower compared to using a command-line tool.

For more information, see the guide for Transferring Files Using a Graphical User Interface.

Globus service

Globus is a data management and transfer service that gives researchers unified access to their data across systems through a web-based GUI. It can be used for data transfers from a personal computer or another HPC center to CARC storage systems. Relative to other tools, it is useful for large transfers and will provide the best transfer speeds. A CLI for Globus can also be used if desired.

For more information, see the guide for Transferring Files Using Globus.

Which method should I use?

Below are four example scenarios that provide some insight into which data transfer method you might use for a given situation:

System 1System 2Example ScenariosMethod
Personal computerCARC file system for small-medium transfersWhen transferring files from a personal computer to your CARC project folder that takes a moderate amount of timeGUI, CLI
Personal computerCARC file system for large or secure transfersWhen transferring files from a personal computer to your CARC project directory that takes a large amount of time or needs to be encryptedGlobus
Amazon Web Services (AWS)Any CARC file systemWhen transferring files from an AWS server to your CARC project directoryCLI
Other HPC centerAny CARC file systemWhen transferring files from another university or research institution to your CARC project directoryGlobus
Back to top