Transferring Files Using Globus

Globus provides a relatively simple and reliable way to access and move research data between systems with faster transfer speeds than traditional tools such as sftp, scp, and rsync.

Globus requires some setup ahead of time, but it is robust and appropriate when your data transfer will take a long time or your connection may be periodically interrupted. It also allows you to share your data with colleagues, move data back and forth between CARC storage systems and personal workstations, and restart or resume a failed or paused file transfer — even a very large one.

Note: Due to security risks, please be mindful of the type of information being transferred. Where possible, omit all information that may be considered confidential. For examples of confidential information that requires additional consideration, visit http://itservices.usc.edu/security/sensitive-info.

Getting started with Globus

To start using Globus, go to https://www.globus.org in your browser and click Log In. Type University of Southern California in the box that says "Use your existing organizational login" and click Continue:

Globus Software Set Up

You will then be taken to the USC Shibboleth page to enter your USC NetID and password:

Globus Software Set Up

After authenticating, you'll be taken to the Globus File Manager page:

Globus Software Set Up

You can toggle the number and configuration of the two panes using the Panels menu in the top right of the File Manager page.

Setting up Collections for file transfers

In Globus terminology, a user sets up a Collection, which is simply a place to transfer files. This can be a folder on your laptop, your directories on CARC systems, or even a remote location you have access to through a scientific computing affiliation.

In the Globus user interface, Collections are managed in terms of Endpoints. From the main Globus File Manager page, click Endpoints in the left menu panel to be taken to the Endpoints page:

Globus Software Set Up

On the Endpoints page, click Create a personal endpoint in the upper right corner:

Globus Software Set Up

Installing Globus Connect Personal

Next, you'll be presented with a link to download and install the desktop service, Globus Connect Personal, on your local computer:

Globus Software Set Up

After going through the standard software installation process for your computer, start the Globus Connect Personal application:

Globus Software Set Up

A new window will pop up asking you to log in again:

Globus Software Set Up

Once logged in, allow the setup to continue:

Globus Software Set Up

Globus Installation Guides

Establishing an endpoint

You can now establish an Endpoint on your local computer, called a Collection. Under Collection Name, choose a descriptive name. Do not choose the High Assurance option — that feature is beyond the scope of this document. After filling out the form, click Save:

Globus Software Set Up

Depending on whether Globus knows your account ID from a previous login or existing affiliation, you may be asked to generate a Setup Key for your collection:

Globus Software Set Up

Save this key in case you are asked for it during the next one or two steps.

In the setup window, clicking Save will take you to the Globus File Manager page in your browser. Click Endpoints in the left menu panel and then the Administered By You tab of the middle menu. You should see the Endpoint you created:

Globus Software Set Up

Click the Endpoint and on the next page, click Open in File Manager on the right hand side of the page:

Globus Software Set Up

This should take you to a listing of the files in the selected directory on your computer, including subfolders into which you can navigate:

Globus Software Set Up

Allowing access to your local files

For security reasons, Globus requires you to specifically allow files and folders on your computer to be shared or transferred.

On Windows, to allow a folder's contents to be transferred, right click on the small g (the Globus icon) in your running task icons in the task bar.

Select the Options... menu item:

Globus Software Set Up

You will be presented with a window to add a folder containing the files you want to transfer. If you click the + sign in the lower right (highlighted in blue) you will be presented with a standard file explorer that gives you the ability to add a folder on your local hard drive. For now, only keep the Writable option checked and the Shareable option unchecked:

Globus Software Set Up

Click Save to continue.

On a Mac, the process is similar. You access the small Globus g icon in the top menu bar and choose Preferences... and then the Access tab:

Globus Software Set Up

Setting up access to your CARC directories

To set up access to your CARC /home1, /project, /scratch, and /scratch2 directories on Discovery/Endeavour, click Endpoints in the left menu panel and then the Shared With You tab of the middle menu. Next, enter USC CARC User Directories in the search box above the middle menu and click the magnifying glass. The USC data transfer node endpoint should appear in the main window:

Globus Software Set Up

After selecting the USC CARC User Directories endpoint, you will be taken to the Endpoint's main page:

Globus Software Set Up

Click the Credentials tab and you will see that authentication and consent are required for Globus to manage collections on the Endpoint. Click Continue:

Globus Software Set Up

You will be asked to select your identity to continue. If you've already authenticated via Shibboleth, you should see your USC NetID listed. Otherwise, you will be prompted to log in with your NetID:

Globus Software Set Up

You will then need to grant Globus a list of permissions by clicking Allow:

Globus Software Set Up

After allowing Globus these permissions, you will be taken back to the Endpoint's main page. Under the Credentials tab, you will now see your NetID listed with an "active" status:

Globus Software Set Up

There is one final step for authenticating the Endpoint for file transfers. On the Endpoint's main page under the Overview tab, click Open in File Manager:

Globus Software Set Up

You will be taken to the File Manager page, where Globus requires one more authentication/consent. Click Continue to complete the final step:

Globus Software Set Up

Click Allow to grant Globus the permissions:

Globus Software Set Up

You will be taken back to the File Manager page, where you should see your /home1 directory, which will look something like the following:

Globus Software Set Up

Transferring files

The File Manager page is the page you'll use for your file transfers, and it has a two-pane bi-directional layout:

Globus Software Set Up

Tip: You can toggle the number and configuration of the two panes using the Panels menu in the top right of the File Manager page.

In the Collection search field at the top of either column, you can search for USC CARC User Directories to access your CARC directories. By default, you will be in your /home1 directory. You can navigate to other directories (/project, /scratch, or /scratch2) by typing their paths in the Path field, or you can enter / to view all directories.

Tip: Your project directory path is of the form /project/<PI_username>_<id>.

In the other column, you can click the Collection search field and navigate to the "Your Collections" tab, where you'll find your personal computer's Collection you set up:

Globus Software Set Up

After selecting the two Collections and navigating to the desired directories, your File Manager page will look something like this:

Globus Software Set Up

This user wants to transfer files from the "test" folder on their computer to their CARC /scratch directory, which is currently empty.

To begin the transfer, highlight one or more of the files in your local folder and click the blue Start button. A green alert will appear in the upper right notifying you that the transfer was submitted and giving you the option to View details:

Globus Software Set Up

Clicking View details will take you to a page where you can watch the transfer's progress and view other information about your transfer:

Globus Software Set Up

If your files are large, Globus takes a few seconds or minutes to index them and get ready to transfer.

You'll also receive an email notifying you of the file transfer success (or failure). Make a note of the Task ID in case your transfer fails and you need to restart it.

When your transfer completes, go back to the File Manager window. In the CARC directory column, click the refresh button in the middle menu (the right-curling arrow right under the Path field). This will pull an updated listing of the files in your CARC directory (in this example, the user's /scratch directory). If needed, scroll through the list and you will see your files there:

Globus Software Set Up

Tip: You don't have to transfer files one-by-one. By highlighting a folder and clicking the Start button, you can transfer the folder and all its contents.

To download files from the data transfer node to your local computer instead, you should highlight files in the CARC column (rather than the column for your personal computer) and then click Start.

File transfer tips

  • The CARC file systems are Linux-based. That means file and folder names are case sensitive and spaces and strange characters are awkward. Do not use characters like slashes and dollar signs in your file names if you are planning to upload them to CARC systems. Replacing spaces in file and folder names with an underscore or dash is recommended.
  • Globus is capable of transferring a lot of small files, but in many cases you will get faster transfers and better results by creating a .tar, .gzip, or .zip file before trying to transfer data to or from CARC systems.

Syncing directories

Globus offers settings that can be applied to your transfer to synchronize your two directories. Syncing directories prevents the same files from being transferred repeatedly, saving you transfer time.

To synchronize your local directory and your CARC directory, access the Transfer & Sync Options menu located in between the two transfer columns. Select the "sync - only transfer new or changed files where the checksum is different" and "verify file integrity after transfer" checkboxes, and click the blue Start button to start your transfer:

Globus Software Set Up

There is also an option to delete files in the destination directory if they aren't in the source directory, as well as options to preserve source file modification times and encrypt the file transfer.

Restarting file transfers

If your transfer fails, you should first look at the last few events in the event log to identify any problems needing human intervention (quota exceeded, out of disk space, etc.). You can view event logs for transfers by navigating to the Activity tab in the left menu panel, selecting the transfer in question, and navigating to the transfer's Event Log tab:

Globus Software Set Up

Note: This transfer was successful, but a failed transfer is accessed in the same way.

After fixing the issue that caused the transfer to fail, you can resubmit the transfer in the same way as you did originally, making sure to synchronize the two directories to avoid re-transferring other files (see the "Syncing directories" section above).

Using bookmarks

The File Manager offers a bookmark feature to access your most-used directories easily. You can add a bookmark to a directory by clicking the bookmark ribbon next to the Path field in either column:

Globus Software Set Up

You can view and manage your Bookmarks when searching for a Collection, under the Bookmarks tab:

Globus Software Set Up

Sharing files

Globus also allows you to securely share files on CARC systems with external collaborators or data providers (with a Globus account). This feature allows external collaborators to both read and write files to a specific subdirectory within one of your directories on CARC systems.

To set up sharing, begin by creating a subdirectory specifically for Globus data sharing via the File Manager. Create a new subdirectory using the New folder button:

Globus Sharing Set Up

Name the directory and click the green checkmark button to save it. Then, right click the directory and select Share:

Globus Sharing Set Up

At the Authentication/Consent Required screen, select Continue. At the Guest Collections screen, click Add a Guest Collection:

Globus Sharing Set Up

At the Create New Guest Collection screen, fill in the form and click Create Collection:

Globus Sharing Set Up

You will then be taken to a Permissions page for this new Guest Collection. Here, you can select Add Permissions - Share With to provide read and/or write permissions to this subdirectory for your collaborators:

Globus Sharing Set Up

Note that the Path field is relative to the shared directory, so the default value of / is correct.

You can also create groups and manage permissions by group instead of individual.

In addition, you can provide public (anonymous) read access to your collection, so that anyone with a Globus account can download the files. It is also possible to create a research data portal based on Globus — contact us at carc-support@usc.edu for support.

For more details, see the official Globus docs:

Using the Globus CLI

Globus also provides a command-line interface that you can install; for more information, see the guide for Globus CLI.

Back to top