In 2020, the Center for Advanced Research Computing performed a major upgrade to its systems. Its new high-performance computing cluster, Discovery, includes additional compute nodes and a rebuilt software stack, as well as new system configurations to better serve CARC users. A new project file system was also deployed, offering more storage space and better performance. A new condo cluster, Endeavour, was launched in December 2020 as part of the new Condo Cluster Program.
The following graphic depicts the CARC cyberinfrastructure and how different systems interact with one another:
The CARC's Discovery computing cluster consists of 2 shared login nodes and a total of around 11,000 CPU cores in around 500 compute nodes. The compute nodes have varying numbers of cores and amounts of memory, and the cluster has a few partitions to organize nodes with specific intended uses. The entire cluster resides on a 56 gigabit FDR InfiniBand backbone.
Each user has access to four file systems for their CARC account: /home1, /project, /scratch, and /scratch2.
The /home1 file system consists of personal directories of 100 GB for each user for their configuration files and personal scripts. The /project file system is a high-performance, parallel I/O file system with a capacity of 8.4 PB of usable space and consists of directories for different research project groups. The scratch file systems — /scratch and /scratch2 — are high-performance, parallel I/O file systems intended for storing temporary files and for I/O-intensive jobs. /scratch has a capacity of 806 TB and /scratch2 has a capacity of 709 TB, with users receiving a 10 TB quota for each file system.
For more information on the different file systems and their intended uses, see the Storage File Systems user guide.
The CARC uses a customized distribution of the Community Enterprise Operating System (CentOS), built using the publicly available RPM Package Manager (RPM). CentOS is a high-quality Linux distribution that gives the CARC complete control of its open-source software packages and is fully customized to suit advanced research computing needs, without the need for license fees.
The CARC’s distribution of CentOS was modified for minor bug fixes and desired localized behavior. Many desktop and clustering-related packages were also added to the CARC's CentOS installation.
A number of white papers, tutorials, FAQs, and other documentation on CentOS can be found on the official CentOS website.