USRC Data Sources

ATLAS is a collaboration with Carnegie Melon University's Parallel Data Lab.  The repository contains traces from LANL supercomputers but also other systems as well, including a financial company.  There are also links to publications and insights garnered from this data.  The data for the ATLAS project is hosted by Carnegie Melon.

This tar file contains uncompressed VPIC restart files. The provided data are a subset of a much larger restart data set. VPIC is an open-source, particle-in-cell code developed at Los Alamos National Laboratory that is primarily used for astrophysics simulations. These data has proven problematic to compress and are frequently used by the Laboratory to test various compression algorithms.

• Download this data source via FTP

The data are System Environment Data Collections (SEDC), sensor telemetry (voltage, temperature, fan speeds, water flow information, etc.). These data cover 2/9/16–2/18/16, during which Trinity was in "open science" testing by users. See the README for more information. LA-UR-17-24849

• Download this data source via FTP

This outlines a collection of data released by LANL under LA-UR-19-28211 and available below.  This data is memory usage data from three open clusters from late 2018 through early 2019.

Citation

The following datasets were released in conjunction with the following paper:

• Gagandeep Panwar, Da Zhang, Yihan Pang, Mai Dahshan, Nathan DeBardeleben, Binoy Ravindran, and Xun Jian. 2019. Quantifying Memory Underutilization in HPC Systems and Using it to Improve Performance via Architecture Support. In Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO '52). ACM, New York, NY, USA, 821-835. DOI: doi.org/10.1145/3352460.3358267 (ACM Digital Library)

Overview

This entire collection of data has been released openly with the identifier LA-UR-19-28211.  The data consists of 87.4GBs of (uncompressed) 19,678 JSON files.  These JSON files contain memory usage statistics from three compute clusters at LANL: grizzly, badger, and snow.  The data is relatively fine-grained (every 10 seconds, where records exist) for every node from each cluster, and identifies which job ID (numerical) used that memory.  Since the job IDs are unique, one can see the parallel jobs running between multiple nodes.  For instance, job ID 123 running on nodes 100, 101, and 102 will have separate memory utilization records for all three nodes for the time it was running.

Each archive has a detailed README which explains the data format and the cluster sizes, memory system sizes, etc.  A brief sample of some trivial analysis from this data is at the bottom of the page.  For a more detailed analysis, we encourage you to see the paper referenced above.

Data

Samples

The following images are intended to give simple snapshots of the raw data.  They are not a scientific analysis but may be useful to give a reader an idea what the data holds.

Hosts

Here we simply look at hostnames reporting data.  This may be indicative of utilization of these hosts.  When combined with job and time information (in the dataset, but not shown here), more interesting and advanced analytics could be performed.  This first plot is from the grizzly0 dataset and we see most hosts reporting about the same number of records.

Metrics

As the README files explain, the datasets contain meminfo.memfree and meminfo.active samples from every node at periodic intervals.  Using this, one can get an idea of memory usage.  For instance, the following plot shows a quick glimpse at the meminfo.active metric for the snow dataset.

JobIDs

The datasets include memory usage data per node with the JobID consuming that memory.  Since these systems are used for parallel computation, one can see the amount of memory consumed by parallel jobs and determine how many nodes that job ran on.  The following plot shows a snapshot of the number of records in just the first hour of the grizzly1 dataset.  This doesn't state how much memory was used, nor how many nodes these jobs ran on, just the number of records in the dataset.  It is presented here (as mentioned earlier) to give the reader an idea of what is in the data.

NOTE: These data are historical in nature and were originally released in 2005.

In order to enable open computer science research, access to computer operational data is desperately needed. Data in the areas of failure, availability, usage, environment, performance, and workload characterization are some of the most desperately needed by computer science researchers. The following sets of data are provided under universal release to any computer science researcher to use to enable computer science work.

All we ask is that if you use these data in your research, please recognize Los Alamos National Laboratory for providing these data. 

All files and content available for download are covered by LA-URs listed in the filename. Each file is a tar.gz archive with a data file (either CSV or text file) and a README with some explanations.

Bianca Schroeder (at Carnegie Mellon University at the time) had been kind enough to provide a frequently asked questions (FAQ) document about the 1995–2005 failure data set. NOTE: These responses should be seen in the context of this dataset and in the time they were written.

All datasets reference anonymized "system numbers." To understand the failure, usage, and event info, you also have to understand the machine and/or system layout. For this, we have provided this Excel file. NOTE: Care needs to be taken when looking at the above data as some of the systems may appear to grow (or shrink) in size for short time periods. Sadly, these time periods are not well documented, but this occasionally occurred to put two systems together to perform a larger calculation than could be done on any one part. The machine and/or system layout data are a single snapshot in time and not a snapshot over time, so they may require other effort to determine these periods. Luckily, this likely only impacts certain types of studies, and the time periods were relatively short.

NOTE: These data are historical in nature and were originally released in 2012.

To enable computer science research, Los Alamos National Laboratory is releasing static file tree data (fsstats) for some of our paralle file systems. These data include aggregate information on capacity, file and directory sizes, filename lengths, link counts, etc. The fsstats cover 9 anonymous parallel file systems ranging from 16 TB to 439 TB totalcapacity used, and file counts range from 2,024,729 to 43,605,555.

The Machine System Number is explained more under the anonymous machine tab elsewhere on this page, including information such as the number of nodes, cores, memory size, and dates of use.

All file sizes are extremely small (approximately 15K) and uncompressed.

All files and content available for download are covered by document LA-UR-07-5769.

NOTE: These data are historical in nature and were originally released in 2007.

These data represent file system statistics gathered from backup information for 3000-plus LANL workstations. The following set of data is provided under universal release to any computer science researcher to use to enable computer science work.

The data are contained in a tar file containing three CSV files that represent the three primary types of workstations used at LANL (MAC, UNIX, Windows). Please refer to the README contained in the tar file for a description of the data gathered. All files and content available for download are covered bydocument LA-UR-07-5769.

Each archive contains a README explaining the data in more detail.

Frequently Asked Questions:

1. Why does version 3.0 have quite a few more nodes than previous versions?
   • Analysis of this run showed that approximately 80 nodes were not surveyed due to an internal problem that prevented the script from getting information on those nodes. Furthermore, approximately 300 nodes were removed from the backup system, and 340 new nodes were added. The net effect was a reduction in total nodes.
2. Why does version 2.0 have a fewer number of nodes (versus version 1.2), yet the number of files and total data size have grown?
   • The new nodes identified by this run accounted for much of the growth in terms of file counts and data size. Approximately 209 million files belonged to these new nodes, while the number of files related to the nodes no longer backed up was much lower. This resulted in a net gain of files.
3. Will anonymous node names and file space names persist over the life of this project?
   • Yes, the node name and filespace will be unique to that particular node over the life of this project. A future survey of the nodes will identify those nodes that have already been assigned an anonymous name, and that name (anonymous) will be retained.
4. Why does the inactive files histogram have min bucket and max bucket values of equal size?
   • This project re-used code that created histograms based on bucket ranges. Because this histogram is meant to show inactive files (1,2,3,4), a range is not necessary. This will be fixed in a future version. See the README contained in the tarball for more information.

NOTE: These data are historical in nature and were originally released in 2011.

These data represent a file system walk with information such as file sizes, creation time, modification time, UID/GID, etc. The data are highly anonymized and are released under document LA-UR-07-5769. Please see the README contained in the individual archives for a more detailed description of the data. The files are text and compressed, and the file size below represents the compressed size for download.

The Machine System Number is explained more under the anonymous machine tab elsewhere on this page, including information such as the number of nodes, cores, memory size, and dates of use.

All files and content available for download are covered by document LA-UR-07-5769.

Frequently Asked Questions:

1. What is the difference between global and non-shared NFS file systems?
   • The global NFS file system is shared among a number of different clusters. The non-shared NFS file systems are visible to only a single cluster.
2. Is the create time field really the time the file was created?
   • Yes. This field is not UNIX "ctime" (change time).
3. Are the user identifieer (UID)/group identifier (GID) or paths consistent across file systems?
   • No. No assumptions can be made across files. The paths within a single file maintain the file tree hierarchy.
4. I notice several instances where the GID is different from the UID, and multiple UIDs have the same GID. Is this right?
   • Yes, this is especially the case for the globally accessible file systems. Multiple UIDs having the same GID are usually users sharing files with a group and UNIX permissions. Local NFS file systems usually have a one-to-one mapping between UID and GID, but this is not required.
5. Is the blocksize field the actual blocksize used to store the file, or is it the size allocated to a file?
   • This field is for the size of each block allocated.
6. Are these file systems/archive mostly used by developers or users?
   • Most of the file systems and machines are open to both developers and users. The exception to this is machine 29, which is targeted solely to developers. Some developers also work on their desktop machines, so these files may not be representative of their entire workloads.