Los Alamos National LaboratoryCrossroads
A critical element for improved predictive capability.

Request for Proposal

Information for the successful bidder



Crossroads Request for Proposal (RFP) No. 511017

All proposals are due by 2:30 p.m. Mountain Time on Monday, March 18, 2019.

Interested parties are invited to submit a proposal for one (1) subcontract for delivery of the Crossroads supercomputer system. The subcontract may result from this RFP is in support of the New Mexico Alliance for Computing at Extreme Scale (ACES) composed of the following NNSA High Performance Computing organizations:

  • Los Alamos National Laboratory (LANL)
  • Sandia National Laboratories (SNL)

Interested parties are advised to monitor this website for potential Crossroads RFP amendments and other Crossroads RFP information updates. The Contract Administrator may notify interested parties of updated Crossroads RFP information via e-mail; however, there is no obligation to do so.

It is the responsibility of all interested parties to monitor this website for current Crossroads RFP information.

Interested parties must submit in writing all communication regarding the Crossroads RFP (questions, comments, etc.) to the Contract Administrator Maureen Armijo

Crossroads RFP Questions and Answers

Q1: The following question is regarding Section 3.3.10 of Exhibit D of the RFP Technical Requirements document:

"Atomic compare-and-swap operations over 16-byte wide operands that comprise two double precision values or two 64-bit memory pointer operands..." Is the comparison implied bit-wise equality? 

A1: The Crossroads team is interested in support for bit-wise equality comparisons of 16B contiguous blocks of data. Other operations are not required to be supported.

Q2: RFP Form B Schedules of Proposed Milestones and Payments discusses the anticipated funding stream for the Crossroads systems.  Does the Crossroads Project Team anticipate installing a portion of the system later in CY22 to match the FY23 funding stream?
A2: The funding stream as originally distributed was incorrect. The correct anticipated funding stream is:

Based on Percent of Total Funding















Q3: We noticed that there's mention of "The small business size standard for the listed NAICS code is 1,250 employees." Does this signal that the procurement is limited to small business (small business set aside) or is large business eligible to bid?
A3: This procurement is not small business set aside. The size standard for the listed for a supplier to determine if they are classified as Large or Small business based on weather the supplier has more or less than 1,250 employees.

Q4: Can the HPCG benchmark source code from git repo be used for characterization and projection instead of version 3.0 on the Crossroads web site? This version has optimizations beyond version 3.0 which permit significantly increased performance.
A4: The HPCG version provided on the Crossroads website must be used for the Base Results per the run rules provided on the Crossroads website and in the HPCG README.  The Offeror is permitted to use the HPCG source code from the git repo for the Optimized Results.  Please follow all submission guidelines for results submission. 

Q5: With respect to Section 3.5.9 of Exhibit D of the RFP Technical Requirements document; regarding MPI_Allreduce and MPI_Allgather Performance: How many MPI tasks per node should be used?
A5: The Offeror should perform the MPI_Allreduce and MPI_Allgather using as many MPI tasks as there are processor cores, not counting finer grain processing units like hyper threads.

Q6: With respect to Section 3.5.12 of Exhibit D regarding MPI_THREAD_MULTIPLE: Is the intended performance comparison here MPI_THREAD_MULTIPLE with a single thread versus MPI_THREAD_SINGLE?
A6: The Offeror should present results comparing MPI_THREAD_MULTIPLE and MPI_THREAD_SINGLE for a single thread, and is encouraged to present results for larger numbers of threads using the results from section 3.5.7 and 3.5.8 to demonstrate an efficient implementation of MPI_THREAD_MULTIPLE. 

Q7: With respect to Section 3.5.16 of Exhibit D of the RFP Technical Requirements document; which states: The Offeror shall describe how the system may be configured to support a high rate and bandwidth of TCP/IP connections to external services both from compute nodes and directly to and from the platform storage, including:

  • Q7.1:The need to provide 100-300GB/s data flow from the platform storage to and from TCP/IP connections is understood.  The wording, “support a high rate of bandwidth of TCP/IP connections to external services both from compute nodes and directly to and from the platform storage” is confusing.  Does this imply that compute nodes need to provide high bandwidth TCP/IP connections to external services?
  • Q.7.2: “Compute node external access should allow all nodes to each initiate 1 connection concurrently within a 1 second window.” Should the above bullet say “Platform storage” instead of “Compute node” or does this bullet imply that external nodes need to connect to compute nodes directly?
  • Q7.3: Transfer of data over the external network to and from the compute nodes and platform storage at 100 GB/s per direction of a 1 TB dataset comprised of 20 GB files in 10 seconds. This statement also states that data can be transferred to and from the compute nodes from the external network.  Is that intended?
A7.1: Compute nodes will need to access external services like NFS for a variety of data input/output in addition to the “Platform Storage”.   It is also feasible that Compute nodes will also need to move data to visualization servers, etc via this connection.
A7.2: Yes, see A7.1
A7.3: Yes, see A7.1. 

Q8: In the Summary file for Branson under 'Benchmark results', explicit formulas for the total memory per node are given. For the 'Particle Memory' and 'Mesh Memory' formulas, should 'n ranks per node' be 'number of nodes'. For the ‘Buffer Memory' formula, should the multiplier be 'n ranks per node' or something else?
A8: Yes, for ‘Particle Memory’ and ‘Mesh Memory’, the quantity should be ‘number of nodes’ instead of ‘n ranks per node’. For the ‘Buffer Memory’ calculation, the quantity is correct in the summary. Each rank allocates its own buffers and handles its own MPI messages. The Branson Summary file will be updated.

Q9: Could the Crossroads Project team explicitly show the calculation for the 50TB number for the large run given for Trinity under the “Benchmark Results in the Branson summary document using the total memory formulas per node in that document?
A9: To calculate the total memory used for the large run using the formulas provided in the Branson summary document:

  • particle: (331776000000 total particles* 88.0 bytes/particles) / 1.0e12 = 29.196 TB
  • mesh: (5.12e8 mesh cells * 1000 bytes/cell) / 1.0e12 = 0.512 TB
  • buffers: (500 grip size * 12 grips/buffer * 6 buffers + 50000 cell map size) * 110592 ranks * 1000.0 bytes/cell = 9.51 TB
  • total: 39.218 TB
NOTE: The Branson summary document provides formulas for total memory per node, whereas the above calculation shows the total memory for this run.

Q10: For Branson, the grip size in the proxy_large.xml input is 500. The Branson summary document mentions a grip size of 10000. Should the Offeror use a grip size of 500 or 10000 for the baseline run?
A10: The Offeror should use the grip size of 500 as is given in the input file. The advice of “10000” applies to more run cases (between 256-2048 cores).

Q11: For UMT, the runs on the final system must be performed with “total memory for data of at least 344,000 GB”. Is it sufficient that the total memory used by the binary (across all ranks) should be greater than 344,000 GB? Or should the memory for data be calculated according to the description in Section 7.2 (Memory Considerations & Use) of the UMT Summary document for Crossraods"?
A11: The Crossroads Project team would like the amount of memory consumed for the run to exceed 344,000 GB. This can include the executables and any appropriate system libraries.

Q12: Also, for UMT, if the calculation in Section 7.2 of the Summary file should be used, then it appears that the 125,000 rank (3907 node) baseline case is not sufficiently large, since it uses 336.08 TB (= 344,146 GB) of total memory. However, not all of this memory is used for data – and the calculations in Section 7.2 give 1.6 GB per MPI rank for numzones = (4,4,4), which is a total of 202,750 GB for data. Is this correct?
A12: See Response 11. The Crossroads Project team would like the total amount of memory consumed to exceed 344,000 GB, with the includes of the executables and appropriate system libraries.

Q13: The Branson benchmark requires the use of ParMETIS. However, the licensing for ParMETIS states:
It can be freely used for educational and research purposes by non-profit institutions and US government agencies only. Other organizations are allowed to use ParMETIS only for evaluation purposes, and any further uses will require prior approval. Are alternatives permitted?
A13: Two source files for Branson are available. One uses ParMETIS and the other uses METIS.

  • ParMETIS version: branson-xroads-v1.0.0.tgz on Crossroads website (tag 0.81)
  • Metis version: branston-0.82.tar.gz on Crossroads website or clone from github using commit 8b42c3caa37d563a27a85519522f236dfd8fd159 (tag 0.82)

The Trinity FOM was run using the ParMETIS version of Branson (tag 0.81). The Offeror may use either version of Branson described above to provide the FOM and should specify the version used in the proposal. The Offeror commits to using the same version in the final statement of work.

Q14: With respect to Section 3.5.9 of Exhibit D of the RFP Technical Requirements document states:

The Offeror shall report, or project, the time to perform the following collective operations for 25%, 50%, and 100% of the compute partition nodes in the system and report on core occupancy during the operations measured by the communication benchmark specified on the Crossroads website for:

  • An 8 byte MPI_Allreduce operation.
  • An 8 byte per rank MPI_Allgather operation.

The MPI source on the benchmark site does not appear to have such collectives.  Will the Crossroads Project team be updating the tarball or will the Offeror be allowed to use something else?
A14: The Offeror is being asked to project the time required for the collective calls listed. The projected time provided in the response can be demonstrated using any benchmark selected by the Offeror, but the APIs to start the call must be compliant with the MPI standard and be reproducible. Any source code used must be supplied during acceptance.

Q15: Per the Instructions to Offerers, Section 3D requires the two following documents to be included in the Commercial Proposal Volume: 1) RFP Form B, Schedule of Proposed Quantities and Prices, 2) Exhibit C Form B, Milestone and Payment Schedule. This vendor is able to locate the RFP Form B, Schedule of Proposed Quantities and Prices, but cannot locate an Exhibit C Form B Milestone and Payment Schedule. Please advise if there are actually two documents or if they are referring to the same form.
A15: There are two different forms referenced that need to be returned. RFP Form A, Schedule of Proposed Quantities and Prices and RFP Form B Schedule of Proposed Quantities and Prices. These two forms will be used to create the Exhibit C Form A Schedule of Quantities and Prices and Exhibit C Form B Milestone Payment Schedule. A SAMPLE of these forms that will be created are in the Sample build contract. These forms titled, "Exhibit C" do not need to be returned with the RFP, they are for reference only.

Q16: Can an Extension be granted. Additional time would be appreciated. 
A16: After consideration, LANL will grant an extension. RFP Responses will be due April 8, 2019 at 2:30.