Los Alamos National LaboratoryBEE: Build and Execution Environment
A unified software stack to containerize HPC applications

BEE: Build and Execution Environment

High-performance computing (HPC) systems have become critical infrastructure for science and industry. However, legacy applications and variations in system configurations mean that applications must be configured and built for specific HPC environments. Building multiple environments requires a significant investment of time and effort or additional technical knowledge. To simplify this, we’ve created a lightweight, portable environment to run on almost any current HPC system. We call it BEE: Build and Execution Environment.

The goal of BEE is to create a unified software stack to containerize HPC applications into a package of code (usually binaries) and all of that code's dependencies (libraries, etc.). Once built, the container can be run on many different platforms. The execution environment on each platform will download and install (for this application only) all of the applications and dependencies into an isolated user environment and then execute the code.

Benefits

BEE provides many benefits:

  • Flexibility. BEE provides the flexibility to choose different platforms as needs change and migrates application states (via checkpoints) to different platforms. Users can choose their own software stack (libraries, compilers, etc.) and not be bound by the installed environment on any one machine.
  • Portability. The execution framework runs across multiple platforms, including HPC systems and cloud computing systems. All dependencies are downloaded and installed at run time.
  • Automation. The deployment of the container-enabled execution environment and the user’s application can be done automatically.
  • Reproducibility. Applications behave the same across platforms regardless of the host software configurations.
  • Interfacing Continuous Integration. The execution framework supports integration with commonly used Continuous Integration (CI) workflows in scientific application development so that it can effectively free users from handling complicated application compilation.

Details

The BEE project uses Docker to containerize applications. Docker has become the de facto standard container system and is used widely in cloud and web environments. Continuous integration services have been built on Docker, allowing application developers to describe compile and execution environments with Docker. When code is checked into a repository, it can be automatically tested across a suite of different software environments.

BEE uses both virtual machines (VMs) and software containers to create a workflow system that establishes a uniform build and execution environment beyond the capabilities of current systems. In this environment, applications will run reliably and consistently across heterogeneous hardware and software. Containers define the runtime that isolates all software dependency from the machine operating system. Workflows may contain multiple containers that run different operating systems, different software, and even different versions of the same software. Containers are placed in open-source virtual machine (Kernel-based Virtual Machine [KVM]) and emulators (Quick Emulator [QEMU]) so that workflows run on any machine entirely in user-space.

To download the current release, visit the BEE GitHub repository: www.github.com/lanl/BEE.