Exascale | Kalyan Perumalla

Exascale Computing for the National Energy Grid

Sat, 01 Jan 2022 00:00:00 +0000

Exascale Computing for the Stochastic Grid Dynamics of the US national energy transmission network is a part of the Exascale Computing Project to tap the world’s largest supercomputer to solve the nation’s energy grid problems.

Exascale Computing Project

Overview

The Exascale Computing Project that enables US revolutions in technology development: scientific Discovery; health care; and energy, economic, and national security. Exascale computing will provide the capability to tackle challenges at levels of complexity and performance that previously were out of reach.

The Problem

How can we secure the national energy transmission network despite unexpected cyber attacks, natural disasters, and unpredictable load fluctuations?
How fast and far into the future can we make it resilient?
How can we ultimately achieve this at the least cost to everyone?

The Solution

Enter Exascale computing and Global Optimization.

This project attacks this nationally important problem with an unprecedented high-technology approach that relies on supercomputing to work out the very best global solutions on-the-fly, rapidly examining millions of configurations involving all the energy generators, transmission lines, event contingencies, and complex physical constraints. Very high-end algorithms implemented with sophisticated parallel processing software is designed for the state-of-the-art supercomputing hardware, combining the know-how of some of the very best minds across the US national laboratory systems, tapping a range of experts in energy domain sciences, computational optimization, numerical algorithms, high-end computing hardware, and advanced software stack organization techniques.

Organization

Sponsor: US Department of Energy (DOE)
- Office: Advanced Scientific Computing Research (ASCR)
- Program: Exascale Computing Project (ECP)
Institutions: PNNL, ORNL, ANL, LLNL, NREL
Period: 2019-2023/24

Abbreviations

PNNL = Pacific Northwest National Laboratory
ORNL = Oak Ridge National Laboratory
ANL = Argonne National Laboratory
LLNL = Lawrence Livermore National Laboratory
NREL = National Renewable Energy Laboratory
PI = Principal Investigator

Gallery

ReveR-SES: Reversible Software Execution Systems

Sat, 01 Jan 2022 00:00:00 +0000

Reversible Software Execution Systems ReveR-SES is a paradigm shift in ultra-scale computing to address the outstanding scaling challenges by enabling an entirely new, orthogonal dimension to all aspects of traditional, forward-only computing.

ReveR-SES

Overview

ReveR-SES not only provides expeditious, novel solutions to scaling problems, but also opens new, longer-term research and development directions in high performance computing.

All traditional computing is done forward-only, but never in reverse order. Only recently it has been discovered that executing codes backwards can be used to greatly increasing the efficiency and usability of high performance computing. However, rendering a program reversible is an extremely challenging endeavor. This project is focused on developing the methodologies to exploit reversibility to enable scaling applications to ultra-scale platforms with 1,000,000 processor cores.

ReveR-SES contains several novel ideas, including reversible compilers, reversible libraries, reversibility extensions to standard interfaces, as well as relation to thermodynamics, information and entropy.

Quad chart

ReveR-SES Confluence

The reversible execution paradigm represents an entirely new research direction, yet, has immediate relevance to existing DOE applications and also the potential for creation of entirely new technologies and the creation of new high-technology jobs in computing. This is due to the orthogonality of reversibility to many existing HPC dimensions.

ReveR-SES Components

Overall, the new execution paradigm provides energy savings in the short-term and better energy-efficient designs in the longer term. The reversible execution systems are directly relevant to important applications such as climate, plasma physics, and materials science simulations.

ReveR-SES was Dr. Kalyan Perumalla’s Early Career Research project 2010-2015, awarded as a single-principal investigator, $2.5 million project.

https://science.osti.gov/ascr/Community-Resources/ECRP-Awardees

Scope

To define, develop, test, and implement the paradigm of reversible software execution for exascale computing.

Primary Scientific Thrusts

Developing fundamentally new reversible computer arithmetic and logic
Designing efficient asynchronous rollback-based recovery via reversible execution
Redesigning traditional physical system models to enable reversible simulation
Reevaluating theoretical computational and energy consumption interplay.

Science Impacts

Enables new capabilities for computational science common to many areas

Extremely efficient method for fault tolerant simulations on next generation heterogeneous (CPU+GPU) systems
Solves the synchronization problem at very large scales of concurrency
Overcomes undesirable reliance on memory
- Addresses the exascale hardware problem of high ratio of computational speed to memory speed
- Computational energy reduced via reduced memory footprint
Provides the most promising approach to debugging at exascale.

Enables Theoretical Advancements

Directly relates energy of computation to computational model characteristics
Positions scientific simulations for future reversible computing hardware (adiabatic circuits, Quantum Computing).

Gallery

Organization

Sponsor: US Department of Energy (DOE)
- Office: Advanced Scientific Computing Research (ASCR)
- Program: Early Career Research Program (ECRP)
Prime: Oak Ridge National Laboratory (ORNL)

Selected Publications

Normalcy, Magic, Miracle and Error: Emergence along a Reversibility Spectrum

Formation of a butterfly from a pupa, extraction of a live dove from a magician’s empty hat, generation of new particles from high-energy particle collisions and spawning a new dream world from mind in sleep are all examples of a common, fuzzy notion called ‘emergence’. In this paper, I pin the concept of emergence to the element of surprise in a phenomenon. I categorise the various notions of emergence into three main classes. These definitions are used to explain instances of emergence, organised along a continuous spectrum as normality, magic, miracle and error.

Kalyan Perumalla

PDF Cite

Efficient reversible uniform and non-uniform random number generation in UNU.RAN

Reversible random number generations are useful in large-scale fault-tolerant parallel computations and parallel discrete event …

Srikanth Yoginath, Kalyan Perumalla

PDF Cite

Towards Reversible Basic Linear Algebra Subprograms: A Performance Study

Problems such as fault tolerance and scalable synchronization can be efficiently solved using reversibility of applications…

Kalyan Perumalla, Srikanth Yoginath

PDF Cite

Reverse Computation for Rollback-based Fault Tolerance in Large Parallel Systems

Reverse computation is presented here as an important future direction in addressing the challenge o…

Kalyan Perumalla, Alfred Park

PDF Cite

Tutorial: Introduction to Reversible Computing

This tutorial provides an introduction to the concept of reversible computing, adopting an expanded view…

Kalyan Perumalla

PDF Cite Slides

Reversibly Finding the Square Root of an Integer

Here we consider a case study in reversible computing, namely, how to reversibly compute the integer square root …

Melissa Yu, Kalyan Perumalla

PDF Slides

Reversible Simulations of Elastic Collisions (Extended arXiv version)

Consider a system of N identical hard spherical particles moving in a d-dimensional box and undergoing elastic, possibly multi-particle, collisions…

Kalyan Perumalla, Vladimir A. Protopopescu

PDF Cite

Reversible Simulations of Elastic Collisions

Consider a system of N identical hard spherical particles moving in a d-dimensional box and undergoi…

Kalyan Perumalla, Vladimir A. Protopopescu

PDF Cite

Introduction to Reversible Computing

Few books comprehensively cover the software and programming aspects of reversible computing. Fillin…

Kalyan Perumalla

PDF Cite

ReveR-SES: Reversible Software Execution Systems

[Pub 137]

Kalyan Perumalla

Cite

Discrete Event Execution and Reversibility: Challenges in the Path to Asynchrony for Massively Parallel Computing

To keep up with the increasing number of processing elements in parallel/distributed computing, traditional tightly-coupled time-stepped models…

Kalyan Perumalla

Cite Slides

Reversible Parallel Discrete Event Formulation of a TLM-based Radio Signal Propagation Model

Radio signal strength estimation is essential in many applications, including the design of military…

Sudip Seal, Kalyan Perumalla

PDF Cite

ReveR-SES: Reversible Software Execution Systems

[Pub 136] http://science.energy.gov/ascr/ascac/meetings/nov-2011/

Kalyan Perumalla

Cite

Reversible Parallel Discrete-Event Execution of Large-scale Epidemic Outbreak Models

The spatial scale, runtime speed, and behavioral detail of epidemic outbreak simulations altogether …

Kalyan Perumalla, Sudip Seal

PDF Cite Slides

Compiler-based Automation Approaches to Reverse Computation

Automation is useful to facilitate reverse code generation from normal code. Here, we describe our …

Kalyan Perumalla, Christopher Carothers

Cite Slides

Reversible Discrete Event Formulation and Optimistic Parallel Execution of Vehicular Traffic Models

Vehicular traffic simulations are useful in applications such as emergency planning and traffic mana…

Kalyan Perumalla, Srikanth Yoginath

PDF Cite

Perfect Reversal of Rejection Sampling Methods for First-Passage-Time and Similar Probability Distributions

We present a perfectly reversible method for bi-directional generation of samples from computational…

Kalyan Perumalla, Aleksandar Donev

PDF Cite

EpiClone: Epidemiological Clonable Simulations

Sat, 01 Jan 2022 00:00:00 +0000

Cloned simulations of epidemiological outbreaks like COVID-19 can now rapidly evaluate millions of what-if scenarios at national and world scales, and efficiently exploit 1000s of GPUs.

EpiClone model elements

EpiClone simulating USA scenarios

Overview

EpiClone represents the next advancements in what-if decision evaluation on state-of-the-art accelerated computing platforms including supercomputers containing thousands of GPUs.

EpiClone addresses global epidemiological outbreaks via an incremental simulation-based what-if decision tree evolution. EpiClone offers new scaling capabilities that were previously not possible before for rapidly simulating thousands or millions of epidemiological scenarios.

The EpiClone approach and results in scalable modeling and fast scenario exploration provide a leap in the analyses of epidemics and other complex systems.

Gallery

Organization

Sponsor: Lab Directed Research and Development, Oak Ridge National Laboratory
Period: 2015-2017

Scalable Cloning on Large-Scale GPU Platforms with Application to Time-Stepped Simulations on Grids

Cloning is a technique to efficiently simulate a tree of multiple what-if scenarios that are unraveled during the course of a base simulation. We present the conceptual simulation framework, algorithmic foundations, and runtime interface of CloneX, a new system we designed for scalable simulation cloning.

Srikanth Yoginath, Kalyan Perumalla

PDF Cite DOI

Mesoscopic Modeling and Rapid Simulation of Incremental Changes in Epidemic Scenarios on GPUs: Fast What–If Analyses of Localized and Dynamic Effects

A mesoscopic modeling approach is described that strikes a middle ground between macroscopic models based on coupled differential equations and microscopic models built on fine-grained behaviors at the individual entity level. Execution of our implementation scaled to 8192 GPUs of supercomputing platforms demonstrates the ability to rapidly evaluate what–if scenarios several orders of magnitude faster than the conventional methods.

Kalyan Perumalla, Maksudul Alam

PDF Cite DOI

Discrete Event Modeling and Massively Parallel Execution of Epidemic Outbreak Phenomena

In complex phenomena such as epidemiological outbreaks, the intensity of inherent feedback effects a…

Kalyan Perumalla, Sudip Seal

PDF Cite

Reversible Parallel Discrete-Event Execution of Large-scale Epidemic Outbreak Models

The spatial scale, runtime speed, and behavioral detail of epidemic outbreak simulations altogether …

Kalyan Perumalla, Sudip Seal

PDF Cite Slides

CloneX

Sat, 01 Jan 2022 00:00:00 +0000

Cloned eXecution of simulations is a powerful decision-making system to incrementally evaluate millions of what-if scenarios and scale to 1000s of GPUs.

Illustration of CloneX what-if tree on diffusion processes

CloneX computational and memory savings with increasing tree depth

Overview

CloneX is a novel system we designed for scalable simulation cloning, consisting of a conceptual simulation framework, algorithmic foundations, and a highly scalable runtime interface. It efficiently and dynamically creates whole logical copies of a dynamic tree of simulations across a large parallel system without full physical duplication of computation and memory.

CloneX Software Architecture

Titan Supercomputer with 1000s of GPUs

CloneX efficiently simulates a tree of multiple what-if scenarios unraveled during the course of a normal (base) simulation. Cloned execution is highly challenging to realize on large, distributed memory computing platforms, due to the dynamic nature of the computational load across clones, and due to the complex dependencies spanning the clone tree.

CloneX has been tested on 1000s of GPUs of a supercomputing system and evaluated with multiple benchmarks – such as heat diffusion, forest fire, and disease propagation models – delivering a speed up of over two orders of magnitude compared to replicated runs.

CloneX represents a major leap in ensemble simulations as a significantly faster and scalable way to execute many what-if scenarios of large simulations.

Organization

Sponsors: US Department of Energy (DOE)
- Office: Advanced Scientific Computing Research (ASCR)
  - Program: Early Career Research Program (ECRP)
- Office: ORNL Strategic Planning Office
  - Program: Laboratory-Directed Research and Develoopment (LDRD)
Period: 2015-2017

Gallery

Mesoscopic Modeling and Rapid Simulation of Incremental Changes in Epidemic Scenarios on GPUs: Fast What–If Analyses of Localized and Dynamic Effects

Kalyan Perumalla, Maksudul Alam

PDF Cite DOI

Scalable Cloning on Large-Scale GPU Platforms with Application to Time-Stepped Simulations on Grids

Exascale | Kalyan Perumalla

Exascale Computing for the National Energy Grid

Overview

The Problem

The Solution

Organization

Gallery

Related Publications

ReveR-SES: Reversible Software Execution Systems

Overview

Scope

Primary Scientific Thrusts

Science Impacts

Gallery

Organization

Selected Publications

EpiClone: Epidemiological Clonable Simulations

Overview

Gallery

Organization

Related Publications

CloneX

Overview

Organization

Gallery

Related Publications

ExaSGD Performance Profiling