Project | Kalyan Perumalla

DarkNet Cyber Resilience

Tue, 22 Mar 2022 00:00:00 +0000

DarkNet Cyber Resilience investigates the impacts of cyber stress and its thresholds of impact on alternative timing signals for the US national energy grid.

Cyber Resilience of Alternative Timing for Energy Grids

Overview

This project is focused on performing a systematic evaluation of the impacts of cyber stress on the system. It is developing tools that enable early detection of performance issues and rapid recovery from cyber-related events. A unique advancement of the project lies in going beyond general cybersecurity vulnerabilities and exposures (CVE). This is achieved by evaluating their implications to the cyber resilience specifically with regard to precise effects on the transmission and receipt of timing signals over the complex networks connecting the timing sources to the geographically-distant subscribers.

Key Links

DarkNet Website

Gallery

Organization

Sponsor: US Department of Energy (DOE)
- Office: Office of Electricity (OE)
- Program: DarkNet
Prime: Oak Ridge National Laboratory (ORNL)

Selected Publications

DarkNet Cyber Resilience

Two key cyber elements affecting the grid’s timing capability are Cyber Resilience and Cyber Trust. The timing services of DarkNet are systematically subjected to a range of cyber phenomena that stress four key performance factors, namely: Accuracy, Manageability, Telemetry, and Visibility. This analysis is designed to provide insights into four important categories of undesirable cyber phenomena: Loss of View (LoV), Loss of Control (LoC), Manipulation of View (MoV), and Manipulation of Control (MoC).

Kalyan Perumalla, Juan Lopez

PDF

CYVET

Sat, 01 Jan 2022 00:00:00 +0000

Our new Cyber-Physical Security Assurance Framework based on Semi-Supervised Vetting applies the latest AI/ML and NLP technologies on hardware testbeds to advance the resilience of critical cyber-physical assets including electric grids and gas pipelines.

CYVET Pipeline

Overview

Organization

Sponsor: US Department of Energy (DOE)
- Office: Cybersecurity, Energy Security, and Emergency Response (CESER)
- Program: Cybersecurity for Energy Delivery Systems (CEDS)
- Award: CESER
Prime: Oak Ridge National Laboratory (ORNL)
- Subcontract: University of Nebraska-Lincoln (UNL)
Period: 2019-2023

Gallery

OSTI.gov: https://www.osti.gov/servlets/purl/1661247

CyBERT: Cybersecurity Claim Classification by Fine-Tuning the BERT Language Model

We introduce CyBERT, a cybersecurity feature claims classifier based on bidirectional encoder representations from transformers and a key component in our semi-automated cybersecurity vetting for industrial control systems (ICS)…The results showed that CyBERT outperforms these models on the validation accuracy and the F1 score, validating CyBERT’s robustness and accuracy as a cybersecurity feature claims classifier.

Kimia Ameri, Michael Hempel, Hamid Sharif, Juan Lopez, Kalyan Perumalla

PDF Cite DOI

Trust-but-Verify in Cyber-Physical Systems

Cyber-physical systems span a wide spectrum, from long-lived legacy systems to more modern installations. Trust is an issue that arises across the spectrum, albeit with different variants of goals and constraints. On the one end of the spectrum, legacy systems are characterized by function-based designs in which trust is an implicitly in-built concept…

Kalyan Perumalla

PDF Cite Slides DOI

Smart Semi-Supervised Accumulation of Large Repositories for Industrial Control Systems Device Information

A solution is needed for vetting the vendor-supplied feature claims and their adherence to cybersecurity requirements and standards. We are presently engaged in an effort to develop such a system. This paper demonstrates one vital aspect of this effort in proposing an end-to-end framework to accumulate a large repository of ICS device information for this vetting system, curate the dataset, and conduct extensive processing. This framework is designed to use web scraping, data analytics and Natural Language Processing (NLP) techniques to identify vendor websites, automate the collection of website-accessible documents and automatically derive metadata from them for identification of product documents relevant to the repository…

Kimia Ameri, Michael Hempel, Hamid Sharif, Juan Lopez, Kalyan Perumalla

PDF

A Novel Vetting Approach to Cybersecurity Verification in Energy Grid Systems

The cybersecurity auditing for Operation Technology is critical and has been largely missing from the cybersecurity research, especially in the energy sector. In this paper, we present a novel “cybersecurity vetting” approach (CYVET) to the problem of verification and validation of cybersecurity in complex cyber-physical installations underlying modern energy grid systems.

Kalyan Perumalla, Juan Lopez, Maksudul Alam, Olivera Kotevska, Michael Hempel, Hamid Sharif

PDF DOI

Additional Background

The cybersecurity auditing for Operation Technology (OT) is critical and has been largely missing from the cybersecurity research, especially in the energy sector. CYVET is a novel “cybersecurity vetting” approach (CYVET) to the problem of verification and validation of cybersecurity in complex cyber-physical installations underlying modern energy grid systems.

In Information Technology (IT), cybersecurity auditing is a widespread practice to ensure privacy, security, and trust. However, for the field of Operation Technology (OT) as used in electric energy systems, this is a relatively novel concept. In fact, OT itself only recently began to embrace IT principles, with the push for automation and centralized control driving this development. OT operators are simply not yet used to the idea of cybersecurity. To ameliorate the gap, product vendors for field devices are advancing the field by incorporating more and more security features into their products. However, customers are often either unaware of them, or do not use them, or cannot use them because of unsatisfied device ecosystem dependencies. There is thus a disconnect between what is offered, what is possible post-deployment, and what the customer expects.

There is a vast lack of cybersecurity oversight and insight, from a certification and a customer perspective alike, for OT systems in the energy sector. With new features constantly being added to new and existing products, customers are predominantly unaware what their purchased solutions are capable of, or not capable of. They often do not know if their current systems meet their own cybersecurity requirements as well as industry standards. Many of these facets not only indirectly depend on device capabilities, but also on device deployment decisions – Does a newly added feature work in an existing context? Can it be used as envisioned? Does it interfere with other cybersecurity requirements? Does it produce side effects that may interfere with other requirements?

Hence, what is needed is a security vetting system designed to provide insight into deployed systems, the match of capabilities to requirements, adherence to certification requirements, and so forth. There are few systems currently available that provide these energy grid security capabilities. OT systems are increasingly cyber-enabled, increasingly complex, and increasingly interdependent. This rapidly accelerating trend poses a clear risk for asset owners to lose confidence and for cybersecurity risk to go undiscovered until exploited by malicious parties.

Deep CYBERIA

Sat, 01 Jan 2022 00:00:00 +0000

Deep CYBERIA is a novel system focused on Detecting Sensors Deeply Embedded in Cyber-Physical Systems via novel machine learning and passive/active/hybrid probing techniques on complex operational technology (OT) networks.

Overview

Deep CYBERIA is designed to address the critical capability gap identified by USAF in discovering, identifying and mapping the edge devices and physical sensors connected to those edge devices. The effort is aimed at the outcome of providing situational awareness of Industrial Control Systems/Supervisory Control and Data Acquisition (ICS/SCADA) traffic and devices operating in a network. The situational awareness will support the requirement of performing deep dive filtering analysis and enumeration of ICS traffic.

Demonstration

Organization

Sponsor: US Department of Defense
Prime: Oak Ridge National Laboratory (ORNL)
- Subcontract: MIT Lincoln Laboratory (MIT-LL)
Period: 2019-2025

Gallery

Trust-but-Verify in Cyber-Physical Systems

Kalyan Perumalla

PDF Cite Slides DOI

Detecting Sensors and Inferring their Relations at Level-0 in Industrial Cyber-Physical Systems

In this paper, we present our research and development efforts aimed at addressing the gap in discovering sensors at level 0 in industrial CPS by building a system called Deep-cyberia (Deep Cyber-Physical System Interrogation and Analysis) that incorporates algorithms and interfaces aimed at uncovering sensors and computing estimates of correlations among them.

Kalyan Perumalla, Srikanth Yoginath, Juan Lopez

PDF Cite Slides DOI

Volatile Memory Extraction-Based Approach for Level 0-1 CPS Forensics

Our focus is to extract volatile and dynamically changing internal information form CPS 0-1 level devices, and design preliminary schemes to exploit that extracted information. As a case study, we apply the proposed methodology to Modicon PLC using Modbus protocol. We extract the memory layout and subject the device to read operations at the most critical regions of memory. This capability of generating a sequence of volatile memory snapshots for offline, detailed and sophisticated analysis opens a new class of cyber security schemes for CPS forensic analysis, taint analysis and watermarking.

Rima Asmar Awad, Juan Lopez, Michael Rogers, Kalyan Perumalla

PDF Cite

Exascale Computing for the National Energy Grid

Sat, 01 Jan 2022 00:00:00 +0000

Exascale Computing for the Stochastic Grid Dynamics of the US national energy transmission network is a part of the Exascale Computing Project to tap the world’s largest supercomputer to solve the nation’s energy grid problems.

Exascale Computing Project

Overview

The Exascale Computing Project that enables US revolutions in technology development: scientific Discovery; health care; and energy, economic, and national security. Exascale computing will provide the capability to tackle challenges at levels of complexity and performance that previously were out of reach.

The Problem

How can we secure the national energy transmission network despite unexpected cyber attacks, natural disasters, and unpredictable load fluctuations?
How fast and far into the future can we make it resilient?
How can we ultimately achieve this at the least cost to everyone?

The Solution

Enter Exascale computing and Global Optimization.

This project attacks this nationally important problem with an unprecedented high-technology approach that relies on supercomputing to work out the very best global solutions on-the-fly, rapidly examining millions of configurations involving all the energy generators, transmission lines, event contingencies, and complex physical constraints. Very high-end algorithms implemented with sophisticated parallel processing software is designed for the state-of-the-art supercomputing hardware, combining the know-how of some of the very best minds across the US national laboratory systems, tapping a range of experts in energy domain sciences, computational optimization, numerical algorithms, high-end computing hardware, and advanced software stack organization techniques.

Organization

Sponsor: US Department of Energy (DOE)
- Office: Advanced Scientific Computing Research (ASCR)
- Program: Exascale Computing Project (ECP)
Institutions: PNNL, ORNL, ANL, LLNL, NREL
Period: 2019-2023/24

Abbreviations

PNNL = Pacific Northwest National Laboratory
ORNL = Oak Ridge National Laboratory
ANL = Argonne National Laboratory
LLNL = Lawrence Livermore National Laboratory
NREL = National Renewable Energy Laboratory
PI = Principal Investigator

Gallery

ReveR-SES: Reversible Software Execution Systems

Sat, 01 Jan 2022 00:00:00 +0000

Reversible Software Execution Systems ReveR-SES is a paradigm shift in ultra-scale computing to address the outstanding scaling challenges by enabling an entirely new, orthogonal dimension to all aspects of traditional, forward-only computing.

ReveR-SES

Overview

ReveR-SES not only provides expeditious, novel solutions to scaling problems, but also opens new, longer-term research and development directions in high performance computing.

All traditional computing is done forward-only, but never in reverse order. Only recently it has been discovered that executing codes backwards can be used to greatly increasing the efficiency and usability of high performance computing. However, rendering a program reversible is an extremely challenging endeavor. This project is focused on developing the methodologies to exploit reversibility to enable scaling applications to ultra-scale platforms with 1,000,000 processor cores.

ReveR-SES contains several novel ideas, including reversible compilers, reversible libraries, reversibility extensions to standard interfaces, as well as relation to thermodynamics, information and entropy.

Quad chart

ReveR-SES Confluence

The reversible execution paradigm represents an entirely new research direction, yet, has immediate relevance to existing DOE applications and also the potential for creation of entirely new technologies and the creation of new high-technology jobs in computing. This is due to the orthogonality of reversibility to many existing HPC dimensions.

ReveR-SES Components

Overall, the new execution paradigm provides energy savings in the short-term and better energy-efficient designs in the longer term. The reversible execution systems are directly relevant to important applications such as climate, plasma physics, and materials science simulations.

ReveR-SES was Dr. Kalyan Perumalla’s Early Career Research project 2010-2015, awarded as a single-principal investigator, $2.5 million project.

https://science.osti.gov/ascr/Community-Resources/ECRP-Awardees

Scope

To define, develop, test, and implement the paradigm of reversible software execution for exascale computing.

Primary Scientific Thrusts

Developing fundamentally new reversible computer arithmetic and logic
Designing efficient asynchronous rollback-based recovery via reversible execution
Redesigning traditional physical system models to enable reversible simulation
Reevaluating theoretical computational and energy consumption interplay.

Science Impacts

Enables new capabilities for computational science common to many areas

Extremely efficient method for fault tolerant simulations on next generation heterogeneous (CPU+GPU) systems
Solves the synchronization problem at very large scales of concurrency
Overcomes undesirable reliance on memory
- Addresses the exascale hardware problem of high ratio of computational speed to memory speed
- Computational energy reduced via reduced memory footprint
Provides the most promising approach to debugging at exascale.

Enables Theoretical Advancements

Directly relates energy of computation to computational model characteristics
Positions scientific simulations for future reversible computing hardware (adiabatic circuits, Quantum Computing).

Gallery

Organization

Sponsor: US Department of Energy (DOE)
- Office: Advanced Scientific Computing Research (ASCR)
- Program: Early Career Research Program (ECRP)
Prime: Oak Ridge National Laboratory (ORNL)

Selected Publications

Normalcy, Magic, Miracle and Error: Emergence along a Reversibility Spectrum

Formation of a butterfly from a pupa, extraction of a live dove from a magician’s empty hat, generation of new particles from high-energy particle collisions and spawning a new dream world from mind in sleep are all examples of a common, fuzzy notion called ‘emergence’. In this paper, I pin the concept of emergence to the element of surprise in a phenomenon. I categorise the various notions of emergence into three main classes. These definitions are used to explain instances of emergence, organised along a continuous spectrum as normality, magic, miracle and error.

Kalyan Perumalla

PDF Cite

Efficient reversible uniform and non-uniform random number generation in UNU.RAN

Reversible random number generations are useful in large-scale fault-tolerant parallel computations and parallel discrete event …

Srikanth Yoginath, Kalyan Perumalla

PDF Cite

Towards Reversible Basic Linear Algebra Subprograms: A Performance Study

Problems such as fault tolerance and scalable synchronization can be efficiently solved using reversibility of applications…

Kalyan Perumalla, Srikanth Yoginath

PDF Cite

Reverse Computation for Rollback-based Fault Tolerance in Large Parallel Systems

Reverse computation is presented here as an important future direction in addressing the challenge o…

Kalyan Perumalla, Alfred Park

PDF Cite

Tutorial: Introduction to Reversible Computing

This tutorial provides an introduction to the concept of reversible computing, adopting an expanded view…

Kalyan Perumalla

PDF Cite Slides

Reversibly Finding the Square Root of an Integer

Here we consider a case study in reversible computing, namely, how to reversibly compute the integer square root …

Melissa Yu, Kalyan Perumalla

PDF Slides

Reversible Simulations of Elastic Collisions (Extended arXiv version)

Consider a system of N identical hard spherical particles moving in a d-dimensional box and undergoing elastic, possibly multi-particle, collisions…

Kalyan Perumalla, Vladimir A. Protopopescu

PDF Cite

Reversible Simulations of Elastic Collisions

Consider a system of N identical hard spherical particles moving in a d-dimensional box and undergoi…

Kalyan Perumalla, Vladimir A. Protopopescu

PDF Cite

Introduction to Reversible Computing

Few books comprehensively cover the software and programming aspects of reversible computing. Fillin…

Kalyan Perumalla

PDF Cite

ReveR-SES: Reversible Software Execution Systems

[Pub 137]

Kalyan Perumalla

Cite

Discrete Event Execution and Reversibility: Challenges in the Path to Asynchrony for Massively Parallel Computing

To keep up with the increasing number of processing elements in parallel/distributed computing, traditional tightly-coupled time-stepped models…

Kalyan Perumalla

Cite Slides

Reversible Parallel Discrete Event Formulation of a TLM-based Radio Signal Propagation Model

Radio signal strength estimation is essential in many applications, including the design of military…

Sudip Seal, Kalyan Perumalla

PDF Cite

ReveR-SES: Reversible Software Execution Systems

[Pub 136] http://science.energy.gov/ascr/ascac/meetings/nov-2011/

Kalyan Perumalla

Cite

Reversible Parallel Discrete-Event Execution of Large-scale Epidemic Outbreak Models

The spatial scale, runtime speed, and behavioral detail of epidemic outbreak simulations altogether …

Kalyan Perumalla, Sudip Seal

PDF Cite Slides

Compiler-based Automation Approaches to Reverse Computation

Automation is useful to facilitate reverse code generation from normal code. Here, we describe our …

Kalyan Perumalla, Christopher Carothers

Cite Slides

Reversible Discrete Event Formulation and Optimistic Parallel Execution of Vehicular Traffic Models

Vehicular traffic simulations are useful in applications such as emergency planning and traffic mana…

Kalyan Perumalla, Srikanth Yoginath

PDF Cite

Perfect Reversal of Rejection Sampling Methods for First-Passage-Time and Similar Probability Distributions

We present a perfectly reversible method for bi-directional generation of samples from computational…

Kalyan Perumalla, Aleksandar Donev

PDF Cite

NAERM

Sat, 01 Jan 2022 00:00:00 +0000

North American Energy Resilience Model provides the next generation planning infrastructure for the US national energy grid using a cloud-based real-time, interoperable, and federated modeling, simulation, and visualization technology.

Overview

NAERM dependencies across energy infrastructure layers

Interactive Contingency Selection

Contributions

As ORNL lead for the NAERM Software Architecture Group, my efforts were focused on enabling cloud-based federated simulations executing in a distributed fashion across PNNL, ORNL and ANL servers. In particular, a dynamically configured and launched network of virtual machines via Amazon Web Services (AWS), coordinated using AWS Step Functions. Automated launches of electric transmission grid scenarios can be triggered in a highly responsive fashion. The software architecture was designed to evaluated using the Siemens PSS/E simulator instances that were licensed to execute in on-premise mode only at ORNL. Additionally, PowerWorld simulations were dynamically interoperated with other instances using federated simulation interfaces of the HELICS system for interfacing with gas network outage models simulated at ANL.

A contingency visualization, selection and launch tool was developed with one of my team members (Dr. Maksudul Alam).

Architecture

Cloud-based Software Architecture for Federated Simulation

Demo Videos

Description

The NAERM will ultimately provide real-time situational awareness and analysis capabilities for emergency events for optimal operations and recovery, so that the Federal Government can quickly and effectively prepare and respond, for example, providing recommendations in coordination with State and local governments, Federal Emergency Management Agency (FEMA), and the National Guard. While the primary focus is on the energy sector, the NAERM will further assist industry in assessing the resilience implications of energy planning decisions on associated infrastructure. NAERM capabilities will also be leveraged by DOE’s National Nuclear Security Administration (NNSA), the Department of Defense (DoD), and the Department of Homeland Security (DHS) in support of their national security missions.
– NAERM Report

The North American Energy Resilience Model project is focused on developing the next generation planning infrastructure for the US national energy grid using a cloud-based real-time, interoperable, and federated modeling, simulation, and visualization technology

Ultimately, NAERM will enable DOE’s provision of situational assessment advice to industry and government to ameliorate the risk of and consequence associated with large- scale service disruptions across infrastructure sectors—and geographic and organizational boundaries—and the lengthy restoration and recovery operations following an extreme event. The NAERM will advance the state-of-science in planning and operations of energy supply in extreme events and provide rigorous resilience and associated economics metrics for these sectors.
– NAERM Report

Organization

Sponsor: US Department of Energy (DOE)
- Office: Office of Electricity (OE)
- Program: North American Energy Resilience Model (NAERM)
Team: ORNL, PNNL, LLNL, ANL, and others

Computational Epidemiology

Tue, 26 Apr 2022 00:00:00 +0000

Global Pervasive Computational Epidemiology

Computational Epidemiology

Overview

NSF Expeditions Project

Organization

Sponsor: NSF
Institutions: University of Virginia
Period: 2020-2025

ZeroIn

Sat, 01 Jan 2022 00:00:00 +0000

Our formulation of a new AI/ML-based, real-time computational framework aims to learn and flag defects in software as early as the time of commit in the developers’ repositories.

ZeroIn Network

Overview

Using novel AI/ML techniques, ZeroIn zeroes-in onto problems in software repositories and aims to identify code vulnerabilities at their very origin, namely, at the time at which developers commit their codes into their repositories.

ZeroIn: Characterizing the Data Distributions of Commits in Software Repositories

A characterization of the software development metadata is presented in terms of distributions of data that best captures the trends in the datasets, to feed into the machine learning components of ZeroIn to exploit connectivity among the sets of repositories, commits, and developers.

Kalyan Perumalla, Aradhana Soni, Rupam Dey, Steven Rich

PDF

Using Machine Learning Towards Early Flagging of Potentially Buggy Software Commits

Using multiple classifiers we verify the feasibility of using metadata from synthetic datasets modeled by a characterization of a few large software repositories and developer profiles. Results show that the metadata-based learning approach appears promising towards early flagging of potentially buggy commits in software repositories.

Aradhana Soni, Kalyan Perumalla

PDF

Characterizing the Distributions of Commits in Large Source Code Repositories

We present preliminary results from characterizing the distribution of 452 million commits in a metadata listing from GitHub repositories. Based on multiple distributions, we find the best fits and second best fits across different ranges in the data. The characterization is aimed at synthetic repository generation suitable for use in simulation and machine learning.

Aradhana Soni, Kalyan Perumalla, Rupam Dey

PDF

Organization

Sponsor: Industry
Institutions: University of Tennessee, Knoxville
Period: 2021-2024

Gallery

DES Grid

Sat, 01 Jan 2022 00:00:00 +0000

Discrete Event Modeling of the Electric Grid is a new non-equilibrium, transient analysis model and solver that is indispensable for new advancements in grids with high renewable penetration.

Novel Discrete Event Modeling and Simulation of Energy Grids

Overview

This project is aimed at creating and demonstrating a new approach to transient analysis for power systems that has three key advantages over existing methods. These advantages are:

an accurate, physically plausible model that creates signals perceived by sensors throughout the transmission and distribution networks;
enabling a natural and accurate representation of modern electrical loads, such as power electronics devices, within the distribution system, and
a computationally tractable method of simulating these models at large scales.

Gallery

Approach

Grid modeling and simulation requires accurate capture of physics combined with internal and external behavioral elements. The physics includes movement of electricity at multiple scales broadly classified as transmission, distribution, and sub-distribution. This is driven by internal behaviors: generation is dictated by physics of power generators plus sophisticated controls to maintain voltage, and by consumption loads driven by a wide variety of traditional energy sinks as well as increasingly sophisticated sensors and controllers introduced by smart grid technologies. In light of advances in distributed energy resources and renewables, most aspects of transport, generation, consumption and responses are also greatly controlled by rich internal behaviors. Our approach provides support in one form or another to modeling and simulating all the aforementioned grid elements.

We approach the grid modeling and simulation problem with a first principles-based, fully generalized, transient simulation view of the grid. Our starting point is that of a new three-phase model that is equally applicable at multiple levels, spanning transmission, distribution and sub-distribution.

Our three-phase model is designed to produce voltage and current wave forms at scales in time and geographical spaces sufficient to understand system-wide dynamics resulting from localized-yet-interconnected behaviors of:

signal processing algorithms and logic in inverters, protection devices, and other equipment that act on millisecond time scales;
electrical and mechanical dynamics of conventional and new solar, fossil fuel, nuclear, and other power generators that evolve over tens of seconds; and
detailed dynamics of loads that may contribute to large-scale distributed controls that counterbalance the volatility of distributed/renewable generation.

We create a new type of power system model that integrates dynamic load and generation models with a dynamic, rather than quasi-equilibrium, model of the transmission and distribution network. This new model enables relevant, simulation-based performance assessments of new sensors, such as next generation phasor measurement units and distribution-based frequency sensors, and provide an unprecedented, indispensable understanding of how sensor characteristics impact wide area control strategies.

The new model developed through this research replaces pseudo equilibrium models of transmissions and loads with dynamic elements that properly resolve the physics that transmit electrical power. With these new models it becomes possible in simulation to measure physically plausible voltage signals throughout a large power system during a transient, and to do so with complete knowledge of the events within the modeled power system that produced the observed signal. This new insight will propel advancing sensing and control technologies that would be infeasible to engineer with present, inadequate models and experimental test beds.

Organization

Sponsor: US Department of Energy (DOE)
- Office: Office of Electricity (OE)
- Program:
  - Grid Modernization Laboratory Consortium (GMLC)
  - Advanced Grid Modeling (AGM)
Prime: Oak Ridge National Laboratory (ORNL)

Additional Motivation

Current methods for electromechanical transient analysis of electrical power systems predate the significant, recent advances in sensing technology that allow measurements of frequency to be made through the transmission and distribution systems. Consequently, current methods of analysis make no attempt to accurately calculate the signals that these sensors are designed to monitor. This creates considerable difficulties when current methods, models, and simulation tools are used to assess how a new device that relies on measures of frequency will affect the response of the power system to a disturbance,

This problem has been widely recognized in recent years, and numerous, essentially ad hoc, solutions have been proposed. While these ad hoc techniques can, and have, been easily integrated into existing tools, they generally lack a fundamental basis in the physics of an electrical power system, are prone to large numerical errors, or both. At best, these proposals must be viewed as a stop gap measure until models become available that properly resolve dynamical effects in the transmission network that, in conjunction with the generator dynamics, ultimately determine how a signal is perceived by a sensor within the transmission and distribution systems.

Selected Publications

A Case Study in Simulation Methods for Power Electronic Circuits

A simplified circuit is used as a case study to uncover and highlight key considerations in the use of traditional numerical simulation methods and compare them with those obtained from alternative methods that are discrete event-based from the outset. Results show the regimes where the traditional numerical methods and the alternative discrete event methods are applicable, and the need for discrete event approaches that precisely and efficiently resolve switching dynamics produced by power electronics systems that are important in emerging grid scenarios, such as large scale renewable energy.

James Nutaro, Suman Debnath, Kalyan Perumalla

PDF Cite DOI

Digital Twin Framework

Sat, 01 Jan 2022 00:00:00 +0000

Our novel Digital Twin Framework (DTF) is designed to improve resilience of critical infrastructure systems by continuously comparing the infrastructure state with automatically generated, AI/ML-based, real-time digital-twin simulation of the system.

Team

Digital Twin Framework Software Architecture

Overview

As the level of automation in critical infrastructure increases, the ability to detect cyber intrusions becomes more crucial and extremely challenging. Recent cyber attacks demonstrate the devastating and widespread affects they can have on critical infrastructure.

DTF is designed specifically to detect and eventually prevent such attacks, with models validated against experimental data from two critical infrastructure experimental emulators – a canal lock system and an electric distribution system – exhibiting very different dynamics. The canal lock system’s digital twin uses a recurrent neural network trained from the experimental data collected via the DTF. A digital twin of the transmission system is created using a commercial real-time power systems simulator and integrated into our DTF along with the hardware, embedded controllers, and live sensor data using the Open Field Message Bus data model, and publish/subscribe communication protocols. A cyber attack is used on both systems to demonstrate the DTF’s detection capability.

Organization

Sponsor: Lab Directed Research and Development, Oak Ridge National Laboratory
Period: 2018-2020

Gallery

On the Effectiveness of Recurrent Neural Networks for Live Modeling of Cyber-Physical Systems

We empirically study the effectiveness of Recurrent Neural Network (RNN)-based models as the basis of DT-based resilience and uncover the important characteristics of an RNN-based solution with experimentation on a lab-scale Canal Lock CPS emulator with live validations and attack scenarios. For the first time, we demonstrate actual, real-time use of a RNN-based model as a DT for performing live analysis on an operational CPS.

Srikanth Yoginath, Varisara Tansakul, Supriya Chinthavali, Curtis Taylor, Joshua Hambrick, Philip Irminger, Kalyan Perumalla

PDF Cite DOI

A Digital Twin Framework for Testing, Evaluation and Deployment of Resilient Cyber-physical Systems

We describe an approach to detecting and preventing cyber attacks by continuously comparing the infrastructure state with a real-time digital-twin simulation of it. Specifically, we describe and demonstrate a Digital Twin Framework (DTF) designed specifically to detect and eventually prevent such attacks. The canal lock system’s digital twin uses a recurrent neural network trained from the experimental data collected via the DTF.

Raymond Hink, Mark Buckner, Supriya Chinthavali, Chris Craig, Timothy Daniel, Joel Dawson, Milton Ericson, Joshua Hambrick, Philip Irminger, Ryan Kerekes, Juan Lopez, Kalyan Perumalla, Nicholas Peters, Stacy Prowell, Varisara Tansakul, Curtis Taylor, Bailu Xiao, Srikanth Yoginath

PDF

EpiClone: Epidemiological Clonable Simulations

Sat, 01 Jan 2022 00:00:00 +0000

Cloned simulations of epidemiological outbreaks like COVID-19 can now rapidly evaluate millions of what-if scenarios at national and world scales, and efficiently exploit 1000s of GPUs.

EpiClone model elements

EpiClone simulating USA scenarios

Overview

EpiClone represents the next advancements in what-if decision evaluation on state-of-the-art accelerated computing platforms including supercomputers containing thousands of GPUs.

EpiClone addresses global epidemiological outbreaks via an incremental simulation-based what-if decision tree evolution. EpiClone offers new scaling capabilities that were previously not possible before for rapidly simulating thousands or millions of epidemiological scenarios.

The EpiClone approach and results in scalable modeling and fast scenario exploration provide a leap in the analyses of epidemics and other complex systems.

Gallery

Organization

Sponsor: Lab Directed Research and Development, Oak Ridge National Laboratory
Period: 2015-2017

Scalable Cloning on Large-Scale GPU Platforms with Application to Time-Stepped Simulations on Grids

Cloning is a technique to efficiently simulate a tree of multiple what-if scenarios that are unraveled during the course of a base simulation. We present the conceptual simulation framework, algorithmic foundations, and runtime interface of CloneX, a new system we designed for scalable simulation cloning.

Srikanth Yoginath, Kalyan Perumalla

PDF Cite DOI

Mesoscopic Modeling and Rapid Simulation of Incremental Changes in Epidemic Scenarios on GPUs: Fast What–If Analyses of Localized and Dynamic Effects

A mesoscopic modeling approach is described that strikes a middle ground between macroscopic models based on coupled differential equations and microscopic models built on fine-grained behaviors at the individual entity level. Execution of our implementation scaled to 8192 GPUs of supercomputing platforms demonstrates the ability to rapidly evaluate what–if scenarios several orders of magnitude faster than the conventional methods.

Kalyan Perumalla, Maksudul Alam

PDF Cite DOI

Discrete Event Modeling and Massively Parallel Execution of Epidemic Outbreak Phenomena

In complex phenomena such as epidemiological outbreaks, the intensity of inherent feedback effects a…

Kalyan Perumalla, Sudip Seal

PDF Cite

Reversible Parallel Discrete-Event Execution of Large-scale Epidemic Outbreak Models

The spatial scale, runtime speed, and behavioral detail of epidemic outbreak simulations altogether …

Kalyan Perumalla, Sudip Seal

PDF Cite Slides

Intelligent Design of Structure for Function

Sat, 01 Jan 2022 00:00:00 +0000

Intelligent Design is a novel paradigm for automated, generalized, and optimized physical structural design of 3D geometry meeting a desired function.

Intelligent Design

Overview

In engineering design, an engineer starts with an initial heuristic structure and then iteratively modifies it until it satisfies all the design requirements. This lacks a thorough theoretical basis to account for the vast design space for a given problem. All current design methods are limited or biased by historical knowledge. What if this traditional design paradigm were dropped and an entirely new, unbiased, thorough framework were adopted?

Our novel Intelligent Design is the first approach to mix additive as well as subtractive steps in intelligently exploring the design space. Furthermore, it is fundamentally delinked from the traditional necessity to start with a closely related design that has been previously proven to be suitable.

(a) Traditional approach with structural determination methods that start with (b) Solid block, and (c) Minimal connection designs

Organization

Sponsor: Laboratory-Directed Research and Development, Oak Ridge National Laboratory

Gallery

CloneX

Sat, 01 Jan 2022 00:00:00 +0000

Cloned eXecution of simulations is a powerful decision-making system to incrementally evaluate millions of what-if scenarios and scale to 1000s of GPUs.

Illustration of CloneX what-if tree on diffusion processes

CloneX computational and memory savings with increasing tree depth

Overview

CloneX is a novel system we designed for scalable simulation cloning, consisting of a conceptual simulation framework, algorithmic foundations, and a highly scalable runtime interface. It efficiently and dynamically creates whole logical copies of a dynamic tree of simulations across a large parallel system without full physical duplication of computation and memory.

CloneX Software Architecture

Titan Supercomputer with 1000s of GPUs

CloneX efficiently simulates a tree of multiple what-if scenarios unraveled during the course of a normal (base) simulation. Cloned execution is highly challenging to realize on large, distributed memory computing platforms, due to the dynamic nature of the computational load across clones, and due to the complex dependencies spanning the clone tree.

CloneX has been tested on 1000s of GPUs of a supercomputing system and evaluated with multiple benchmarks – such as heat diffusion, forest fire, and disease propagation models – delivering a speed up of over two orders of magnitude compared to replicated runs.

CloneX represents a major leap in ensemble simulations as a significantly faster and scalable way to execute many what-if scenarios of large simulations.

Organization

Sponsors: US Department of Energy (DOE)
- Office: Advanced Scientific Computing Research (ASCR)
  - Program: Early Career Research Program (ECRP)
- Office: ORNL Strategic Planning Office
  - Program: Laboratory-Directed Research and Develoopment (LDRD)
Period: 2015-2017

Gallery

Mesoscopic Modeling and Rapid Simulation of Incremental Changes in Epidemic Scenarios on GPUs: Fast What–If Analyses of Localized and Dynamic Effects

Kalyan Perumalla, Maksudul Alam

PDF Cite DOI

Scalable Cloning on Large-Scale GPU Platforms with Application to Time-Stepped Simulations on Grids

Srikanth Yoginath, Kalyan Perumalla

PDF Cite DOI

Kensor: Coordinated Intelligence from Co-located Sensors

Sat, 01 Jan 2022 00:00:00 +0000

Kensor uncovers coordinated intelligence about normal and abnormal phenomena from multiple sensors co-located in close proximity in secure installations.

Kensor

Overview

Given a set of co-located sensors, we seek an intelligent approach that would automatically determine the “normal” patterns of behaviors among the correlated sensors.

After normal behavior is extracted, later monitoring should detect any deviant variations over time.

An example application is an entry monitoring and alert system for facilities such as nuclear reactors, where badge readers, door locks, lights, weight trackers and other co-located sensors at the entry point are collectively tracked.

Gallery

Organization

Sponsor: Nuclear Safety
Period: 2019-2020

Kensor: Coordinated Intelligence from Co-Located Sensors

Here, we focus on coordinated intelligence about normal and abnormal phenomena from multiple sensors geographically co-located, monitoring and controlling a set of co-located devices. Given a set of co-located sensors, we develop an intelligent approach that automatically determines the ’normal’ patterns of behaviors among the correlated sensors. After normal behavior is extracted, later monitoring detects deviant variations over time.

Olivera Kotevska, Kalyan Perumalla, Juan Lopez

PDF Cite DOI

HELICS

Sat, 01 Jan 2022 00:00:00 +0000

Hierarchical Engine for Large-Scale Infrastructure Co-Simulation (HELICS) is the first open-source software platform that co-simulates the nation’s power grid across transmission, distribution, and data communication systems.

Hierarchical Engine for Large-Scale Infrastructure Co-Simulation (HELICS)

Overview

HELICS enables grid planners and operators to understand how to navigate challenges such as integrating large-scale renewable energy resources and digital devices and defending against natural disasters and cyber threats.

HELICS was nominated to the R&D100 awards and was selected as an R&D100 Award finalist in 2018.

R&D100 Finalist Award for HELICS

Organization

Sponsor: US Department of Energy (DOE)
- Office: Office of Electricity (OE)
- Program: Grid Modernization Laboratory Consortium (GMLC)
Period: 2015-2017
Team: ORNL, PNNL, LLNL, ANL, and others

BLOCKTRI: Parallel Block Tridiagonal Solver

Sat, 01 Jan 2022 00:00:00 +0000

Our Block Tridiagonal Solver is one of the fastest parallel solvers for scientific codes, written in FORTRAN and MPI using the block cyclic algorithm, and tested with plasma equilibrium simulations for fusion energy tokamaks and astrophysics applications.

BLOCKTRI

Organization

Sponsor: US Department of Energy (DOE)
- Office: Office of Science (SC)
- Program: Fusion Energy and International Thermonuclear Experimental Reactor (ITER)
Prime: Oak Ridge National Laboratory (ORNL)

Selected Publications

Bcyclic: A Parallel Block Tri-diagonal Matrix Cyclic Solver

A block tri-diagonal matrix is factored with minimal fill-in using a cyclic reduction algorithm that…

Steven Hirshman, Kalyan Perumalla, Vickie Lynch, Raul Sanchez

PDF Cite

Revisiting Cyclic Reduction and Parallel Prefix-Based Algorithms for Tri-diagonal Systems of Equations

Direct solvers based on prefix computation and cyclic reduction algorithms exploit the special struc…

Sudip Seal, Kalyan Perumalla, Steven Hirshman

PDF Cite

Scaling the SIESTA Magnetohydrodynamics Equilibrium Code

We report the results of a scaling effort that increases both the speed and resolution of the SIESTA…

Sudip Seal, Kalyan Perumalla, Steven Hirshman

PDF Cite

Improved Parallelization of the SIESTA Magneto-hydrodynamic Equilibrium Code Using Cyclic Reduction

SIESTA is a parallel three-dimensional plasma equilibrium code capable of resolving magnetic islands…

Sudip Seal, Steven Hirshman, Kalyan Perumalla

Cite

Efficient Heterogeneous Execution on Large Multicore and Accelerator Platforms: Case Study Using a Block Tridiagonal Solver

The algorithmic and implementation principles are explored in gainfully exploiting GPU accelerators in conjunction with multicore processors on high-end systems…

Alfred Park, Kalyan Perumalla

PDF Cite

NetWarp

Sat, 01 Jan 2022 00:00:00 +0000

NetWarp is a novel time-synchronized virtual machine(VM)-based parallel simulation framework that accurately lifts the devices and the network communications to a virtual time plane while retaining full fidelity.

NetWarp

Organization

Sponsors: US Department of Defense
- Office: Army Research Laboratory (ARL)
- Office: Missile Defense Agency (MDA)
- Programs: Computational Sciences, STTR
Prime: Oak Ridge National Laboratory (ORNL)

Gallery

summary: Secure Virtual Environment for Cyber Resiliency Validation building on NetWarp technology for cybersecurity in hardware and software of missile defense systems

Virtual Machine-Based Simulation Platform For MANET-Based Cyber Infrastructure

In modeling and simulating complex systems such as mobile ad-hoc networks (MANETs) in defense communications, …

Srikanth Yoginath, Kalyan Perumalla, Brian Henz

PDF Cite DOI

Taming Wild Horses: The Need for Virtual Time-based Scheduling of VMs in Network Simulations

The next generation of scalable network simulators employ virtual machines (VMs) to act as high-fidelity models of traffic producer/consumer nodes…

Srikanth Yoginath, Kalyan Perumalla, Brian Henz

PDF Cite

Optimized Hypervisor Scheduler for Parallel Discrete Event Simulations on Virtual Machine Platforms

With the advent of virtual machine (VM)-based platforms for parallel computing, it is now possible to execute parallel discrete event simulations (PDES)…

Srikanth Yoginath, Kalyan Perumalla

PDF Cite

IAS Fellowship

Sat, 01 Jan 2022 00:00:00 +0000

Duham Unversity Institute of Advanced Study Fellowship gathers together the world’s finest scholars and non-academics (such as intellectuals, artists, writers, journalists, policy makers, and politicians) from the full spectrum of science, social science, arts and humanities disciplines to address themes of global significance.

Overview

The Institute of Advanced Study (IAS) provides its fellows with a setting that offers them freedom to think in an unconstrained way, exempt from the day to day demands of their normal professional obligations, and in the company of other thinkers from very different backgrounds. The IAS seeks to develop a truly global perspective by ensuring that its fellows are recruited from all over the world, including the global south. The IAS instigated a range of activities focused on advancing reflexive understandings of interdisciplinarity.

The IAS fellowship provided Dr. Perumalla with the opportunity to develop own research and ideas in a thriving community of intellectuals of national and international standing, within the Institute, through the College system, and by forging strong links with at least one department at Durham.

For the duration of his stay, Dr. Perumalla was provided with a single occupancy office in Cosin’s Hall, the home of the Institute of Advanced Study. The Hall, a magnificent listed building dedicated to the IAS and elegantly refurbished, is situated on a World Heritage site that includes Durham Cathedral and Durham Castle. In addition, all fellows were welcomed into a college community where they were offered half board accommodation (in a one bedroom flat), and membership of the college’s Senior Common Room. The IAS covered the costs associated with travel to Durham, UK from ORNL and provided an honorarium.

All Fellows

Dr. Perumalla’s Cohort

Organization

Sponsor: Duham University and Oak Ridge National Laboratory

Gallery

Normalcy, Magic, Miracle and Error: Emergence along a Reversibility Spectrum

Kalyan Perumalla

PDF Cite

On the Definability of Art

There are many key concepts that, even while being part of everyday life, elude definition. One such is “Art.” Here, possible ways are identified to define Art, along with a description of a few factors that underlie the challenge in arriving at a definition. Additionally, a candidate definition from a scientist’s viewpoint is proposed for an abstract, encompassing model.

Kalyan Perumalla, Richard Read

PDF

Rocks3D-HPC

Sat, 01 Jan 2022 00:00:00 +0000

Rocks3D-HPC is an efficient parallelization of the Rocks3D high-fidelity earth moving equipment simulation based on the Discrete Element Method implemented using FORTRAN, OpenMP multi-threading, and message passing interface (MPI) for high performance computing.

Organization

Sponsor: Caterpillar, Inc.
Institutions: ORNL

Gallery

MutEnt

Fri, 15 Apr 2022 00:00:00 +0000

Mutual Entropy-based Image Registration

Overview

MutEnt

Organization

Sponsor: Department of Defense

Computing a Non-trivial Lower Bound on the Joint Entropy between Two Images

In this report, a non-trivial lower bound on the joint entropy of two non-identical images is developed, which is greater than the …

Kalyan Perumalla

PDF Cite DOI

Computational Speed and Matching Quality using an Upper Bound on the Normalized Mutual Information

URL

Kalyan Perumalla, Maksudul Alam, Devin A White

Cite

Gallery

Visualizing the National Energy Grid

Sat, 01 Jan 2022 00:00:00 +0000

EGIVAC, System for Electric Grid Interactive Visualization, Animation, and Control, is one of the ground-breaking, feature-rich, large-scale visualizations of the US national energy grid.

Visualization on the wall

Overview

In 2006-07, developed the first grid visualization system from the ground up, scaling to the full national grid sizes, 100KV to 700KV+.

The system opened up, for the first time ever, the ability to dynamically customize geophysical layers and icons together with grid network buses and lines, shown with accurate latitude and longitude coordinates synchronized with rich geographical layers.

Gallery

Interactive, visual differentiation based on voltage ratings was achieved for the first time at the national scale incorporating large interconnects including ERCOT and Eastern Interconnect.

The capability for the rendering to scale even from smaller personal desktop displays up to the largest, wall-sized multi-monitor systems was achieved for the first time for national energy grids.

Dynamic, interactive animation capabilities provided a natural, visual evaluation of contingencies to decisionmaking teams, with the potential to tie real-time weather data streams with grid state.

Impact

This visualization system went onto inspire large investments by the US Department of Energy and other agencies in grid monitoring centers and the development of next generation systems such as VERDE and EAGLE-I.

My work also featured prominently in the research highlights of the Computational Sciences and Engineering Division across multiple years, and won the laboratory director’s award among all the lab-directed research and development projects in 2007.

Organization

Sponsor: US Department of Energy (DOE)
- Office: Office of Electricity (OE)
Prime: Oak Ridge National Laboratory (ORNL)

Features

EGIVAC is a graphical user interface designed to help electric grid operators to visualize, animate and control the operation of the grid in a highly interactive fashion. The system has been designed with the objectives of interactivity of display/control and of reflecting geographical correspondence of electrical elements. The tool is also intended to serve as an intelligent front-end to control, such as for rapid contingency analysis using parallel/distributed execution of multiple future scenarios.

Visualization

Rendering tuned to minimize clutter at nation-scale display
Color coded buses/lines for high contrast in multitude
Large screen display on Power Wall
Interactive display filters based on voltages (KV)
Population/customer coverage & reach (work in progress)
Hands-off panning
Zoom in/out

Animation

Time-driven animation
Pause/Resume
Interactive slow/fast speed control
Phasor Measurement Units (PMU) animation (planned)
Morphing geographical topology into electrical circuit (series/parallel) (work in progress)
Dynamic graphs of affected operational measures (work in progress)

Control

Visual alert effects (blinking, color changes)
Event impact analysis (e.g., population affected by hurricanes)
Parallel targeted contingency analysis (work in progress)

Resolution

Tested with 45,000 buses and 60,000 lines
Smooth animation (1 to 30 frames per second)

Platform

Java (portable to Mac, Windows, Linux, and other platforms)

Fact Sheet

RealSim: Real-Time Simulations

Sat, 01 Jan 2022 00:00:00 +0000

Real-time Simulations is a simulation framework for fast evaluation of national emergency scenarios such as hurricanes, evaluated with large data streams and real-time computation on the latest hardware platforms on the edge.

RealSim