Company News

Overcoming Imperfect Grid Data with Machine Learning

Authored by:
Last Updated: 
February 14, 2022

Last week, we shared the exciting announcement that Camus Energy, along with the Pacific Northwest National Laboratory and Kit Carson Electric Cooperative, was awarded $750,000 by the Solar Energy Technology Office (SETO) for the US Department of Energy. The award will fund a twelve month project to leverage artificial intelligence (AI) and machine learning (ML) algorithms to fill data gaps from unmetered solar and support grid operators. While our official announcement shares general information, this blog post provides context on why we are incredibly excited about this project and how it fits in to the future of grid management.

Data is king! Except when it’s not.

In the United States, more than 95 million smart meters provide tens of billions of data points to utilities each day. Adding information from line sensors, substation monitoring, and other utility-owned equipment, grid operators have a wealth of data at their disposal. That information is powerful…when it’s put to use.

That’s not exactly what’s happening with the petabytes of data that utilities collect each day. Unfortunately, the physics-based models that grid operators rely upon are not particularly well-suited to flaws and gaps in data. While the AMI feed provides information on millions of meters, a few pieces of broken equipment or coding errors can really muck up the models – often leading to a hesitancy to incorporate such data into real-time grid modeling.

Taking it one step further, when customers install devices that meaningfully shift their consumption patterns, such as rooftop solar, a faulty meter or data error can hide behavior that is significantly different from expectations. That’s usually OK for a single solar installation or even a small set; but for a grid with high penetration of rooftop and community solar, the lack of integration of real-time data can lead to significant differences between actual and modeled grid state.

So how can utilities incorporate real-time data while keeping their grid model running smoothly?

Smart Inferences & Practical ML for Utilities

While artificial intelligence (AI) and machine learning (ML) are sometimes (rightfully) panned as buzzwords in the utility industry, filling data gaps and fixing flaws through model-based inference is a perfect use case for this technology. In fact, our team has already deployed machine learning for similar use cases before, at Google and SpaceX. With this approach, algorithms identify gaps and errors in data then use secondary data sources and prior values to make inferences in real-time. It’s similar in concept to a grid operator looking at conditions near a data gap, consulting their prior experience and similar-looking assets, and making an informed guess – but this all happens automatically by the AI. Operators can see which values are measured and which are inferred, and the physics models are able to run efficiently and reliably – providing enhanced situational awareness.

Data Imputation Diagram

Nerding Out (The Specifics)

This is the part where we dip our toes into machine learning nerdiness. (Not feeling it? Skip ahead)

Our project with PNNL and Kit Carson will empirically test, down-select, and incorporate advanced machine learning (ML) and data analytics (DA) methods – helping to identify the best methods for use in managing a grid with high penetration of local solar generation. We plan to assess methods across two primary data sets: 1) loads/injections at the network endpoints and 2) power flows and voltages over the network.

The project is broadly split into two phases. First, we develop network endpoint situational awareness – meaning that we identify the best spatial and temporal machine learning methods to create historical, real-time, and forecasted time series data for network endpoints – without gaps or data errors. Second, we develop full network situational awareness – identifying methods that can effectively connect the network endpoint data with the network data (e.g. power flows and voltages) – using existing models or creating a fully data-driven method (for networks without a model).

Across both phases, we will test a variety of machine learning and data analysis methods, including (but not limited to):

In addition, because data-driven prediction of intra-day (>1 hour) solar PV data is challenging due to the dynamic nature of the atmosphere, we plan to integrate key Weather and Research Forecasting (WRF-Solar) outputs into our data pipeline to create our intra-day PV forecasts. In addition, we plan to leverage deep learning (specifically a long short-term memory neural network architecture) to integrate the WRF-Solar data with data from individual PV sites to improve these forecasts.

By analyzing a variety of machine learning and data analysis methods, this project will help identify the tools, all available through open source, that utilities can use to fill gaps in their data and improve operations on grids with high penetration of local solar.

The Unsexy Valley: Reducing Energy Costs with Machine Learning

Filling data gaps may not be an awe-inspiring application for machine learning, but it’s a crucial step to building the grid management system of the future. As operators gain better visibility into their grids, they can meaningfully reduce system costs with better utilization of cheap renewables and local flexibility. For communities eager to reinvest those dollars into resilience, decarbonization, and economic development, these savings matter.

On our team, applications like smart inferences are what get us out of bed in the morning. They are effective and practical uses of advanced (but proven!) technology, delivering benefits to real people in communities like ours. They may not show up in TechCrunch (that’s why we call our area the “unsexy valley”), but they sure can move the needle for our customers.

We know from experience that cloud-hosted tools have the potential to revolutionize industries. By building software that is reliable, secure, and flexible, we empower utilities, energy providers, and entire communities to move towards a more distributed, lower-carbon, and more equitable energy future. This project, alongside our full-scale commercial deployments, helps move our customers and the utility industry towards a new approach to grid management. We’re excited to get started!

If you’re interested in learning more about this project, please reach out to the Camus Energy team at

Contact Us

Start your community energy transformation.