From The Team

The Big Data Problem For Utilities

Authored by:
Last Updated: 
December 27, 2023

This is the fourth installment of our series: The Top 5 Challenges Facing Distribution Utilities. Subscribe to automatically receive future installments.

In today’s digital world, the old adage “Knowledge is power” is often eclipsed by the increasingly appropriate  “Data is king.” More data is generated every day than ever before; by some accounts, roughly 90% of existing data was generated in the last two years alone. 

Thanks to electrification, decarbonization, and the combined forces of the energy transition, data management is increasingly critical for utilities and grid operators tasked with managing the growing complexities of a dynamic grid. Better use of data can directly translate to enhanced grid visibility and better-informed active management, providing opportunities to cost-effectively improve grid reliability, sustainability, and operational efficiency. 

However, for most utility teams, the sheer amount of incoming data feels overwhelming. Current analytics tools and datasets tend to be siloed within utility teams. Traditional processes are not optimized for integrated, efficient management of massive amounts of data across the utility. 

In order to make informed, strategic decisions about grid operations, utilities must evolve their approach to managing data. By adopting cloud-native technologies that can store and analyze massive amounts of data at extremely low costs, utilities can put their data to better use – gaining real-time and day-ahead visibility and forming the foundation for robust, long-term system planning. 

A complex grid requires better data management

Utilities rely on data from numerous software systems to make informed decisions about everything from outage mitigation to planning for renewable integration and future system capacity needs. But when it comes to managing real-time operations and the evolving role of flexible resources, utilities typically leverage data from five key software systems:

  1. Advanced Distribution Management System (ADMS) supports core grid operations and optimization, including automated outage restoration. Many utilities use a Distribution Management System (DMS) and an Outage Management System (OMS) that provide related capabilities.
  2. Supervisory Control and Data Acquisition (SCADA) delivers real-time visibility of grid conditions at crucial points on the grid, typically substations and feeders, and control of utility equipment
  3. Geographic Information Systems (GIS) provides geographic and spatial data, including often a connectivity map for “as-planned” switching conditions (e.g. what equipment is upline/downline) 
  4. Advanced Metering Infrastructure (AMI) provides meter-level information from smart meters at homes and businesses; typically at ~15-minute intervals and with a several hour collection delay
  5. Distributed Energy Resource Management System (DERMS) uses DER telemetry via direct device integrations to provide information on available DERs and their current behavior 

Together, data from these systems is capable of providing an extremely robust view of what’s happening on the distribution grid – and what’s likely to happen in the coming hours and days. But that’s only possible if data from each system is tied together into a consistent, accurate view. That’s unfortunately quite a large challenge for utilities today.

Despite collecting sufficient data for robust awareness, the siloed nature of utility software systems makes it difficult for utilities to turn data into actionable, impactful insights. As a result, utility teams face significant time and resource burdens to clean and process data and produce meaningful insights. With teams already stretched thin by growing demands and a rapidly changing industry, cleaning and analyzing data often falls by the wayside – and with it, hopes for more robust grid awareness.

The good news is that as the grid’s needs are evolving, so too are the tools that support data integration and analysis across the utility. 

Empower operators with integrated data management

Grid operators are tasked with managing an increasingly dynamic and complex system. Extreme weather threatens resilient operations, renewable generation creates new challenges for reliability, and electrified devices like EV charging stations produce difficult-to-predict demand patterns. In the face of such challenges, data can play a critical role in helping utilities plan for the future while maintaining an affordable, reliable service for their customers.  

The scale of these challenges necessitates a transition from siloed systems to purpose-built, cloud-native technologies that aggregate data from many sources to produce structured, high-quality data usable across teams. A shared data foundation that integrates and organizes data across silos opens up a world of possibilities for utilities to improve grid operations and plan effectively for the future. Here are three examples drawn from real-world experiences of utilities.

Gain end-to-end grid visibility 

While an ADMS can monitor the core operations of the grid, it lacks visibility down to the meter – and was not designed to handle ingestion of disparate meter and DER data. By integrating AMI and DER data from the grid’s edge alongside a utility’s GIS-based connectivity model, the utility can feed aggregate data into the ADMS to gain near real-time visibility from DERs to substations.

This integrated view provides better overall visibility to operators. Perhaps equally important, it enables the ADMS (or DMS plus OMS for utilities without an ADMS) to operate more effectively even with increasing distributed energy resources. For example, feeding aggregate real-time loading inclusive of distributed generation into the ADMS (or OMS) enables the system to more quickly and accurately identify viable switching options during abnormal conditions – such as during scheduled maintenance or unplanned outages. With more powerful insights, operators can deliver more reliable and cost-effective operations. 

Create accurate meter-level forecasts

By integrating AMI and SCADA data, utilities can also create a more comprehensive picture of future grid conditions. Thanks to advances in AI and machine-learning, data from the edges of the grid can be used to generate accurate, meter-level forecasts for a fraction of historical costs. These hours- and days-ahead forecasts can help utilities make informed operational decisions and better manage the challenges of changing load patterns and growing demand. 

Meter-level forecasting can also support safe restoration from grid outages by providing utilities with accurate predictions of gross load that uncover demand hidden by distributed generation. (Learn more about meter-level forecasting here.) 

Protect transformers and other critical equipment 

Changing load patterns and spiking demand from EV adoption can overload and damage upline equipment, such as distribution transformers. With a combination of AMI, DERMS, and GIS data, a utility can protect its infrastructure by analyzing the impact of EV charging on upline grid equipment, identifying high-stress equipment, and actively managing fleets of EVs and other DERs to prevent equipment overloading.

The same data can also be leveraged to simulate the impacts of future EV and other DER adoption scenarios on existing utility equipment, providing better information with which to prioritize new equipment investments, manage upgrades, and incorporate DERs into non-wires or bridge-to-wires solutions. 

The tip of the iceberg

While each of the three examples above can enable more effective, data-driven grid management, that’s just the tip of the iceberg. Integrated, cloud-scale data management can generate powerful insights across all areas of grid operations and planning. 

For utilities interested in how to leverage existing data for enhanced real-time visibility, forecasting, and system planning, our team would love to share our insights.  Learn more in our blog entries on why utilities need to embrace cloud computing and the alphabet soup of utility software systems. Or reach out to chat with one of our team members.

Contact Us

Start your community energy transformation.