Analyst Report: Smart Grid Data Analytics: A New Approach for Utilities

1 Summary

Smart grid data analytics are a key sector of the industrial Internet of Things (IoT), with the potential to help utilities address key operational, financial, and customer challenges. Leveraging the large volumes of data created by the smart grid, along with information available through third-party data services, these solutions are made possible by modern technological developments that allow the integration, processing, and analysis of multiple sources of data quickly, at unprecedented scale.

Software-as-a-Service and managed services make it possible to implement solutions rapidly and at lower cost than traditional, on-premises software implementations. For maximum success, utilities must clearly define problem sets, data source requirements, and the employees who could best make actionable use of data-driven insights.

Key findings of this report include:

  • Smart grid analytics enable utilities to more rapidly and effectively address issues regarding improved grid operations, customer engagement, and financial management. Often, improvement in a single activity, such as revenue protection, can justify the investment in data analytics.
  • Driven primarily by distribution grid optimization and customer engagement improvements, the current global smart grid data analytics market will grow from $1.3 billion to $4.8 billion by 2022, with a compound annual growth rate of 16 percent.
  • Utilities can choose between point solutions aimed at a specific problem or business area, or providers offering a suite of solutions that span the enterprise, operating from a single software platform. Point solutions may offer a quicker route to solving a specific problem, but enterprise-wide approaches offer greater flexibility to solve future problems with existing analytical tools.
  • Data analytics can be implemented on-premise or as a managed service. While on-premise implementation provides the most direct control, it requires significant upfront investment, upfront IT work, and ongoing involvement. SaaS enables faster time to deployment, mitigates talent and execution risk, reduces capital expenditures, and provides an overall lower total cost of ownership.

2 Overview and Opportunity

Utilities can use modern data analytics to inform a broad range of decision-making, from split-second distribution grid optimization to daily collections activities, long-term load forecasting, project financing, and strategies for effective customer engagement.

The analytical results will drive utility responses that may be fully automated (such as automatically sending a command to an integrated operational system to make an adjustment), partially automated (such as integrating to a work management system for field service follow-up), or conducted only by human analysts and responders (such as evaluating capital investment options or creating customer segmentations for marketing purposes).

Utilities are keen to understand new revenue sources, especially in markets where they are not regulated. Smart metering enables many new billing use cases, but which are worth pursuing and which are not? How should dynamic pricing or demand response programs be structured, assuming they are needed at all? How do intermittent energy sources and distributed generation affect grid stability, and what course of action will retain grid stability? These questions are best answered with a deep understanding that is only available through data analytics.

Further, many current utility business processes were created decades ago and are revealing their limitations. Credit and collections often rely upon 30-year-old billing systems. Medium-voltage (MV) and low-voltage (LV) distribution grids were engineered many years ago, and many utilities still see customers as nothing more than ratepayers. Utilities should scrutinize these processes to anticipate and respond to:

  • Greater non-technical losses
  • The latest generation of connected devices on distribution grids
  • Newly deregulated markets facing alternative competitors
  • Energy efficiency programs that require customer opt-in and active participation

The Latest Data Deluge

The modern utility can take advantage of data from smart grid sensors and devices, other operational sources, historical repositories, business operational sources (such as billing or CRM systems), and third-party data sources. Smart grid devices generate data in volumes unimaginable only a few years ago. MV distribution grids now include voltage and frequency sensors that install in minutes, simply by being clipped onto the conductor. Equipped with radios for wireless data transmission, the sensors can capture up to 15,000 readings per second—a potential for 473 billion readings per year per device. In the high-voltage world, a transmission network with 100 phasor measurement units (PMUs) will generate
46 terabytes of synchrophasor data per year, with each PMU producing up to 60 readings per second. Meanwhile, an installation of 2 million smart meters, reading consumption every 15 minutes, will generate 71 billion interval data records per year. Those same meters are also capable of reporting over 200 power quality measurements at sub-second intervals, if desired.

That is just a sampling of data generated within the utility. Externally, utilities need data from intermittent generation assets, such as wind and solar, which may be centralized or distributed. Also relevant are weather, demographic, economic, and customer behavioral data, depending upon the types of analyses intended.

Use Cases that Drive Demand

Utilities typically focus on three improvement areas with data analytics:

  • Improved grid operations. Automating revenue protection identification and investigation efforts, and lowering some operational costs
  • Improved customer engagement. Increasing customer satisfaction, increasing customer retention, and improving the tailoring of services to customer needs
  • Improved financial management. Improving return on investment, lowering capital investment or shifting some spending to flexible operational costs

A data analytics system can pay for itself through efficiencies that it identifies throughout the utility’s business. For example, in the United States, non-technical electricity losses typically consume 0.5 to 3.5 percent of a utility’s electricity, representing nearly $80 billion in lost revenues per year. In some Latin American countries, such as Brazil, the national average for non-technical loss is 15 percent. Using smart grid data analytics to better identify and resolve non-technical losses could save large utilities tens of millions of dollars per year.

The key use cases for any utility are likely to be unique, driven by its unique business environment. Sample use cases for data analytics that may apply to many utilities include:

  • Conservation voltage reduction (CVR) enables a utility to energize its MV network at lower levels and remain within safety bands. In addition to newly available MV sensor data, a utility might want to use power quality data from smart meters to determine how and where specifically to deploy CVR. This is a use case where data from discrete systems combine to support a single decision.
  • Better workforce management analytics enable more efficient dispatch of the field force—with the right equipment—to perform multiple tasks in a single neighborhood visit. With enough grid visibility due to enterprise-wide analytics, a single site visit to repair a meter can also include replacing a frayed conductor and trimming trees during the same visit.
  • Predictive and preventive maintenance: Data analytics can move utilities away from run-to-failure maintenance strategies and toward preventive approaches. Analyzing real-time network sensor data, smart meter data, asset maintenance records, and weather data can predict the physical pipelines, substations, poles, and other major physical assets at risk of failure.
  • Data analytics can identify complex patterns of theft, such as recurring energy consumption peaks that do not fit the normal consumption profile for a single customer. If that same customer is also delinquent in payment, then the risk of non-technical loss increases dramatically. Data analytics can see and correlate these factors on a daily basis and recommend immediate investigation or collection activities, as opposed to the end-of-the-month processes enabled by traditional credit and collection systems.

Smart Grid Data Analytics Forecast

Navigant Research forecasts that spending on software and managed services for smart grid data analytics is poised for rapid and sustained growth. Starting from a base of about $1.3 billion in 2014, the market is forecast to reach nearly $4.8 billion by 2022, at CAGR of 16 percent, as shown in the following chart.

Analytics Software and Services Spending by Region, World Markets: 2014-2022

Screen Shot 2015-02-05 at 4.35.23 PM

Source: Navigant Research

The key market drivers for smart grid data analytics are likely to be distribution grid optimization and customer engagement. The latter will be driven by replacement of antiquated customer information systems, including mainframe-based systems that some utilities have been running for over three decades.

Europe’s smart grid market is driven largely by the European Union’s aggressive 20-20-20 goals: By 2020, increase energy efficiency across the EU by 20 percent, decrease greenhouse gases compared to 1990 by 20 percent, and increase renewable source energy generation to 20 percent of all generation. The longer-term goal in Europe, Plan 2050, forecasts complete elimination of fossil-fired utility scale generation. All of these goals require far better understanding of transmission and distribution networks than is currently available.

The Asia Pacific market for smart grid data analytics is driven by the massive electricity infrastructure build-out in China. Most of this build-out is new construction, not grid updates, so the majority of the equipment is current, IT-enabled equipment that generates data useful in grid analysis. It is likely that a similar infrastructure build-out in India later in this decade will keep demand going for data analytics in Asia Pacific.

The 16 percent CAGR suggests steady growth, but it will not be as high as might be expected in a nascent industry. Growth could accelerate if utilities see a greater need to adapt to the challenges threatening them. Likewise, it is incumbent upon data analytics vendors to demonstrate the value proposition of their offerings.

3 Business Impact for Utilities

Electric utilities face an increasingly competitive market, with a variety of new external threats. Some of the major market trends that threaten traditional business models include:

  • Increased use of intermittent energy sources, mainly solar and wind
  • Distributed generation and microgrids that reduce utility revenues and may require utilities to purchase excess energy
  • Increased ability of alternative technologies, such as demand response, to bid into capacity markets
  • Decreasing cost of batteries for energy storage, which may multiply the attractiveness of distributed generation
  • Increased uptake of electric vehicles may worsen existing demand peaks or create new peaks at unusual hours
  • More energy efficient devices, such as LED lighting in commercial buildings, apply further downward pressure on utility revenue

For utilities, new high-speed analytical capabilities can help them to understand, model, and address these myriad external threats. The business benefits from data analytics adoption will be operational efficiencies, increased customer understanding and engagement, and financial management.

Operational Efficiencies

Utilities can achieve substantial financial benefits from operating grids more efficiently. For example, in 2013, there were over 3,200 publicly reported outage incidents in the United States, affecting over 14 million people and causing an estimated $150 billion in damage to the economy, according to Eaton Corp.’s “Blackout Tracker.” Preventing outages and optimizing energy inputs into networks can create substantial savings for both utilities and the broader economy.

Data analytics can enable important decisions regarding grid optimization, such as:

  • Volt/volt-ampere reactive (Volt/VAR) deployments enable utilities to energize MV networks at optimum values and to keep voltages stable throughout the network. But Volt/VAR is needed only at trouble spots on the network, not throughout 100 percent of the network. Data analytics can direct utilities to the most effective points in the network to install Volt/VAR.
  • Volt/VAR is today accomplished using capacitors and resistors. However, Navigant Research has forecast that battery prices will decrease enough that Volt/VAR using mass storage batteries will be economically feasible by the end of this decade. Data analytics can help a utility understand whether to deploy Volt/VAR now or wait until batteries are economically viable.
  • Data analytics can also help utilities reduce CapEx requirements by using grid optimization to shave peak loads and thereby reduce the need for new generation assets.
  • Improved workforce management and lower equipment failure rates can reduce the amount of staff required to serve a given area. With the coming talent drain in utilities as many experienced workers retire, the staff freed up can be used to backfill other areas.

Well-designed data analytics can improve the way utilities do business in many areas of operation. But improvement means change. For example, as noted earlier data analytics can identify suspicious energy consumption for a given customer and recommend immediate investigation as opposed to the usual end-of-the-month process. However, the change from monthly to daily collection activities implies a substantial rewrite of the collection department’s work processes. Likewise, the field crew that now handles four distinct tasks during one neighborhood visit will likely use different processes to receive and complete their work lists.

Customer Satisfaction/Customer Engagement

Utilities are transitioning from considering end users as “ratepayers” to regarding them as “customers.” In the past, utilities simply had to deliver energy reliably, send bills, and collect payments. Today, there are a number of situations where utilities need to cultivate the goodwill and cooperation of their customer base:

  • Executing demand response programs can help a utility deal with intermittency in energy supply and demand peaks. Customers must opt in to demand response programs in most jurisdictions.
  • Providing time-of-use billing often requires tailored financial incentives for customers to shift some of their power consumption away from peak demand periods.
  • Meeting customer satisfaction level requirements mandated by local or national governments.
  • Meeting energy efficiency requirements mandated by local or national governments.

Each of these situations requires the utility to understand its customers and, in some cases, to understand them at a very specific, individual level, so that the right incentives or efficiency programs can be offered. Data analytics can gather customer profile information either individually or in aggregate, to be used in deciding how to structure time-of-use billing, demand response, or other customer-targeted offerings.

Financial Management

SaaS offerings enable utilities to stand up pilot smart grid data analytics deployments quickly, because the hardware, software, telecommunications, and staff are already in place and running. This approach has been used extensively over the past five years when piloting smart meter deployments. Even utilities that will ultimately run their smart metering in-house have often opted to run the pilot from a managed service environment.

Depending upon the physical implementation of data analytics, SaaS adoption can reduce CapEx or even eliminate it from the books. SaaS offerings can be licensed completely through OpEx accounts, requiring no upfront capital investment.

However, investor-owned utilities (IOUs) in the United States are required to earn a return on capital investment. IOUs have traditionally seen increased capital investment as the only route to increased returns because of existing regulation related to power generation, transmission, and distribution. Further, increased OpEx reduces return on capital because it reduces the net operating profit that is used in calculating the return on capital. This has led some IOUs to undertake a number of complex programs in house, such as smart metering deployments and data analytics. IOUs can justify the increased OpEx for SaaS data analytics if they can demonstrate that the efficiencies resulting from the analytics enable operational savings that they can pass on to their customers. Regardless, some SaaS providers have developed accounting treatments that enable IOUs to capitalize a portion of the SaaS provider’s cloud environment.

For utilities that are not required to earn return on capital, the predictable monthly price of a managed data analytics service and the forecast benefit can be used to calculate net present value (NPV) and internal rate of return (IRR) for the analytics program. If NPV and IRR meet or exceed the utility’s required thresholds, then the data analytics program can be justified without the requirement to find capital to start the program.

Utilities that are Leading the Way

Here is a sampling of utilities that have smart grid data analytics programs underway. These utilities have invested in data analytics when the market was in its infancy, and therefore carried higher risk, so it is fair to characterize each of these utilities as forward thinking:

  • Baltimore Gas and Electric Company (BGE) initially aimed at using its smart grid data from 2 million smart meters financed partially by U.S. government stimulus funding to better operate the business. The data analytics program was fully deployed in six months, leading to forecasted annual savings of $20 million in reduced non-technical losses.
  • Florida Power & Light is engaged in an analytics program focusing on revenue protection and customer operations, using smart meter and customer information systems to create relevant flags for follow-up.
  • CenterPoint Energy uses real-time analytics to provide situational awareness for telecoms, outage, distribution, dispatching, and distribution operations. CenterPoint Energy is also using analytics for asset management and preventive maintenance.
  • Pepco Holdings Inc. has implemented a reliability planning application as a part of its broad program to improve the reliability of service to customers. This deployment includes real-time grid operations analytics solutions.
  • Duke Energy has undertaken a data integration and data mining activity across various systems by monitoring, collecting, mapping, and analyzing data related to transformers, lines, capacitors, and billing systems, etc.

4 Solution and Process Considerations

The nascent market for smart grid data analytics means that many different types of companies are vying for a share of this market. They range from pure-play SaaS data analytics solutions to MDM vendors, with traditional ERP and BI suppliers selling with or without systems integrators.

Any new technology rollout should include a change management component that considers the impact upon employees’ daily work and prepares employees for the change. Nowhere is this truer than with data analytics. This level of change in daily work can be disconcerting for long-time employees. Beyond training employees in a new way of working, performance metrics may also require adjustment to reflect the new way of working after the analytics are in place. Performance metrics that affect employee compensation will be the most sensitive areas. In geographies where employee work councils must approve work changes, those councils should be involved at the outset of the data analytics deployment.

Enterprise Solution or Point Solutions?

Point solutions in data analytics can solve a particular problem quickly, such as understanding how to deploy CVR or how to design dynamic pricing. These can be tempting, especially if a utility is suffering from a particular pain point or is pressed to comply with new regulations in a short timeframe.

However, point solutions limit a utility’s options. For example, data analytics that only address grid optimization may be of no use in determining how to get the optimal customer enrollment in demand response programs. A data analytics solution that addresses a broad range of issues—grid optimization, customer engagement, and financial operations—can provide the utility with maximum flexibility to solve business problems.

Data Scalability and Format

Continual technical advances in sensing equipment suggest that data volumes will continue to grow. Even without any advances, however, the data collection ability of currently available devices has not been reached. For example, a smart metering network of 5 million meters could, in theory, collect multiple power quality readings, each at sub-second intervals, from every meter. Such an amount of data would dwarf anything seen today. In reality, smart metering networks have not been configured to handle traffic at those volumes so, for the time being, this is not a threat.

Data analytics must capture all relevant information necessary for the utility to make informed decisions. Some of this data may be external, and some may be completely unstructured—that is, not conforming to a traditional row-and-column orientation. Smart grid data analytics will benefit most from database technology that can accommodate both structured and unstructured data types, especially for applications that may rely on both historical and near-real-time information.

Analytical Capabilities: Prebuilt, Customizable

Once a vendor has demonstrated the ability to collect data in the volumes exceeding the utility’s requirements, it is time to examine the vendor’s analytical capabilities. Experience in other industries, such as financial or health care, does not translate very well into utilities. The electricity industry has unique demands; only vendors who understand that should be considered.

A vendor with a large percentage of the utility’s desired analyses already built into its offering is likely to offer the most reliable system. This requires that the utility understand ahead of time which analyses it needs to accomplish its objectives. A single analysis may encompass tens of use cases, but, as previously stated, it is best if the utility has defined these use cases before vendor selection.

Software Architecture

An enterprise-wide analytics platform enables expansion into additional utility departments with reduced incremental expense compared to launching new projects. For example, a utility may be capturing all data coming from smart meters but, at present, is only using the interval data—the meter readings—for applications such as revenue protection or load forecasting. Smart meters can also capture volumes of power quality data, or register data, that can be used for other analyses. A modular approach enables diverse analyses, such as CVR, energy management, or workforce management, to augment existing analytics at lower cost, with less disruption, and with fewer technology changes.

5 Key Takeaways

Smart grid data analytics is still a relatively new market with many players and many approaches, but it is an opportunity to significantly improve a utility’s business, grid, and customer operations. Some key points to remember from this analysis:

  • Enterprise-wide approaches to data analytics offer the most flexibility and a higher probability of usefulness for solving future problems, even ones that do not exist today.
  • As in any early market, selecting the right supplier is critical. Suppliers should have a solid knowledge of utility grid operations, financial stability, and reference clients.
  • Data analytics products can only be successful in a utility if they can scale up to data volumes larger than the utility’s maximum possible scenario and perform the analyses most important to a utility within required timeframes.
  • Data analytics software should offer maximum flexibility out of the box. Utilities should be able to accomplish what they need without any modifications to the data analytics software. If software changes are required, then customization introduces a higher probability of error. Therefore, an analytics package that has a high proportion of the utility’s required analyses already built in is likely to be the best option.
  • SaaS approaches to data analytics enable a utility to focus on its core business, mitigate talent risks, deploy or decommission pilots quickly, and can be accomplished with little or no CapEx.

6 About Bob Lockhart

Bob Lockhart is a Gigaom Research Analyst and a principal analyst at Tractica. Previously, he was a research director at Navigant Research, leading the group’s Smart Utilities program. His personal research focuses on cyber security markets, meter data management markets, and smart grid managed services.

Lockhart is a frequent speaker, blogger and author, having recently published market forecasts and analyses on smart grid cyber security, industrial control system security, and meter data management. He has extensive leadership experience in the global technology and outsourcing industry, having supported more than 200 multinational clients. He spent 31 years at EDS, 17 of them in information security management. His career encompasses support for smart grid, manufacturing, aerospace, government, financial services, outsourcing, and healthcare industries in both the United States and Europe. Lockhart has served multiple roles, including chief security officer, software manager, security portfolio executive, solution architect, and data center manager.

7 About Gigaom Research

Gigaom Research gives you insider access to expert industry insights on emerging markets. Focused on delivering highly relevant and timely research to the people who need it most, our analysis, reports, and original research come from the most respected voices in the industry. Whether you’re beginning to learn about a new market or are an industry insider, Gigaom Research addresses the need for relevant, illuminating insights into the industry’s most dynamic markets.

Visit us at:

8 Copyright

© Knowingly, Inc. 2015. "Smart Grid Data Analytics: A New Approach for Utilities" is a trademark of Knowingly, Inc. For permission to reproduce this report, please contact