The buildout of smart grid infrastructure is about to drown utilities in a sea of information. So points out Jack Danahy in an interesting article in Smart Grid News recently, in which he presents data that shows how much information a typical smart meter will produce. If 140 million smart meters are installed over the next ten years they could produce a massive 100 petabytes (1 PB is 1 quadrillion bytes) of information, according to FERC data he cites. Info from utility trials suggests that a smart meter that updates energy info every 15 minutes can deliver 400 MB per smart meter per year. For perspective, Google processes about 20 PB of data per day, and 1 PB is equivalent to the amount of data contained in 20 million four-drawer filing cabinets filled with text.
On a grand scale, and for individual utilities, the idea of collecting, processing and storing this much data is daunting. While 400 MB is less than an average CD worth of data, multiply that amount by hundreds of thousands, and even millions, for each smart meter for each utility. The result is that the utility sector will need to invest a very sizable amount into data storage infrastructure and information management programs.
First off, that means utilities will be purchasing a lot more memory and storage capacity — in other words a place to house all that smart meter data. Danahy says that California utility PG&E has added about 1.2 PB of memory for 700,000 meters that will update twice a day and could produce 170 MB per meter per year, and plans to add more capacity for the rest of its buildout. For the case of Austin Energy’s 500,000 meters, the utility’s yearly data storage needs grew from 20 TB to 200 TB.
Utilities will also increasingly be investing in traditional data centers. Cisco predicts that utility spending on traditional data centers could make up “a large chunk” of the potential $20 billion that will spent on the smart grid over the next several years, reported CNET’s Martin LaMonica reported earlier this year.
On both fronts, utilities will be looking to the computing industry’s memory, storage and data center leaders for products and services. Any company that has built a business off of storing, securing and managing data would be smart to launch a utility-focused service or division. In addition entrepreneurs interested in jumping into this soon-to be-booming market should consider what unique needs utilities have compared to other sectors and target services to fill those needs. Security and privacy will also be major areas of growth for smart meter energy information.
Utilities will have a range of information storage and management needs. Small rural utilities and large investor-owned utilities will have different levels of storage needs, and the information unearthed from smart meters will also depend on how sophisticated those meters are. As you can see from the earlier example, smart meters that update energy info every 15 minutes, compared to twice a day, produce more than double the amount of energy info.
Through investment, utilities will be able to ultimately manage the flow of information, but will need to pay very a close attention to the practices that the computing industry has built, including keeping data safe and secure.
Do the utility companies need to keep all this detail for years?
Or do they need to store only some summary statistics?
[...] Smart Grid Data About to Swamp Utilities . [...]
that was my second question after re and re reading so I could get a direct compare between “smart” meters and google and see the smaller footprint of meters. still big. but how significant after the moment when agregated to give readings to plot time of day, season etc. It seems the data should condense very fast, considering only the “utility” value. Of course there is the psychlogic and legal and “because we can” dimensions…..
and it leaves me with a question. Will the storage and processing make up for the avings? oh ironic day.
In other fronts I read that some utilities will be “compensated” for loss of revenues thst result from customers using less. Oh ironic rip off. Rate payers paying to compensate non using…I wonder if I owe money to a mafia dealer I nevereven met or used!!!!!
[...] Smart Grid Data About to Swamp Utilities Posted October 13, 2009 Filed under: Uncategorized | http://earth2tech.com/2009/10/12/smart-grid-data-about-to-swamp-utilities/ [...]
In California, and perhaps other states, utility regulations were rewritten to allow utility companies to continue to profit while selling less power. This puts the utility company in the business of selling conservation.
We get special discounts that bring us $0.50 CFLs and rebates from the utility company for replacing inefficient appliances with less power-hungry models.
Remember that ‘new’ power is more expensive than power from paid off nuclear plants, dams, etc. If we use more power and have to build expensive ways to generate that power we will all pay more per kWh. Our bills will go up.
So the end result might be that the utility company continues to make money and we use less power and our total electricity bill doesn’t rise.
I’m fine with that.
Is your utility company making more money if you use more power?
If so, that’s not helping us quit the coal habit….
Thinking more about the data storage issue…
All that is needed is aggregate statistics. There is no need to store data from 2230 Second St. every fifteen minutes and data from 2231 Second St. every fifteen minutes and data from 2232 ….
Power companies know the data that matters. They can aggregate the data as it arrives and toss most of the detail.
And they can store a representative sample of individual records just in case a new question arises in the future. They won’t need to know the usage behavior of an individual user, just the profile of users in general.
And the detail data does not need to be stored on spinning storage. Historical/detailed data can be stored on parked disks (replicated and checked periodically) and only spun if someone wants to pull out the details for future analysis.
I suspect Jack just got a bit over concerned….
Actually aggregated statistics are not sufficient for a variety of expected smart grid functionalities.
Whether it is power generation coming into the grid (remember one house won’t be an issue, but a community of wired homes might), or the presence/absence of PEV’s at public locations, or simply a finer granularity to better balance power, the individual readings matter.
And my concerns are not just about maintaining the data, it is receiving it, analyzing it and attributing it. As an example, if a customer wants to contest billing, claiming that they are maintaining a particular power profile (Solar on, off-hour use, PEV), their meter data will need to be saved with their authenticating information, if the utility is going to be able to manage a credible and non-repudiable view of their usage.
Also, pulling, spinning, and retrieving out of even a couple of petabytes will already be a challenge. These organizations, utilities, have not typically dealt with anything like these volumes, and I’m not a big fan of hoping for the best.
Let’s start with the separate uses of collected data.
1) Billing. On an individual household basis, I can’t see the need to store detail past the amount used per “peak/off peak price” and the amount sold back to the utility at different price points. All the detail can be discarded.
If an individual customer suspects that his/her meter is malfunctioning then there can be a process to add a second recording device and a refund can be paid if proved appropriate.
Even if all the details do need to be stored, they don’t need to be stored in perpetuity. Give customers a window to appeal (60 days, 90 days) and then toss the detail.
2) System control. The grid controllers need aggregate data, not specifics. And once the control decisions have been made (fire up a turbine, shut down a turbine, etc.) all that needs to be stored is some summary data as to what triggered the decision and how well it worked. A year or two out that data can probably be further compressed to an even smaller form.
3) System planning. Again, no need for huge detail. What is needed is profiles and behavior patterns. This just takes good sampling skills, the sort of things that pollsters and scientists do all the time.
Handling the data. “Neighborhood aggregation” – feed data streams from defined portions of the grid into separate processing systems. Feed only the summary statistics to control centers and send the billing/history data off to storage. Grid controllers don’t need that data.
The idea that utility companies would need to store and sift through years and years of petabytes of data just doesn’t make sense to me….
RE: Mr. Wallace’s comment that utilities know the data that matters. I’d say that’s been true in the old model, but the advent of AMI, 2-way comms and net metering sets the stage for a wave of applications (see: Soft Grid). And those apps will often seek to leverage fine grained data in ways that benefit consumers, utilities or both. So in this case, the past may not be a good indicator of future data needs.
Andy – I think I addressed most of that in my reply up the page.
There’s a difference between saving too much data (every single 15 minute reading/whatever from every single meter) and not saving enough.
Remember, here’s where this discussion started…
“If 140 million smart meters are installed over the next ten years they could produce a massive 100 petabytes (1 PB is 1 quadrillion bytes) of information,….”
It’s all about sampling and summary statistics.
400MB/meter/year works out to about 33/MB/month. That’s a lot of data usage per month. What are the estimates for data plan costs per month to support a typical smart meter?
Cost to store a month’s data (if there was any reason to store all that data)?
Let’s see. They could go to NewEgg and get a 1.5TB hard drive for about $110.
33MB = 0.00003142 TB.
Could store the data from 47,740 customers on one 1.TB HD.
Cost would be $0.0023 per customer per month.
(Warning: Morning coffee deficient math.)
That said, why would the utility company want or need to store all that detail?
Sorry, I meant data plan costs over the carrier’s network. If a smart meter is transmitting 33MB/month that’s got to be quite expensive. Data aggregators such as Kore and Jasper will usually charge about $7/month for 1 MB plan.
Let me first clearly state – I know “jack” about the cost of transmitting data.
That out of the way – “$7/mo for 1 MB”?
Just seems extremely high. There’s something that I’m not understanding.
I’m looking at a wireless internet connection to replace my satellite ISP. For service over the basic fee/5 gig use limit the company that I’m considering charges $2.50 per gig.
Bob, you’re right. I’ve been working too long with mobile devices that use dedicated GPRS and CDMA data plans. (sorry)
(Again, posting while possessing very little actual knowledge….)
Why wouldn’t power companies simply send their data over the power lines? Broadband over power line is established technology isn’t it?