ESG reporting and the importance of getting the data right

Doing your best and acting in good faith when putting together an environmental, social, and governance (ESG) report are crucial to building trust. Unfortunately Pure, in their most recent ESG report, appears to have published unsubstantiated and unrepresentative comparisons with an unnamed competitor. On the face of it, these comparisons are so unlikely that it’s hard to assume that the company is acting in good faith.

This ESG report infuriates me, because as I type this, large swathes of northern Europe are aflame. Summer temperatures are breaking records, and scenes that I came to accept growing up in some of the more fire-prone regions of Australia seem bizarre and almost surreal when seen as part of the English countryside. Many people believe that these conditions are just a part of a natural cycle of weather events. However, that belief is not reflected in significant sections of the investment community. As a result, ESG investing has become, in the words of The Economist, one of “the hottest trends in finance.” In fact, “the titans of investment management claim that more than one third of their assets, or $35trn in total, are monitored through one ESG lens or another.”

Unfortunately, there is a fair amount of debate about the validity and comparability of ESG efforts. With such large amounts of money involved, some less ethical companies try to game the system, or to misuse the metrics. This practice undermines the trust that ESG reporting is meant to create, the very trust we are trying to build, to make the world a better place. I know that sounds corny, or like the tagline to a TV program, but it's true. At least for the people I work with at NetApp; we want to leave this place better than we found it, and the ESG report is one of the tools we use to do that.

NetApp has a long history of measuring our efforts to improve in all three of the ESG areas. One of the goals of NetApp’s founders was to “build a model company,” and it is often said that you cannot manage what you cannot measure. As a result, a high level of intentional accountability is built into the NetApp culture. NetApp isn’t perfect, but I’m proud of the work we do in continuing to build that model company after 30 years and to hold that work up against the work of our peers. Although it’s tempting to use ESG reports to compare companies, there are good reasons why it should be done with great caution. Each company is unique, each faces different challenges, different supply chain environments, different risks, and different ways of collecting data, all of which make direct comparisons almost impossible with any validity.

Which brings me back to Pure, and their most recent ESG report. Specifically, Pure claims that an //X70 storage system consumes 61,917 megajoules of power per year, while their unnamed competitor’s array consumes 405,360 megajoules. Both of these figures include roughly 30% overhead for cooling, which would equate to a competitive all-flash array that requires around 9KW of power. As an indication of how unlikely it is for that to be a fair or representative comparison, a NetApp^® AFF A800 all-flash storage system delivers significantly better performance and scalability than an //X70 but consumes an average of only 1.4KW of power when filled to its maximum internal capacity.

Even though Pure’s comparisons are suspect, they take these figures and dumb them down to “84% Lower CO2 Emissions—Pure is responsible” at the top of their main webpage. That 84% figure appears to be entirely unrepresentative of the industry at large, and therefore these claims undermine everything that a responsible ESG disclosure should be about.

I’m not saying that Pure doesn’t try their best to reduce their environmental impact. Good people work there, some of whom I consider to be my friends. But calculating lifecycle emissions is difficult, and the way it’s done is unique to each company, due to a wide range of factors including differences in supply chains, business practice, and more. ESG reports like the NetApp report aim to help us identify what we can do better and to track our progress against these important sustainability objectives. For example, Dell understands this in their outline of what can and cannot be inferred from their life cycle analysis (LCA) and product carbon footprint (PCF).

• What is streamlined LCA intended for? In general, LCA and PCF can be used to:
  • provide a reasonable estimate of the range of carbon impact of a product.
  • identify the major drivers of impact, or carbon intensive hotspots in the lifecycle of the product, thus identifying where reduction strategies should be focused.
  • estimate changes in the environmental impact of a process or product over time.
  • spark conversations related to sustainability with suppliers or to innovate new processes/materials uses
• What streamlined LCA cannot be used for? In general, LCA and PCF cannot be used to:
  • compare results of ICT products head to head (i.e. laptop A against laptop B) in terms of carbon impact.
  • compare products between other manufacturers.
  • aggregate individual LCA/PCF studies to derive a figure for the total carbon impact of ICT and use this figure as a benchmark for reduction targets.

When several colleagues and I became aware of Pure’s 84% claim, we were suspicious, so we dug into the data. After a thorough review, it appears that Pure intentionally selected proof points.

For the first part of this review, we double-checked our own power consumption figures by running an audit across the tens of thousands of pieces of hardware that report back to NetApp for preventive maintenance in the NetApp Active IQ^® management and alerting infrastructure. Then we checked to make sure that what was reported by the software matched power draw figures from equipment in our own labs and data centers. We were pleasantly surprised to find that on average, our equipment was about 30% better in terms of power draw than the conservative theoretical estimates we use in our sizing tools. Those updated figures will be posted on our product pages soon, and until then the updated data is available on request.

The next part of our review was to figure out what we considered to be a fair comparison based on directly comparable information and to see how NetApp and Pure stack up for our primary offerings. The most obvious option was to compare how much power was needed for a given unit of raw flash capacity. We could have used “effective” or “usable” figures, but these are highly workload dependent, and once we start talking about using a full range of data and power minimization techniques such as FabricPool and temperature-sensitive data reduction, the comparisons become far less direct.

If we had wanted to intentionally select data to prove the point, we could have done so easily, because NetApp allows far more capacity per controller than Pure does across the range. But although some customers do have systems at the absolute maximum of their scale-up potential, that’s hardly what I would call a representative configuration. A fairer, more representative comparison would be to look at the power draw from a system with either internal drives only for integrated systems like the NetApp AFF A250, or a system with one or two shelves, which is a fairly typical configuration for a modular all-flash array like the NetApp AFF A400 and AFF A900. Even though I've used the largest media size in the following figure, the relative numbers don’t change significantly for similar capacity points across the range when using smaller drive sizes.

When we do these kinds of representative comparisons of typical configurations, the //X70 does quite well, thanks primarily to supporting a large maximum capacity within a single enclosure. It’s important to note, however, that in terms of power draw, the NetApp AFF A800, which is the array that we typically see positioned against the //X70 and the //X90, performs better than both. And the AFF A250 smokes everything in Pure’s entry level, from the //X10 all the way to the //X50.

Keen observers might note that I have not included the numbers for QLC-based arrays, including the FlashArray //C or the NetApp FAS500f. This is because Pure does not appear to disclose their nominal or peak power draw figures for the //C.

The next question might be whether we are really comparing apples to apples. A cautious observer might wonder if the AFF A800 draws less power than the //X70 and //X90 simply because it isn't as fast or as scalable. The following figure looks at that possibility.

Based on public benchmarks from Pure and NetApp, the AFF A800 easily outperforms a Pure //X70 or //X90. And in terms of scale-up capacity, an AFF A800 has 782TB of raw capacity with direct-attached NVMe shelves, or 3672 TB with switched shelves. I don’t have performance numbers for the smaller Pure controllers. However, I’m quietly confident that that AFF A250 is similarly superior in terms of performance and capacity.

Perhaps Pure got carried away and made an honest mistake in their ESG submission, so we should give them the benefit of the doubt. But it’s clear from the figures above that Pure’s claim of 84% lower greenhouse gas emissions isn't true of NetApp. And based on what I know of Dell, HP, and IBM’s midrange offerings, it almost certainly isn’t true of them either.

I’m calling on Pure to either justify their claim and prove that they’re acting in good faith or to remove it from their website and cease using it immediately across all their marketing and communications. If they continue to use figures from their ESG report that almost certainly misrepresent their position in the industry, then the community should be made aware that they are going against the purpose of the ESG, which is about behaving ethically and responsibly.

I believe that when shown the facts Pure will act in good faith, and I hope that we can all do our part to keep ESG reports a force for good in this world rather than turning them into misleading marketing vehicles.