ETH Zurich’s ‘ClimateBERT’ discovered that companies may be guilty of “cheap talk and cherry-picking”
It’s boom time for corporate climate disclosures — or so it seems. Today, there are over 1,800 institutions signed up as supporters of the Task Force on Climate-related Financial Disclosures (TCFD), up from 101 in June 2017. Adherents to the framework are also allegedly publishing more climate-related information. The TCFD’s latest status report said that disclosures aligned with its 11 recommendations increased by six percentage points on average between 2017 and 2019.
However, appearances can be deceiving. The TCFD’s analysis was designed to assess the presence of disclosure — not the quality of disclosure. Obviously, the mere act of disclosure is no guarantee that stakeholders will be able to glean decision-useful information on a firm’s climate exposures.
It’s true that evaluating the actual substance of corporates’ climate-related information is tough work (I should know). But that didn’t stop academics at Swiss research university ETH Zurich and Uni Zurich, who recently conducted an innovative and wide-ranging analysis of companies’ TCFD disclosures using a specially-trained natural language processing (NLP) model. Their findings were published in a working paper on Tuesday.
What’s special about the ETH Zurich approach is that it overcomes two big challenges linked to assessing TCFD reports: how to do so objectively and at scale. Since the discipline of climate disclosure is in its infancy, reporting is heterogeneous: each firm has its own reporting style and syntax, and what makes for a “good” disclosure in one industry may be unhelpful or confusing if copied by another. Teasing out the relevant information from company reports is time-consuming, and determining its usefulness without bias is tricky for human evaluators.
Enter ‘ClimateBERT’, the model used by the ETH Zurich researchers, based on the Google-developed Bidirectional Encoder Representations from Transformers (BERT) NLP framework. This model was “trained” to recognise TCFD-related content from hundreds of sentences extracted from company annual reports, standalone sustainability disclosures and webpages. It also learned to identify relevant sentences by TCFD sub-category: governance, strategy, risk management and metrics and targets.
Importantly, ClimateBERT uses a context-based algorithm, which allows it to differentiate between TCFD-specific content and non-climate-related information on governance, strategy and risk management. This makes its findings more nuanced and reliable than those produced by statistical language models, which typically hunt for keywords in text without regard for context, and therefore lack precision. The TCFD itself used a statistical language model for its own recent analysis.
Once trained, ClimateBERT was unleashed on a sample of 818 TCFD-supporting companies’ annual reports for fiscal years 2014 to 2019.
What it found may make for uncomfortable reading among the TCFD secretariat:
Our results show that supporting the TCFD seems to be cheap talk and is associated with cherry-picking disclosures on those TCFD categories containing the least materially relevant information. We observe a slight increase in the information disclosed as required by the TCFD categories since 2017, relative to the periods before the TCFD recommendations launch. Disclosures on strategy, and metrics and targets, are particularly poor for all sectors besides energy and utilities.
This conclusion undermines the idea that the TCFD recommendations have had a discernible effect on corporate disclosure. Indeed, the researchers argue that the small, 1.9 percentage point increase in disclosure since the launch of the recommendations in 2017 “suggests that TCFD supporting firms might not have increased their level of disclosures. Instead, they might simply have re-structured already existing information such that they comply with the TCFD recommendations”.
However, these findings need to be put in context. The paucity of disclosure identified by ClimateBERT may not all be due to cheap talk and cherry-picking. In fact, the roll-out of the TCFD recommendations preordained a low level of disclosure in its first years. That’s because the group prioritised widespread adoption of its recommendations over in-depth disclosure. It figured that after lots of firms started reporting some amount of climate-related information, organisations’ and stakeholders’ understanding of these issues would grow, which would eventually lead to disclosure of more decision-useful disclosure down the line.
The group even laid out a five-year implementation pathway with “more complete, consistent, and comparable information for market participants” expected late in the process. Taking their cues from the TCFD, then, some organisations likely opted to take a conservative approach with their first disclosures, deferring the tough stuff for later on.
Source: TCFD
There’s another plausible explanation for the patchiness of disclosures. It’s simple: large companies roll out strategies for the long-term, and it’s hard to shift them in short order. An agreed-upon strategy has management buy-in and institutional support throughout the organisation. In many cases, success fulfilling the strategy is linked to employees’ and directors’ compensation. Reconfiguring a strategy to accommodate climate issues, therefore, requires an organisational shake-up and — you guessed it — effective governance. This dynamic may in part explain the paucity of strategy disclosures and overweighting of governance disclosures in the ClimateBERT sample.
More broadly, it’s possible ClimateBERT’s limited scope influenced its findings. First of all, the researchers only loosed the model on firms’ annual reports, leaving out standalone sustainability or TCFD-specific documents. Why? Because they wanted to assess the degree to which climate-related information was being included in mainstream disclosures.
This reasoning makes sense, but the fact is many firms haven’t quite figured out how to incorporate their climate-related information into the structure of an annual report. The fear of “polluting” what constitutes the most highly-scrutinised document they issue may explain the scant amount of TCFD-specific information ClimateBERT found within these filings.
Second, ClimateBERT is only capable of assessing text in the main body of annual reports. It therefore ignores charts, tables, and infographics — which are often rich sources of TCFD-relevant information. Take Commonwealth Bank of Australia’s 2020 annual report. It includes the following series of infographics detailing its exposures to the energy sector:
Source: Commonwealth Bank of Australia
This detailed, quantitative information would likely have been overlooked by ClimateBERT and therefore not factored into Commonwealth Bank’s disclosure “score”.
A fair point made by the researchers in response to Climate Risk Review is that any forward-looking information — especially in relation to the TCFD strategy and metrics recommendations — should include explanatory remarks in the body text. Otherwise, the graphical data is at risk of misinterpretation. Put another way, disclosing graphics without accompanying text is half a loaf, and indicative of an immature approach to disclosure.
These questions on ClimateBERT’s scope notwithstanding, the ETH Zurich paper offers plenty of food for thought. It’s also hard to argue against their most important conclusion — that the TCFD’s voluntary framework isn’t doing enough to get climate-related financial information out into the world. The most compelling evidence supporting this is the regional breakdown of the ClimateBERT findings. This shows that France — the only country where detailed disclosure of climate information is mandated by law for financial institutions — has a “much higher level of overall disclosures”.
To truly get climate disclosure booming, then, lawmakers will have to flex their legislative muscles.
