Investigating public perceptions of climate change using ClimateQ&A

Introduction and context

To communicate effectively about climate change and to design public policies aimed at mitigating and adapting to it, it is crucial to understand public perceptions of this phenomenon. Researchers seek this understanding by quantifying two main dimensions: climate awareness and climate literacy. Awareness refers to concern about and belief in climate change, while literacy encompasses the knowledge, attitudes, and skills that lead to informed decision-making and action. Climate awareness and literacy vary across geographical regions and demographic groups and correlate with individual factors such as physical exposure to the phenomenon and educational attainment.

A large body of research is investigating awareness and literacy across different groups. The most comprehensive and international studies include the Sulitest and The People’s Climate Vote. Many others focus on specific interest populations around the globe, ranging from national populations to age groups and rural versus urban populations. These studies often analyze the group’s perceptions across different sub-dimensions of awareness and literacy, such as the causes and effects of climate change, energy, economy, transportation, and agriculture. In these surveys, response categories are pre-determined and include a range of textual answers or incremental values through which respondents express their fear or concern about, interest in, and opinions on these topics. For example:

“How serious of a threat is global warming to you and your family?” Response categories included: Not at all serious, Not very serious, Somewhat serious, and Very serious.

This survey method allows for the comparability of answers among individuals within a group and among different interest groups. However, by pre-determining topics of interest and standardizing answers expressing individuals’ perceptions of climate change, this type of analysis may not identify all concerns and interests that are specific to each group and may not identify knowledge gaps critical to understanding climate change awareness and literacy, which in turn are essential to inform outreach efforts.

Our paper

We propose a novel method to gauge the public perception of climate change by analyzing the questions posted to ClimateQ&A, a tool that Ekimetrics’ innovation lab developed to make IPCC and IPBES reports more accessible. The tool offers a chatbot interface that gathers over 14,000 pages of knowledge from the two institutions and uses LLMs to generate answers adapted to the user’s language and level of understanding. The answers systematically quote sources to trace content back to the original document.

Our method consisted of extracting a sample of the questions posted to the platform to analyze the users’ topics of interest and concerns. The originality of our approach stems from the fact that these topics and concerns are captured with the questions that users can formulate freely, with the ‘space’ to express themselves in an open chat.

To perform our analysis, we first removed from the sample the questions suggested by the platform and the ones that were too short to identify relevant topics. We used classical open-source NLP techniques (BERTopic) to categorize questions across 130 salient topics. We conducted two different analyses by manually reviewing each cluster:

• Topic analysis to derive a classification of general topics and sub-topics across climate change, biodiversity loss, nature, economics, society, and more.
• Intent analysis to identify whether each question expressed a personal inquiry, meaning elements of subjectivity that may relate to the user’s personal experience – geographical areas of interest (e.g., places where the user lives or goes on holiday), individual actions to fight climate change (e.g., transportation and food choices), individual concerns and feelings (e.g., climate skeptics, feelings of fear and anxiety), among others.

Results

In our sample, the most prevalent topic was Climate Change and GHGs (43.1%), well ahead of Biodiversity (2.9%). This may stem from the fact that the tool emphasizes climate-related queries (e.g., through the naming ‘ClimateQ&A’ and the suggested questions) and could also be attributed to the lower coverage of biodiversity in mass media and public discourse compared to climate change.

Another interesting finding was that our intent analysis revealed that up to 25.8% of questions express personal interrogations regarding nature phenomena as defined above, suggesting that users often want to know how they are concerned and affected personally by these phenomena.

These findings imply that social surveys aimed at understanding public perceptions of climate change and biodiversity loss may benefit from complementary alternative techniques where users can freely express themselves. This allows for the identification of topics of interest and personal concerns. They also imply that IPCC and IPBES reports, intended for policymakers, may not meet this demand for science-based information and may affect awareness and understanding of these issues. Therefore, these organizations could consider alternative communications efforts to engage with broader audiences.

Future work

To continue gathering insights on public perceptions of climate change, we are considering extending the sample of analysis, improving our topic classification, and tracking the frequency of occurrences of each topic across time.

Further, improving the tool on several fronts would be beneficial, for instance by including additional data sources from the two institutions.

Additional content

• Link to ClimateQ&A: https://climateqa.com/
• Learn how the IPCC is working to make reports more accessible to different audiences: https://www.youtube.com/watch?v=Rp0DEeIddbM (in french: special mention to ClimateQ&A at [8:10])
• Poster: https://s3.us-east-1.amazonaws.com/climate-change-ai/papers/iclr2024/3/poster.pdf
• Full paper: https://arxiv.org/pdf/2403.14709 with access to sources and references

‍