The main reason I enjoy researching what machine learning models have understood is that it feels like communicating with intelligent aliens.
They don’t resemble us, they don’t think like us, but they have solutions to our problems. That’s a fantastic starting point for research. Not everyone who develops machine learning models considers XAI as important as I do, and that’s completely fine.
You can certainly use a machine learning model that performs well without knowing what information it has encoded or the assumptions it has made. Lamentably, even the most harmless-seeming machine learning models can end up extracting the kind of knowledge that we don’t want them to use. I fully realized this as I was reading a publication written by Facebook researchers in 2018.
The publication is called “Facebook Language Predicts Depression in Medical Records,” and the researchers noted that up to one-quarter of the American population experiences depression at some point in their lives, yet fewer than half of them receive treatment. (The exact numbers from the study are respectively 7–26 percent experiencing depression and 13–49 percent receiving treatment. See Eichstaedt, Johannes C. et al.: “Facebook Language Predicts Depression in Medical Records,” Proceedings of the National Academy of Sciences of the United States of America, PubMed, October 30, 2018.)
With depressive disorders expected to become the leading cause of disability in developed countries by 2030, it’s clear that something must be done. The researchers concluded that the current procedures for identifying and diagnosing people with depression are not sufficient and that we need new ways of thinking about the issue. It’s easy to agree with this conclusion; however, deciding exactly which new methods we should develop is more challenging to agree on.
Facebook’s researchers presented the following option. They obtained consent from patients with depression to analyze their behavior on Facebook, including text that the patients had written, such as status updates and comments. Based on this data, researchers trained a machine learning model that identified correlations between language use and depression. Since the patients had already been diagnosed with depression, the researchers could compare the model’s predictions with the actual diagnoses. Remarkably, by using historical Facebook data, the researchers managed to identify signs of depression a full three months before the patients received their first diagnoses.
Feel free to read that last sentence again: A machine learning model developed solely on language data from social media—that did not have access to information about the person’s feelings, their medical history, or anything else that a doctor might consider relevant—managed to detect depression three months earlier than the individuals themselves or their doctors did. This publication was not the first of its kind; prior to 2018, language data from both Twitter and Facebook had already proven helpful in predicting depression, suicidality, and post-traumatic stress disorder, but those studies had been based on the participants’ self-reported mental states. The 2018 study was based on clinical diagnoses that were made after the point when the behavioral data was collected.
For me, this study was a turning point, because I realized two important things. The first was that data that seems totally innocent—like a Facebook status or a tweet—can still contain intimate details about our mental health when viewed in context, such as alongside status updates and comments we’ve written. Second, I realized that knowledge about mental health can be represented in a machine learning model without us even knowing it. A massive collection of numbers, apparently impersonal and meaningless, can contain insights about us that even we aren’t aware of.
Regrettably, we are seeing fewer and fewer publications like the one I read in 2018 from Facebook. It’s not likely that this is because they’ve stopped conducting this kind of research but may be because companies are noticing how negatively most people—among them politicians—react when such research is conducted. Of course, this is just speculation on my part; however, it’s still hard to say just how much the different social media platforms know about us. In my opinion, it’s safest to assume that they know more about us than we do—and then some.
Regardless of the assumptions we’ve made, we should be having three discussions: The first is about how we should relate to tech companies that possess information about their users’ mental health. The second is about how we can use this powerful technology to make our lives better. The third involves my belief that we need regulations requiring anyone developing machine learning models that interact with humans—such as those used in personalized advertising—to examine the concepts their models have learned. Just like being “in check” is a relevant concept for a chess player, “poor impulse control” is a relevant concept for a system that generates personalized marketing.
Sadly, all three of these discussions are so complex and, to some extent, so abstract that they rarely make it into public discourse. It’s much easier for politicians to spend an entire campaign season arguing about tariff barriers than to discuss how tech companies’ algorithms might be using knowledge about our mental health to influence our behavior. At the risk of sounding a bit self-indulgent, I still believe that all three topics will shape our future. And as long as we avoid these discussions and fail to make deliberate decisions, these decisions will either be made by the leaders of tech companies, or they won’t be made at all. I don’t know which is worse.
What I do know is that these discussions revolve around several very different areas: technology regulations, data sharing practices, and the development of methods to better understand machine learning models. Methods for enabling humans to understand machine learning models are continually being developed. As for the two others, some promising news is that the European Union (EU) is in the process of implementing broad regulations for artificial intelligence. In August 2024, the EU passed a comprehensive legal framework called the Artificial Intelligence Act.
Editor’s note: This post has been adapted from a section of the book Machines That Think: How Artificial Intelligence Works and What It Means for Us by Inga Strümke. Inga is a Norwegian physicist specializing in artificial intelligence and machine learning. She was born in 1989 in Gummersbach, Germany, and grew up in Narvik, Norway. Strümke holds a master's degree in theoretical physics from Norwegian University of Science and Technology (NTNU) and a doctorate in particle physics from the University of Bergen. She is currently an associate professor at NTNU. Strümke is also known for her work in AI ethics and has received an award for science communication from the Norwegian Research Council. She published Maskiner som tenker in 2023. The book was recognized with the Brageprisen, a prestigious Norwegian literature prize.
This post was originally published 12/2025.