The Octopus Writes My Emails Now
There was this one week where everyone asked me about ChatGPT. First it was friends from my Ph.D. days, which is nothing unusual: we often talk about the hype cycles that roll through our field. Next, people at work brought it up: Have I looked into it yet, can we use it for our own machine learning problems? A few days later I was mixing colours from pigments at my weekly art class, and the woman next to me confided that she was mixing a plum shade of purple because ChatGPT had told her to do so. “I feel sorry for ChatGPT”, she said “Being trapped there, and people teasing it and playing with it all day, even though it has access to all of these higher dimensions.” I smiled awkwardly and fought down my urge to explain what ‘high-dimensional vector space’ means in machine learning, because I want to be the kind of nerd that still gets invited to things.
My emotional tipping point came in a Skype call with my mum, who informed me that my 84-year-old grandmother had asked about ChatGPT, and isn’t that my specialty? I wanted to sigh and roll my eyes and say it’s not that big of a deal. But clearly it is, because all of a sudden my field of research moved from existing in academic papers and conferences to being discussed in international media. And the further away I step from the NLP crowd, the more often I get the question: Can we trust ChatGTP? What does it know?
If the person asking the question is sitting comfortably or has specifically invited me to do a presentation about the topic, I bring up the paper “Climbing towards NLU”(1) by Bender and Koller, which won an ACL best paper award in 2020. In this paper the authors propose a thought experiment, which exposes a critical weakness of large language models. It applies to ChatGPT just as easily as to the models that were state of the art in 2020. (Those of you familiar with the experiment are welcome to skip ahead.)
Imagine two people, A and B, stranded on two separate deserted islands. There is no way for them to communicate with the rest of the world, but the previous inhabitants of the island left an under-sea communication cable behind, that allows A and B to send text messages between their respective islands. A and B love sending messages back and forth, and do it all the time. At some point a hyper-intelligent deep-sea octopus discovers the cables. It finds a way to listen in on the conversation. Being an octopus, it doesn’t understand human language or concepts that pertain to living on land. All it can do is to perceive and learn the statistical patterns of the messages. At some point listening is not enough for the octopus and it cuts the communication line and starts to answer B’s messages, impersonating A. At what point will B notice that she is not talking to a human any more?
Let’s imagine B writes something like “What a nice sunset!”. The octopus has never seen a sunset and doesn’t even know that the word connects to a specific concept. But it observed that A normally answers statements like this with analogies, so the octopus might answer: “Reminds me of a lava lamp.” And B would probably fall for that.
Let’s imagine another scenario in which B figured out how to build a coconut catapult and writes something like: “I build a catapult, it works like this:...” The octopus would again have no clues what a catapult is and would not be able to reproduce the contraption at the bottom of the sea, or give helpful advice on how to improve it. Because the octopus observed that messages about building things were often met with encouraging replies, it could answer “Well done!”, and B would still be none the wiser.
The deceit collapses in the following situation: All of a sudden B discovers that there is a wild and aggressive bear on her island. In a desperate attempt to save herself she messages: “Help, there is a bear! All I have is a stick, what should I do?” The octopus, having no concept of the word bear, the associated dangers or the functions of a stick, would likely give an unhelpful answer, just before B is eaten.
What does this analogy have to do with ChatGPT? Just like the octopus, ChatGPT only ever sees written text, without any connection to the actual world (something NLP people call extralinguistic reality). Just like the octopus, ChatGPT is good at reproducing what it has previously seen, but it doesn’t have a working model of the real world that motivates its decisions. The reason why B on the island and my grandmother on her computer feel like they talked to a person, is that humans are wired to see an eloquent answer as evidence for a shared view of the world. Humans assign personality to potted plants and roombas, so assuming intelligence from something that fluently uses language is not a far step.
But does it even matter that ChatGPT does not connect words to extralinguistic realities? Asked what to do with a stick in case of a bear attack, it gives reasonable answers, possibly because it has seen similar cases in its training data. If it has seen all the text available on the internet, could we theoretically assume that it would give us trustworthy answers to all our questions? Or, to use another analogy: Would we accept medical advice from a doctor who has read all existing medical literature but has never seen a human being, has no empathy and with whom we can only communicate via text?
I shared bagels and iced tea in a cafe with a friend. I was on holiday and she was taking a break from writing her master thesis proposal in sociology. She researches migrant organisations and democratic participation, and she told me that out of curiosity she gave ChatGTP the prompt of writing the research proposal for her. “It was okay”, she said. “But I didn’t like it. It sounded somehow… hollow?” I wonder if this is due to the fact that ChatGPT can only learn from text already written about the topic. The objective of a research proposal is to come up with a question that adds new knowledge to a scientific field. It shouldn’t be a mere recombination of things that are already known. Is this one way in which the lack of extralinguistic reality in these models shows up, and would models that include audio and video data do better? But then again, video and audio data are just another way to encode information, and not an actual link to the real world.
Large multimodal language models are being released, most prominently GPT 4. In a recent tweet Emily Bender, one of the authors of the octopus paper, stresses that without trancparency about training data and model architecture the model isn’t trustworthy. Interacting with it sets a dangerous precedent. It has been said before and it bears repeating: Large Language Models are no replacement for doctors, lawyers and psychotherapists. And, as I pointed out in a previous post(2), often enough they won’t even say gay.
This post was written by Sabine Weber. Sabine is a queer person who just finished their PhD at the University of Edinburgh. They are interested in multilingual NLP, AI ethics, science communication and art. They organized Queer in AI socials and were one of the Social Chairs at NAACL 2021. You can find them on twitter as @multilingual_s