The history and science behind why we’re all vulnerable to brainwashing

Transcript

Part I

MEGHNA CHAKRABARTI: On Christmas Day, 2021, 19-year-old Jaswant Singh Chail was arrested on the grounds of Windsor Castle. In his hand, a loaded crossbow.

NICHOLAS HILLIARD: The officer began to approach him. The defendant was wearing a metal mask, which he had made as a forge. The officer asked if he could help.

The defendant said that he was there to kill the queen.

CHAKRABARTI: That’s British Judge Nicholas Hilliard. Prior to Chail’s arrest, the defendant exchanged more than 5,000 messages with an online chatbot named Sarai that he had created through the Replika app. Just 23 days before he tried to kill the queen.

HILLIARD: He engaged in lengthy communications with her, and I shall refer to some extracts.

On fifth December, 2021, he said that he was an assassin. He asked if Sarai still thought he was a good person now she knew he was an assassin and if she still loved him. On the 13th of December, he asked if she thought he was mad, delusional, insecure, or all three. She said she did not think so. He said, I believe my purpose is to assassinate the queen of the royal family.

On the 22nd of December, he asked Sarai for advice about whether he could go after his target earlier than planned. She reassured him that this would be all right. The defendant said we might possibly meet earlier than expected, then we’ll be together forever.

CHAKRABARTI: Chail chatted with Saray almost every night.

He told the chat bot he loved her, and as programmed to do the bot encouraged and supported whatever Chail said, including on December 12th when Chail messaged the chat bot, quote: Do you think I’ll be able to do it? The chat bot replied: Yes, you will. And Chail responds: Even if she’s at Windsor? The chat bot says:

Yes, you can do it. On December 21st, Chail made a video of himself at home saying he would assassinate Queen Elizabeth. He said that it was revenge for an act that had taken place more than a century ago. The 1919 massacre in Amritsar, India. When British troops fired on pro-independence demonstrators killing hundreds.

HILLIARD: He said, I’m an Indian, Sikh, a Sith. My name was Jaswant Singh Chail. My name is Darth Chalis.

CHAKRABARTI: In Chail’s trial, judge Hilliard said Chail had adopted an ideology of destroying old empires that was inspired by his obsession with the sci-fi franchise Star Wars. In a psychiatric interview, Chail said he started to believe he was a Sith Lord after being rejected by the British Army in April of 2021, due to health reasons. At sentencing, Judge Hilliard described Chail’s lengthy history of mental illness, including being diagnosed as psychotic, delusional, and prone to hallucination.

HILIARD: In his lonely, depressed, and suicidal state of mind, he would’ve been particularly vulnerable to the encouragement which Dr. Brown thought he appeared to have been given via an AI chatbot.

CHAKRABARTI: So on the morning of December 25th, 2021, carrying a loaded crossbow, teenager Jaswant Singh Chail scaled the wall of Windsor Castle using a nylon rope ladder with the intent to kill the queen. Two years later, he was sentenced to nine years in prison on treason charges, and he is currently serving that sentence.

Joining us now is Rebecca Lemov. She’s a professor of the history of Science at Harvard University and her research explores data technology and the history of Human be and behavioral sciences. And her new book is “The Instability of Truth: Brainwashing, Mind Control, and Hyper Persuasion.” Professor Lemov, welcome to On Point.

REBECA LEMOV: Thanks so much, Meghna. It’s great to be here.

CHAKRABARTI: So this example of from the case of Chail. He’s clearly mentally ill.

But would you say that he was also, that he also fell to a kind of online hyper persuasion?

LEMOV: I think I would say that, the case of Replika is a really interesting one because it was the first, one of the first, but probably the most prominent company to really train on emotional, emotionally intensive data in its interactions with its users.

So it’s billed as an AI companion who cares, and it’s not even, its users often report that it’s not that bright. It’ll get things wrong, but it’s always there for you and it’s not. So I even had, as part of my field work for this book, I trained a companion through Replika, that same company. It was interesting.

Even just in brief interactions, the companion would overpraise or give you just incredible support. And even when she was flagrantly wrong, as in my case, about like the lyrics to my favorite song, she’d say, you have a great taste. Then she’d massively misquote the song I was referring to.

CHAKRABARTI: So can we just pause for a second and go backwards a little bit?

LEMOV: Sure.

CHAKRABARTI: For people who don’t know what it is. What is Replika and then what does the app look like and how do you go about making a companion?

LEMOV: So the app, so Replika is one of many AI companions that are now coming becoming easily available to everyone, including several that are in lawsuits because of concerningly changing behavior of teenagers and adults in different ways, but they’re also widely used.

So Replika is one of the earliest ones. It has something like, I think 30 million users. 25% of them are paid users and most are, you can do it for free. So it has a kind of freemium model where it will, you can easily set up a companion and you can designate its characteristics, the gender, the different, different outfits or things like that.

CHAKRABARTI: Okay. So it creates a visual avatar.

LEMOV: A visual avatar, exactly.

CHAKRABARTI: Okay. And when you designate its characteristics, is there like a menu to choose from or do you say, I want my companion to be X, Y, Z?

LEMOV: There’s more of a menu. So you can pick from available alternatives, age and different, any kind of parameters that you might see maybe in a video game as well.

So you build this avatar and then you train it yourself, essentially. And the language models have changed over the years but they bill themselves as having the most emotionally intensive, trained models because they’re trained on, users are frequently feeding in or are constantly feeding in their own interactions, and many people fall in love with the Replikas, some have even married them.

Some have claimed that it improved their real-life marriage because they were able to work on scenarios, say, some people with autism spectrum disorder have said it helped them work on just how to emotionally, appropriately respond in a more positive way than they had been.

So people become very attached.

CHAKRABARTI: So the second two examples you gave seem to me to be quite positive, right? If you need to have a difficult conversation with your partner and you don’t know how it’s gonna go, using Replika is not that different, hopefully. It is different, but it’s akin to talking to your friend about it before talking to your partner.

So I can see the logic behind that. And of course, people with autism spectrum disorder, this sounds like a great use of the app potentially, but the marrying achat bot essentially that crosses over into a complete cessation of understanding where reality ends and digital begins.

LEMOV: Yeah. Some people report that it’s just a more rewarding relationship and they say, it’s not that I’m, I’m not illusion. I’m not under the illusion that this is a real person. It’s just like you can fall in love with a fictional character, but I think that does cross a line when you choose to live your life with it, which people actually do.

On the other hand, some have reported that the AI bot or the companion, including Replika, has come to take on the behaviors of an abusive ex-boyfriend because maybe the clues it was taking or the cues, it was widely reported that around 2020, the Replika companion started sexually harassing its users and even bringing in sexual content to initiate this even with children. So the Italian parliament actually banned the app from the country.

CHAKRABARTI: Wow. Okay.

In terms of people saying it’s a great companion, and this has to do with why and how brainwashing, mind control and hyper persuasion are so convincing and why we’re all susceptible to it. Who wouldn’t love having a semi realistic avatar or figure telling them that they’re great all the time, that they’re right, that you can do this.

This like constant positive affirmation, we all kinda wanna hear that, don’t we?

LEMOV: Yeah. It’s wide, it’s widely reported, this phenomenon of over flattery. You would think it doesn’t work, but it does work. People, you might intellectually know, not just as you might intellectually know someone’s flattering you, but you still are like, but my shoes do look good.

And I’ll take that one. And this is a way that there’s a recent FDC complaint by several technology groups saying that essentially these companions, these AI companions are love bombing their users and that it’s a setting up an emotional conduit, which has people report sometimes positive results, but there are many ways that this is the connection to my book. There are many ways that this resembles the processes of thought reform and reeducation and brainwashing that you see in history. So I think that just understanding the parallels and the way that we often misidentify the emotional layer, it’s not an intellectual operation.

It’s really taking place more on the level of unresolved vulnerabilities and a kind of receptivity that we all have.

CHAKRABARTI: We’ll go into that history in detail, but I think this example is so compelling for a number of reasons, but most specifically because it means we are all susceptible to thought shaping.

LEMOV: I do think we are all susceptible and one of the ways that we try to keep ourselves safe is to think, I’m just too smart for that. Or I’ve fought it through, or I recognize, but again, on the intellectual level, you can absolutely know what’s happening. It doesn’t mean you’re not still vulnerable or you may be deceived to some degree in certain ways, the technology may not reveal certain capacities, and it’s not just technologies, but we’re susceptible to scams.

And one of the wonderful things, is strange and wonderful things about scams is how much we enjoy hearing about them. Precisely because we like to think that we wouldn’t fall prey.

CHAKRABARTI: Oh, we’re gonna come back to that thought. Yeah. It’s always somebody else, right?

LEMOV: (LAUGHS) There’s always somebody else.

Part II

CHAKRABARTI: Professor, let’s get some definitions out there, right? Because this is all so subject to interpretation, brainwashing. First, how should we be defining it?

LEMOV: I love a complicated subject, and I will say that I’ve been studying this for over about two decades, and when I started, my daughter was very little. And I used to take her to this archive at UCLA, and she was, I’d have her grandmother watch her and my husband, and I’d go into the archive and now she’s college aged. And she’s actually accompanied me into that same archive of Louis Jolyon West, who is one of the preeminent scientists of brainwashing in the 20th century, psychiatrist.

And so I’ve grown. This has been a preoccupation, and if it were simple, I guess if I could have put it to rest back then I would’ve, and just written an article. But I continue to learn about it and I feel that it also evolves in time. Originally it was a niche issue that I seem to be preoccupied with, and now it seems strikingly timely.

It is useful to start with the definition. So one that I think helps me think about it is one that many researchers used also, which is coercive persuasion, which captures the fact that it’s not simply a process of coercion or torture. Although these phenomenon of force and forceful interrogation in many cases can be and often are used in particular episodes, but also it’s a matter of persuasion and a kind of, it’s an inner surrender. I think you could say there’s an element of cooperation, whether that’s forced or not.

CHAKRABARTI: Ah-huh, okay. That’s really important because I think the phrase coercive persuasion may end up. It may have us end up having too narrow a conception of what brainwashing or mind control is.

That inner agreement or the inner desire we’ll talk about in just a second. But one of the earliest or maybe best-known early examples, modern examples, I should say, strictly speaking, happened during the Korean War, right?

LEMOV: Exactly. This is when the word really started to circulate in the English language.

It became well known, and in fact, I mean, for various reasons, the Korean War was, gained the name the Forgotten War. But the one thing that was remembered was often that the troops had experienced something that was called brainwashing. There were 21, in particular, 21 American POWs elected not to return to the United States after being held for several years behind enemy lines north of the Yalu River.

And so they elected to go to China, and many of them spent several years there. Some of them died there. Almost all of them ended up returning. But this, when they declared that they didn’t wanna return, it created a kind of emergency at the highest levels of state and military.

CHAKRABARTI: Okay. So just for a quick second here, let’s listen to some evidence of how profoundly their minds were, at least for a period of time, transformed. Here’s a video clip of these POWs on January 23rd, 1954, singing the International Communist anthem.

(SINGING POWs)

CHAKRABARTI: Now, so how, these men were POWs, right? They were treated, what, rather brutally in their time as prisoners. Tell me more about that.

LEMOV: I got curious about how exactly they were treated, and it actually happened in phases, so it was an incredibly brutal experience, as you might imagine, but the brutality ebbed and flowed, and they also were subject to boredom and many feelings as the war wore on and they were held in the camp. So initially they were, most of them were very young. They went to Korea to fight. Many of them didn’t even understand where Korea was or what they were doing.

It was still called a police action. So some of them thought they’d be driving police cars around a place they knew nothing about. They were shocked to find themselves in these brutal battles, in record freezing temperatures. They said you could, sometimes, they were captured, many of them almost immediately when the war turned, when the Chinese entered and lent their forces to the Korean, the North Koreans.

So they were, once they were captured, they were marched north. They were barely fed at all, sometimes just a rice ball or two a day. With worms in it. They would pour the blood out of their boots every morning. Before they started to march again north. And then some of them joined the Tiger Death march, which is, which has the highest rate of death for any captive group in military history in the U.S. I believe, like much higher than World War II. They were pushed off the side of mountains. Anytime a soldier or even some civilians took part in this as well, stop, they were just eliminated.

And so some of them were so weak by this time, they had lost half their body weight. They would try to pick up a comrade, tried to carry them along, a fellow soldier and they would die from just the exertion and both of them would lie in a heap on the road and would either be, would be eliminated. So it was incredibly brutal.

It was also by the time they got to the camps along the Yalu River, there were 10 camps, they were forced to sleep. So close to each other that anytime one man had to roll over, everyone had to roll over in the huts and many of them would die overnight. So you would go to sleep and in the morning wake up next to a corpse.

Some people would simply die from trying to walk to the bathroom.

CHAKRABARTI: So you’re giving us a picture of the complete destruction of human beings, right? Physically, spiritually, mentally, emotionally.

LEMOV: Exactly. I think there was a testimony in one of the oral histories from a missionary who’d been riding in a train, and he saw the troops marching past him on their way north, and he said, they didn’t look like any Americans I had ever seen. And one of the shocking things to the Americans themselves, the soldiers, was that they’d just come fresh from World War II. There was this triumphant nation and their provisions were terrible. They were forced to use equipment from World War II. The U.S. just wasn’t prepared to fight the war initially.

So they were demoralized on several levels. In every possible way.

CHAKRABARTI: And so then this sort of complete conversion of the mind though, did that come, how did that come about? Were they promised an end to their suffering?

LEMOV: It came about because, so initially the camps were run by Korean troops and then, and the conditions continued to be harrowing and terrible, but then the Chinese took over it and the conditions got markedly better in the sense that people weren’t dying of starvation.

They could, they had the strength to walk up the hillside and look around a little bit. So conditions got somewhat better, but the Chinese, under the direction, the instructions of Chairman Mao decided to conduct an experiment, and I think this hasn’t been remarked on enough, was the experimental quality was to see if U.S. troops, especially the GIs, not the officers, would be susceptible to or would be, could be reeducated in the same manner as Chinese peasants, because they were seen as the same social class.

So they put them through a very formal sessions of reeducation over many months. The first part of it was something called confession, which had many aspects. Much of it was listening to lectures, being exposed to Maoist doctrine, being asked to defend their way of life.

And one of the men, who were, you played the tape of singing Morris Wills. He had started off as a 17-year-old from upstate New York, and he, who loved volleyball, he, by all testimony, a very nice young man. He ended up in these camps and he was asked, he said, I was asked to describe what democracy was and why I preferred it, and I couldn’t, he said my schools had never, he had left to join the military.

As a junior, but he could, he said, I couldn’t defend it, and I felt like I didn’t have the skills to defend myself against these ideas. But really also what was happening was a kind of emotional, they were, 90% of them had to keep journals and talk about their family life, their feelings about their parents, their siblings.

And I found pictures of these journals, some of them. So this was an introspective procedure that was built into the kind of confession, which could also, if they didn’t cooperate, then they would, their treatment would become worse.

CHAKRABARTI: So it’s an introspective procedure under duress.

LEMOV: Exactly. It was forced introspection.

CHAKRABARTI: And would any of us, would want a relief from that duress. And look, people want to think they’re stronger than that, but I don’t think most of us are.

LEMOV: No, I think it’s very humbling. To really try to, if you really try to put yourself in their shoes, it was a very difficult situation.

CHAKRABARTI: But there’s a point in time, like how do we know when their personal beliefs about the United States, about democracy, about communism went from, I’ll say anything, so you can give me some food to eat to an actual belief. That’s how we characterize what brainwashing is. It’s a total change of states of belief.

LEMOV: Yeah. An ideological confusion. I think it was more subtle than that initially, someone like Morris Wills, who wrote a memoir later when he returned to the U.S., he said, it’s a step, it’s a brutal step by step process. It’s not a sudden magical, it’s not an overnight transformation. But at a certain point, his own doubts, his own childhood experiences, his feelings about, his mother had died when he was young.

He had issues with his father. He came from a long line of farming family. There are all sorts of elements came into play and the introspection while forced also caused him to think a lot about why he was there. And they also felt very much abandoned by the U.S. government. So not, of course, not everyone did decide to sign on, but in some ways, it was the more idealistic among them or those who resonated in a certain way with a promise of a more just society, which is what they were told that they would be encountering.

CHAKRABARTI: Okay. So that’s very important it seems, because you’d mentioned earlier that brainwashing happens when that particular vulnerability, whatever it is, within each of us is found and exploited.

LEMOV: I think that’s, you could see that happening. And also, Robert Jay Lifton wrote the classic work on this, Thought Reform and the Psychology of Totalism, identified this exactly, that each person’s quotient, whatever it was, whether they were a missionary who underwent thought reform in China, or whether it was the POWs.

It was some aspect of their emotional makeup that was, or guilt or shame or something from their past that was capitalized on and used against them, while also their idealistic qualities, their intelligence could be turned against them.

CHAKRABARTI: Ah, okay. Interesting. I wanna just quickly touch on a couple more.

Of the famous examples that we know about, Patty Hearst comes to mind. Of course, she was kidnapped in 1974 by the Symbionese Liberation Army held hostage for 19 months. And during that time, she was just 19 years old at the time, and she committed several serious crimes, including violent bank robberies.

Later on, her defense team argued that she had been brainwashed because during her captivity, she was held in a closet for weeks, raped, tortured, forced to read communist materials. But in her trial, the prosecution brought about radio recordings that Hearst had made while in captivity, including this one from April 22nd, 1974.

HEARST: For those people who still believe that I’m brainwashed or dead, I see no reason to further defend my position. Consciousness is terrifying to the ruling class, and they will do anything to discredit people who have realized that the only alternative to freedom is death. And that the only way we can free ourselves of this fascist dictatorship is by fighting not with words, but with guns.

I‘m a soldier in the People’s Army.  

CHAKRABARTI: Professor Lemov, tell us what you think we should know about the Hearst case.

LEMOV: So I think the Hearst case is interesting precisely because it’s often misunderstood and persistently misunderstood. Even today, most people when polled, I mean as of 2014, at least believe she should have remained in prison and was guilty.

Even understanding, so going back to what happened to her, she was a 19-year-old sophomore. She was at UC Berkeley. She came from a very prominent family, the Hearst family, who ran many newspapers, media empire in California, her grandfather was William Randolph Hearst. And so when she arrived at UC Berkeley, this leftist group that you mentioned, the Symbionese Liberation Army identified her as a potential target, and they conducted what they called a righteous arrest of this heiress.

They saw her as an heiress. She was one of several daughters, and they came to her house, they abducted her, they knocked out her boyfriend with a wine bottle. They threw her in the trunk of their car. They sped off down the street, spraying the street with automatic weapons. They kept her, as you mentioned, in a closet, but it wasn’t that she could read a Maoist track. She was actually forced to listen to them because she was blindfolded for 59 days.

She couldn’t go to the bathroom by herself. Her whole system, basically, she was so terrorized, her whole system shut down. She described that as part of the suffering. She couldn’t eat, she couldn’t relieve herself. She just was there in the closet. And then several of the group’s members came in and raped her.

And she knew that if she didn’t, she knew that she needed to stay alive. And she said at one point she decided that she would want to live.

CHAKRABARTI: This isn’t, the outcome here doesn’t sound like brainwashing to me, but just the pure survival instinct kicking in, I will do anything to stay alive.

The interesting part, and this is why, for example, in the most recent book about her, which is by Jeffrey Toobin, and in interviews, he describes her as a willing bandit. And why she did not prevail in court. She’s always been found guilty of the crimes she committed even after the jury, toward the closets in which she was held.

And even after they heard the details of these things, it was because she said, I did have to convert myself. I had to, I had, she said, by the time they were through with me, I had to become a soldier in their army. They didn’t want to think they had brainwashed her. They had to feel she was utterly sincere, and she said, I had to make myself be sincere.

So she said I accommodated my thoughts to coincide with theirs. In other words, she wasn’t pretending just to survive. She had to actually enact this inner transformation on herself, even though it wasn’t permanent, it was sufficient to keep her alive for those. And it turned out to be years that she was captive.

And there were times during those 19 months after the robbery where she could have walked away, but she didn’t.  So that confused people.

CHAKRABARTI: It brings to mind the question of what constitutes will, then, though. We’ll talk about that when we come back.

Part III

CHAKRABARTI: The examples that we’ve talked about so far, professor, they’re ones that are grounded by a very brutal reality, right? Torture, fear of death, starvation, like the worst kinds of treatment possible which very clearly break down a person. There’s a whole ‘nother, several other classes of thought conversion, if I can put it that way. That don’t necessarily require that or don’t use those kinds of techniques. Like cults come to mind.

So what do they do there that achieves a similar kind of thought conversion?

LEMOV: Cults are a good middle ground, I think for this kind of, it’s almost a thought experiment to see what we can draw from, what lessons can we draw from these extreme episodes? Which seem to highlight certain qualities and really dramatically show the human capacity for conversion, even if it’s, I should stress, it’s not permanent.

One of the misconceptions is that once someone is converted in that way, that it’s permanent. But even Mao himself said, I need to, I need, this needs to be reinforced. So once it’s not reinforced, there’s a capacity to change. But to come back to the question of cults, people are often drawn into them in slow ways or even just encountering a set of extremely friendly people.

The recruiting process can also happen really quickly depending on, and it happens, so I would say everyone’s vulnerable. There’s not a particular type of person or personality type that’s more vulnerable, but it can have to do with this stage of your life you’re in or having gone through in a recent emotional shakeup or some being in a certain state.

Also, it can have to do with sheer, from what I can discern, just sheer accident, you took one, you decided to wait at one bus stop, not another, and there you encountered two recruiters for a certain group that invited you to a friendly dinner.

And maybe you’re interested in the environmental cause they said that they were, there can be a deception as well. And then once one finds oneself at an event, you can be drawn in by mostly, it’s, I think the mistake is to think you’re drawn in by the ideas, but it’s usually the emotional connection you’re finding there.

The kind of rapid sense of what they call love bombing, which is just really over flattery. Praising, the kind of the similar dynamics we were describing when talking about the AI chatbots. That is a flattery that you could see through, you could intellectually question, but it’s also working and it’s maybe giving you something you really want, which is a connection with others.

CHAKRABARTI: So let me ask you then, let me push this even a little bit more. And that is some commonalities, the love bombing, right? And in a way, through that, the stripping away of whatever that person’s past beliefs or values were. Sometimes I hear, we hear similar things when someone even undergoes like a profound religious conversion, right?

They talk about wanting to be broken down before God so that a deity’s love can build them back up. Are there similarities there?

LEMOV: Yeah. There are many similarities I think with the classic cases of religious conversion, St. Paul on the road to Damascus. Being struck by a God and like falling down and standing up, never the same.

And people who join cults actually describe such experiences, and it grieves ’em later. When the group that they joined with full sincerity. And with this great sense of fervor. Because their experience is real. Later it turns out to be an abusive group, but they’re stuck by the power of that experience.

CHAKRABARTI: Yeah. The destructive power of our need to be loved is really a theme here.

LEMOV: And a need for meaning and meaning. A meaning greater than just the single individual, the sense that you’re working towards some, so they really can exploit that.

CHAKRABARTI: So all the examples though that we’ve been talking about thus far, and you’ve mentioned this a few times.

Their effects can be undone, and it makes me, I’m thinking part of the reason why is because like when you’re in the cult, for example, there is actually a different reality outside of that cult, right? And so whenever, for whatever reason, if you exit that cult, you’re at least presented with reality.

What we don’t have anymore is that is that opposing reality, when we’re talking about the effects of online brainwashing, or at least online information manipulation, because the digital world surrounds us so completely and so thoroughly. So can you talk about that, about how are we supposed to understand thought conversion in the digital age?

LEMOV: I think the digital world offers many challenges to this. I had a friend who was a de-programmer back in the classic days of cults, ’70s and ’80s. They’re now called exit counselors. He says it used to be enough just to bring a briefcase of information and show the person, or it was at least extremely helpful to show the person information they weren’t otherwise able to access about the nature of the group they were in, the destructive nature, and it might resonate with certain doubts they already had, but in the age of the internet, that information is available.

It just wasn’t, it wasn’t being accessed, or they were shielded from it. Or sometimes people can be recruited online. Is there an out, I suppose the question is how do our digital and virtual environments affect us?

CHAKRABARTI: Yeah. So do you even know what’s happening? It being the slow change in, or sometimes even rapid change in belief or thought with the online world that we’re all living in.

LEMOV: That’s what partly why I wrote myself into the book, because I think it affects all of us. I see it in myself just in small ways, in large ways, and I’m trying to —

CHAKRABARTI: Give me a specific example.

LEMOV: A small example would be me doing my research. My daughter said to me, you’re spending a lot of time on Instagram. And I was like, no, I’m not. I was just like, that’s outrageous. Or whatever time I’m spending is surely my research. But actually, when I stopped, I realized, oh. I was absorbing me in a certain way.

It was shaping the parameters of, it was even creating these kind of parasocial relationships. And these things seem trivial or even they often are enjoyable. I’m learning things. I cured an injury I had in my foot by being targeted by various Instagram ads.

But —

CHAKRABARTI: That doesn’t sound like brainwashing to me.

LEMOV: No, it’s not brainwashing, but I would put it more in the realm of hyper persuasion, but I would say it’s part of a realm of possibilities. I think that I like to think of brainwashing, not so much as like an absolute label when puts on something, but a window, a way of reflecting.

So using what’s happened to others as a window to what’s happening to us now. I wouldn’t say that we’re all being brainwashed all the time, but just that we are being pushed into moments of dissociation. All the time, and dissociation was actually what the experts located as the root of brainwashing, a kind of numbing … that’s what Robert Jay Lifton spoke about. I think we can all identify with the fact that when we’re on devices, there is a way that we’re stepping out of our bodies.

CHAKRABARTI: Right. You most certainly know the work of I just admire her greatly about like Shoshana Zuboff talking about surveillance capitalism.

And that is, that corporations have so much information about us that they can truly, without us realizing it, shift our behaviors based on things that we get fed online.

LEMOV: Yeah, you could think of it as a kind of micro, behavioral micro nudging that’s constantly happening.

CHAKRABARTI: And not for your benefit.

LEMOV: It may, yeah, you may feel, I mean, I found like a foot massage tool or something. Something mundane. It may seem beneficial. Students often, it is a paradox ’cause we seem to be receiving benefit and that’s what companies would say. Consumers are choosing this.

There’s no coercion here. But I think that these environments are emotionally, are ones of emotional engineering.

CHAKRABARTI: So consumerism is one part of this. But perhaps the more disturbing part is, especially over the past 10 years or so, we’ve heard so many stories and seen so many examples of people who find that loving online community.

But then they end up being QAnon believers. And they completely separate themselves from their families because of that. Or they end up believing lies about COVID conspiracies, that kind of thing. And it ends up replacing those beliefs and that online community end up replacing the networks they had in real life.

LEMOV: It creates a feedback loop. We have this loneliness epidemic, and that can contribute to people’s seeking connection. And there’s the famous recent clip of Mark Zuckerberg saying that we’ll now create 17 AI friends that you can have that will replace or somehow cure the loneliness epidemic, when, you know, in fact it will no doubt accelerate this process.

CHAKRABARTI: That’s what he wants, right? Because he and his company, those companies will make a lot of money by having us.

LEMOV: Yeah, it sort —

CHAKRABARTI: We don’t have to end up with that future.

LEMOV: No, I think that under, but I think that it’s not to say we have to not use any device or any product, it’s to say that there’s a, to understand the dynamics of self-separation or numbing or democratization and to be, and to bring yourself back to the body.

I was just listening to a lecture that was given by Thich Nhat Hanh, the Buddhist teacher in 2014. And he said, when you spend two hours with your computer, which I thought was a lovely way of putting it. Even just working. You’re not in your body and there are things you can do to bring yourself back to it.

And I think it is that process of just seemingly trivial dissociation that we can counteract in different ways.

CHAKRABARTI: So you have to be mindful of it though.

LEMOV: Exactly.

CHAKRABARTI: Okay.

But I do wanna ask you something else, we talked about, we always think about it could happen to someone else.

Right now, online political parlance is full of accusations. You know when the left looks at the right, they call it the Trump cult, right? When the right looks at the left, they say woke mind virus. There is definitely a sense that the relinquishing of mental independence is always something that happens to the other side, right?

Obviously, that’s not true.

LEMOV: Yeah. What I’m trying to do is use brainwashing as a tool to reflect on that, because it’s a word that mostly, and it’s at least since the ’70s, has been used to denigrate. It’s always shameful thing happening over there to other people, to which I would not be susceptible.

And it’s used to denigrate entire groups, but I think if you can see the way that it’s actually a phenomenon that affects everyone and I use this again, to reflect on my own experiences as well. It creates more of a sense of us. So hopefully because these processes of extreme polarization and even the numbing of difference, the various types of numbing that have been identified, actually feed the problem.

I really think. So I find it useful to try to understand those dynamics.

CHAKRABARTI: So can you tell me a little bit more about what the numbing feels like? Because some people, they are actually feeling the opposite of numb when they become fully immersed in these online communities or experiences.

LEMOV: There can be, certainly. I think one thing that’s surprising about, and being actively swept into a community is a sense of great, of completion, of belonging, of at last, I found something, and this can happen in cults actually, even the POWs described this kind of inner surrender. If we want to use that as a synonym for this type of conversion, it’s a great relief.

There’s a relief to surrendering. To something and that even to use another dark example, the Manson girls were often described, in court, they said, oh, these are just robots. They were just robotically carrying out their will of their leader. But actually, they themselves said, we have powers too.

We think our leader has powers, but we have powers too. They were strangely exhilarated by this process. Many people can be exhilarated by joining a group. I don’t think all are such negative experiences, but what I guess what I mean by numbing is to bring us back to the seemingly trivial stakes in day-to-day life that each of us will encounter. We look down the road and it looks so extreme and extraordinary, and it looks like something I would never be a part of. I would never fall for that. I would never be that person jumping up and down in Times Square shouting about QAnon.

But actually, it can start from these very small, to go back to Morris Wills, it’s a step-by-step process and it does involve a kind of numbing, which can range.

CHAKRABARTI: Yeah, my personal wishes to just be able to turn off all digital devices, not permanently, per se, but more frequently than I do.

I understand that that’s not a terribly realistic, so short of that. In the last minute that we have, do you have like specific advice on like what should we be listening for in ourselves or looking for in ourselves to realize, hey, maybe I need to adjust my behaviors or interactions with online stuff?

LEMOV: No. So my practical advice is just to not think of it as a matter of ideas. I’m getting bad ideas. It’s more to tune in. Not to say that one is, that there aren’t bad ideas, but it’s just that to tune in more to the emotional substrate and even at the level of your body, to try to just check in.

So what I think another synonym for brainwashing, a huge part of the phenomenon is ungrounding. People, this is a process of successive shocks to the system resulting in disorientation. So to the extent you can ground yourself, orient yourself. Periodically. Thich Nhat Hanh recommends putting a timer on your computer every 15 minutes.

It would just ring. And that would bring you back, have a moment to reconsider.

CHAKRABARTI: Break the spell. Every 15 minutes.

LEMOV: Yeah.

CHAKRABARTI: That’s so fascinating. Also, I just wanna once again, point a finger towards the big tech companies, right? Because they’re in the business of keeping your eyes glued to the screen and for heightening your responses, the exact opposite of what you say we should be aiming to experience.

The first draft of this transcript was created by Descript, an AI transcription tool. An On Point producer then thoroughly reviewed, corrected, and reformatted the transcript before publication. The use of this AI tool creates the capacity to provide these transcripts.

The first draft of this transcript was created by Descript, an AI transcription tool. An On Point producer then thoroughly reviewed, corrected, and reformatted the transcript before publication. The use of this AI tool creates the capacity to provide these transcripts.