Genius or sadist: What the bestselling non-fiction books of the 21st century say about us

Forget our current love of true crime; it was around the turn of the millennium that we were truly sick and twisted. Back then, the biggest trend in non-fiction was “misery lit”, as readers young and old binged on stories of abuse. We were initiated into this bleak era by the viral 1995 publication of A Child Called “It”, Dave Pelzer’s childhood memoir of sadistic abuse at the hands of his narcissistic alcoholic mother. Pelzer continued to pump out many follow-ups alongside a host of copycats – all grouped together on a shelf in Waterstones ominously labelled “Painful Lives”.

In retrospect, all that torture and trauma in books reflected (and helped to prompt) a democratisation of suffering; we began to talk with others about our childhood struggles. It was the beginning of anonymous connectivity with the internet, blogs and online chat rooms. In the UK, major legislation was established around child protection systems and criminal record checks for the first time ever.

Analysing non-fiction trends is a powerful and reliable way to understand our culture. Now we’re a quarter of the way through the century, there is plenty of data on our reading habits so far (here I’m using The Sunday Times bestsellers list of the past 50 years together with statistics provided to The Independent by Nielsen BookData). So, what do the bestselling non-fiction books in the UK of the 21st century say about us? Are we still voyeuristic sadists or do we have higher aspirations that have elevated us out of the gutter?

While we might’ve moved on from the misery of the mid-late 2000s, we still love to read stories about the private lives of other people. According to Nielsen BookData, the largest product group within non-fiction in the UK market since 2000 has been biographies and autobiographies (swamped, of course, by ghostwritten celebrity memoirs). Heartwarmingly, though, the bestseller by a long shot, has been Adam Kay’s medical memoir, This Is Going to Hurt: Secret Diaries of a Junior Doctor (2017).

The book, which was adapted in 2022 by the BBC in a series starring Ben Whishaw, details the extreme stress involved with working in the NHS. Sales were boosted during the pandemic, when a light was shone on the vital role of key workers such as doctors, amid an ongoing debate over junior doctors’ pay. Encouragingly, the enduring love for this book reflects a majority support for the NHS in the UK. According to the latest British Social Attitudes Survey, satisfaction with the NHS has fallen to record lows – but 94 per cent of Brits believe it should remain free of charge; This Is Going to Hurt is being bought and read by people who defend access to healthcare as a human right. Its success revealed the humanity in our reading choices: that we appreciate the human aspects of healthcare, that we have empathy for the emotional and psychological toll on medical workers.

If Brits are going to read non-fiction now, we want to be challenged by the depth and breadth of what we read. The majority of bestsellers stood out as ridiculously ambitious and surprisingly academic. They included Sapiens: A Brief History of Humankind (2011) and Homo Deus: A Brief History of Tomorrow (2015), both by Yuval Noah Harari, Prisoners of Geography by Tim Marshall (2015) and Surrounded by Idiots: The Four Types of Human Behaviour by Thomas Erikson (2014).

Maia Snow, news editor at the British publishing industry magazine The Bookseller, notes that all of these books simplify enormous concepts that would previously have been considered too complex for a layman to comprehend. “Harari’s two books are obvious examples of this – they cover a massive subject, rather than niche non-fiction which would only appeal to a smaller group,” says Snow. “[Bill] Bryson’s A Short History of Nearly Everything – another leading bestseller of this period – also falls into this category.”

This move towards embracing complex subjects reflects the information revolution of the 2010s and a growing number of people online (in 2000, only 25 per cent of people in the UK had internet access; by 2017, that figure had skyrocketed to 90 per cent). Where we were once amazed by the primitive internet offerings of the 2000s, we now intuitively – and constantly – access vast amounts of information daily. As a result, we broadly possess superficial knowledge of a wide range of topics. We scroll past mind-bending issues distilled into five-slide infographics or a pithy joke in a caption. This is where non-fiction like Sapiens, which thinks critically and gives context about our world, emerged as forerunners to help us fill in the blanks. These books counteract that oversimplification in our content consumption.

Clearly we’re seeking a deeper understanding of humanity and our psyches, a fascination reflected elsewhere in the mental health awareness movement of the 2010s to the true crime boom in pop culture. Culture writer-editor, Annabel Nugent, who covers books for The Independent, says of these titles’ popularity, “We’re interested in why others are the way they are. It’s less about self-help and self-improvement, more about understanding anthropological reasons the way the world is.”

We’ve been intrigued by our psychology, yes, but fascinated by other people’s and why it has shaped or even damaged our own brains and lives (see the scathingly titled 2019 bestseller: The Book You Wish Your Parents Had Read by Philippa Perry, about how parents have relationships with their children).

That said, there were a couple of self-help books that did make a mark. Nielsen BookData found there has been a recent boom in the personal development field. A stand-out success of the 21st century, alongside Kay’s medical memoir, is James Clear’s Atomic Habits: An Easy and Proven Way to Build Good Habits and Break Bad Ones (2018). If you haven’t read it (unlikely at this point), you’ll probably have the cover seared into your memory: “Tiny changes, remarkable results.” In it, Clear impresses on readers an array of data about how to change our habits (ie everything we do unconsciously) to improve our lives.

What’s so interesting about non-fiction bestsellers is that – unlike fiction – they really are decided by what people want to read

Another freak phenomenon of 2018 was that of Vex King’s Good Vibes, Good Life: How Self-Love is the Key to Unlocking Your Greatness, a basic and colloquial self-help book by a wellness and spirituality Instagram influencer. That year was evidently a micro-turning point in culture where what had once seemed embarrassing or American earlier that decade (girlboss culture, Goop, wellness) was embraced this side of the Atlantic. Why shouldn’t we become our best selves?

The pandemic only furthered our journey inwards: we infamously became more contemplative and began “working on ourselves” en masse. That these books have continued to be steadily popular since then shows just how seriously we’re taking our self-realisation – feeling so comfortable as to even edge into the spiritual (Vex’s more recent 2023 book preempted 2024’s hot topic: manifestation).

What’s so fascinating about non-fiction bestsellers is that – unlike fiction, which is woefully dependent on a big advance that leads to a solid marketing spend and priority in bookshops – they really are decided by what people want to read. Despite the prestigious Baillie Gifford Prize being launched in 1999, none of the leading bestsellers mentioned here won the prize and they’re almost entirely absent from the longlists. “It’s interesting to me how awards can have such a massive effect on fiction, but the same doesn’t seem to apply with non-fiction – despite the Baillie Gifford essentially being the Booker equivalent for non-fiction,” Snow says.

The fact is we’re no longer in school; no one can tell us what we choose to learn about. When it comes to non-fiction, there is no curriculum or cultural pressure (no, I haven’t read the latest Sally Rooney). We’re not Mark from Peep Show so why on earth would we buy Stalingrad?

Non-fiction does feel more like an intellectual commitment. It’s less of an escape from the world than something to make us more a part of it. Given what these flattering reading patterns tell us, we should take this rare moment to feel good about ourselves.

This post was originally published on here