Computer scientists say ‘I told you so’ as AI Companies Hit Scaling Wall

Artificial intelligence (AI) companies are hitting a scaling wall, according to a Reuters report referring to experts and investors in the AI space. The report suggests that the results from scaling up pre-training— making models bigger and feeding models more data— are no longer providing proportional capability improvements. AI developers are reportedly facing challenges in developing a model that is better than GPT 4. The developers are struggling with the following key challenges—

Scaling costs 

Hardware-induced failures due to the complexity of the training process

Lack of readily available data

Power shortages

To cope with this, companies are using a process called “test time compute” where the model learns more while it is coming out with inferences. So for instance, when you ask a model for the answer to a specific question, it might show you two options instead of pre-emptively picking one. This helps the model allocate more of its processing power to challenging tasks like maths or coding.

Scaling concerns in Google, OpenAI and Anthropic:

While Reuters published a comprehensive report about the subject, concerns around AI scaling have been in discussion for a while now. As per a recent Bloomberg report, OpenAI’s new model Orion did not live up to the company’s expectations, given that it wasn’t as big a step up as GPT 4 was from GPT 3.5. Similarly, Google and Anthropic are also struggling to make any major breakthroughs. 

This comes as a concerning development, especially keeping in mind the large quantities of funds companies are allocating on AI. Meta said that its investment in AI continues to require “serious infrastructure” and that it expects to invest significantly in them in its latest earnings call for the quarter ending in September 2024. Similarly, as per Google’s recent earnings call, the company spent $7.2 billion on sales and marketing, these expenses were a result of Google’s investment in advertising and promotional efforts related to the Made by Google launches, as well as for AI and Gemini. 

AI scientists say they told us so:

Commenting on the Reuters report, Meta’s Chief AI Scientist Yan Le Cun says “I told you so”. He explains that auto-regressive large language models (LLMs)— models that use past data to predict future trends — are hitting a ceiling and that he has been concerned about the same since before most people heard of LLMs. “I’ve always said that LLMs were useful, but were an off-ramp on the road towards human-level AI. I’ve said that reaching human-level AI will require new architectures and new paradigms,” Le Cun explains.

Fellow computer scientist Gary Marcus argues that he spoke about deep learning models hitting a wall in 2022. “We all know that GPT-3 was vastly better than GPT-2. And we all know that GPT-4 (released thirteen months ago) was vastly better than GPT-3. But what has happened since? I could be persuaded that on some measures there was a doubling of capabilities for some set of months in 2020-2023, but I don’t see that case at all for the last 13 months. Instead, I see numerous signs that we have reached a period of diminishing returns,” Marcus mentioned in his blog post in April this year. 

At the time, he flagged struggling AI projects like Inflection AI and Stability AI that were struggling with financial difficulties. “If enthusiasm for GenAI dwindles and market valuations plummet, AI won’t disappear, and LLMs won’t disappear; they will still have their place as tools for statistical approximation,” he explained. 

Meanwhile, OpenAI CEO Sam Altman addressed the Reuters report with a single-line tweet saying there is no wall. 

Chef Jack Bishop Breaks Down ‘The Science of Good Cooking’

11/15/2024

Jack Bishop. Credit: America’s Test Kitchen, Daniel J. van Ackere
What’s the secret to making a fluffy omelet or the perfect pie dough? In this interview from 2012, Jack Bishop, now senior content advisor at America’s Test Kitchen, debunks cooking myths and highlights some of the surprising finds from the show’s cookbook, The Science of Good Cooking.
Bishop tells us that water is one of the key parts of the perfect pie crust. You need it to roll out the dough, but it also forms gluten which makes the dough chewy. So, how can your pie dough be both soft and easy to handle? Use science—and a little bit of vodka.

Further Reading

Donate To Science Friday
Invest in quality science journalism by making a donation to Science Friday.

Donate

Segment Guests

Jack Bishop

Jack Bishop is the senior content advisor at America’s Test Kitchen in Boston, Massachusetts.

Segment Transcript

The transcript of this segment is being processed. It will be available within one week after the show airs.

Meet the Producers and Host

Denise Chow
About Denise Chow

@denisechow

Denise Chow is a sci-tech editor at Live Science and a former associate producer for Science Friday.

Ira Flatow
About Ira Flatow

@iraflatow

Ira Flatow is the host and executive producer of Science Friday. His green thumb has revived many an office plant at death’s door.

‘Your Monster’ Movie Ending Explained & Summary: Is Monster Real Or Imaginary? 

What most of us end up expecting romcoms to deliver are these groundbreaking emotional resolutions that are reassuringly final. But since people are ever-growing, ever-changing things, is any emotional resolution ever truly the final thing? Emotions change, so do states of mind. So what I remind myself to look for in these films that are supposed to be feel-good is a breakthrough–a step toward something better, with a long road left ahead. And that’s exactly what I found in Caroline Lindy’s spooky tale of self-love. Yeah, the Monster’s adorable and super supportive, but I think Your Monster is more about Laura’s personal growth than anything else. Let me explain why.

Spoiler Alert

What happens in the film?

Laura couldn’t be having a worse time. No, seriously. Imagine being young and full of dreams and being diagnosed with abdominal cancer. At least she’s got her super loving and reliable boyfriend Jacob by her side as she goes through surgery, right? Nope, no luck there either. He basically abandons Laura when she needs him the most, because apparently, he didn’t sign up to actually love her when he got together with her. Taking care of her is too much for Jacob, I bet he never moved a finger to help her anyway. Laura’s practically got no one when she moves back into her childhood home. Her mom’s done her part by sending a rally of pies and a $5000 check, and her friend Mazie’s all words no action when it comes to being there for her friend recovering from a very serious surgery. There must be something that a bright woman like Laura can look forward to, right? Well, as far as her career goes, her loser ex has stolen all the credit for the Broadway musical Laura helped him write. Seems like Laura’s life has set the perfect stage for her to be open to something new. But Laura doesn’t know what to make of it when that new thing turns out to be a full-blown, charming monster who’s been hiding in her closet. 

How does Laura cope with Jacob taking everything from her?

There’s almost no way to keep yourself from being swallowed by self-doubt when you’ve been with someone as horrendous as Jacob for years. So it’s only normal that Laura’s been drained of the last ounce of her confidence. She’s become increasingly passive, jittery with every word she speaks. She doesn’t think she deserves anything. Despite getting no calls from Jacob about the audition for the lead character, Laurie, a character that’s practically based on Laura, she shows up for the audition. Of course, she chokes up in fear and self-consciousness and bombs the audition that she should’ve been a shoo-in for. But not everything’s going awful in this weird fairytale-like circumstance Laura’s found herself in. Monster started out pretty hostile to Laura at first, practically forcing her to leave her own house because he only likes his own company. But Laura grows on Monster. So much so that he lets her stay. Maybe on a subconscious level, the validation of the friendship that grows between Laura and Monster gives her the confidence to put herself out there. She was completely crushed when Jackie, the actress who everyone’s head-over-heels for, got the part that was meant for her. But she swallows the hurt that Jacob’s caused and takes the pity job he offers her anyway.

There’s no denying that Laura still has feelings for Jacob, and that’s partly what motivates her decision to be Jackie’s understudy. She’s still hung up on him even after she walks in on him knocking boots with someone else. But when you look underneath all these insecure, desperate actions, you see a girl who’s after the closure that Jacob never gave her. All her life, she’s been made to justify the pain that people have inflicted on her. And it’s gotten to a point where she actually believes that she doesn’t deserve any better. It’s only when Monster pushes her to open her eyes after that wretched Halloween party with the cast that she even begins to acknowledge that what Jacob did to her was horrible. That realization and the rage that follows are what contribute to the strength she shows when she goes ballistic on Jacob for waving his giant ego around and treating everyone else like trash. But here’s the thing about growth. It’s almost never linear. The romance brewing between Laura and Monster was good for her. It’s the only thing that puts a smile on her face for a change. But Laura isn’t completely over Jacob yet. It’s a total high for him to hear how much Laura loves him. What else do you expect from a narcissist? Even though he fully knows that he’s gonna fire her for “going out of line,” Jacob sleeps with her to feel good about himself. When Laura lashes out at Monster, she’s only projecting how terrible she feels about herself. And when Monster leaves her all alone, you see that little self-destructive streak in Laura and recognize how it’s made her push away love because she has a hard time letting herself be happy. At this point, she feels too empty to even be happy about the fact that she’s beaten cancer.

Is Monster Real Or Imaginary? 

Your Monster‘s final act is a long trail of reality checks for Laura. While she did stand up for Jackie when Jacob was bullying her, Laura’s been under the impression that Jacob and Jackie have been an item for a while. You can chalk all that up to all that flirtatious energy between the director of the play and the lead actress. But Jackie’s a girl’s girl. She takes a few steps back the moment she gets to know how Jacob had very abruptly dumped Laura and snubbed everything she cared about. When she goes over to Laura’s place to clear the air, they plan to make a fool of Jacob on the big opening night and let Laura play the role that she meant to play. Carrying out the plan comes at the tall cost of Laura realizing that Jackie wasn’t the one sleeping with Jacob. It was actually her own supposed friend, Mazie. But she was a terrible friend anyways. So no big loss there. 

In Your Monster’s ending, Laura pours her heart and soul into the performance of her dreams. Monster may not be in her life at the moment, but being around him has shifted something in Laura. So when Jacob attacks her backstage, her rage is what summons Monster. It’s curious how Monster kills Jacob the exact same way Laura said she’d kill him when he told her that she was destroying his life. It makes you wonder if Monster is real or the manifestation of all of Laura’s pent up rage. When Laura met Monster, he mentioned that they’d met before when they were both kids and then teenagers. The first time was when a little Laura felt terribly alone and scared, and the second was when a boy got mad at her for her having rejected his advances.

And then, Monster came back into her life when she was wronged, hurt, and abandoned. Judging by how her mom wasn’t there for her even when she fought cancer, it’s safe to say that Laura didn’t feel loved even as a child. So maybe the idea of the monster under her bed was the closest thing to a friend she had growing up. Laura silently accepted the cards she was dealt. But there must’ve been a lot of anger someone in her circumstances had inside. So Monster could very well be an imaginary friend who came back into Laura’s life to remind her not to take Jacob’s awful treatment lying down. It’s only when she fully embraces her rightful rage that Laura feels strong enough to fight back. But then again, I guess it’s totally up to you to decide whether you believe Monster is real or not. If you see Your Monster as a fairytale, you’re right to believe that Laura found love in a very real, very sweet Monster who’s protective of her, sometimes in a homicidal way. Or, you could also see Your Monster‘s ending as Laura learning to love the side of her that’s been lying dormant for a long time–the side of her that people wouldn’t dare mess with.

Related