Tech bros have built a cult around AI, part 1

This is a transcript from a Behind the Bastards podcast episode named Tech bros have built a cult around AI, part 1, the first of a two-part series. I've annotated the bits that I found the most important in green highlighter.

See artificial intelligence.

Robert Evans, the show host, has written a wondrous article about the matter^[1].

SPEAKER_02: [...] today we're here to talk about AI. And particularly, we're going to talk about how a lot of the conversation and a lot of the fandom around AI is, has turned into more or less a cult. So that's, that's, that's the premise of this episode. I want to start by going back to a place I was about a week ago as we record this, the consumer electronics show. Every year they've been holding it for a long time now. And it's kind of how the tech industry talks about itself and what it has planned for the future. A lot of it's hype. You know, they're kind of talking up what's coming out that they're hoping we'll get buzz. We'll make money. But you also get an idea of like, what do you think we want and what are you trying to get us excited about? And I think the most revealing product that I saw this year was the Rabbit R1. And it's a little square shaped gadget. It's got a screen. It's got a little camera that can swivel. And it's, it's an AI that basically you talk to it like you would an Alexa, but it can use your apps. And it's supposed to reduce friction in your life by basically routing every move you make online through this machine intelligence. So you tell it what you want to do and it does it instead of like you using like physically using your smartphone as much. You still have to click it sometimes. And I want to play a clip of this where the, this is the CEO Jesse Lyu's keynote speech.

SPEAKER_12: Our smartphones has become the best device to kill time instead of saving them. It's just harder for them to do things. Many people before us have tried to build a simpler and more intuitive computers with AI a decade ago. Companies like Apple, Microsoft and Amazon made Siri, Contana and Alexa with these smart speakers. Often they either don't know what you're talking about or fail to accomplish the tasks we asked for. Recent achievements in large language models, however, or LLMs, a type of AI technology have made it much easier for machines to understand you. The popularity of LMS chapas over the past years has shown that the natural language based experience is the path forward.

SPEAKER_02: Now, I don't know that I entirely agree with that because I think the biggest influence that these chapas have had on me is that whenever I try to deal with like an airline or something, I get stuck on chat GPT and it's a pain in the ass to do anything.

SPEAKER_06: What's so funny about the rise in AI right now is if we really think about it and you threaded the needle right there, when you really think about it, all AI is just the evolution of the shittiest part of calling customer service. Yes, yes, yes, all the one thing that we do as soon as we call is like zero zero zero zero. Let me get straight to a human.

SPEAKER_02: Let me talk to a human.

SPEAKER_06: Yeah. And these like eggheads are like, what if we did more of that? What if we remove your solace from this? What if that was every way of your life?

SPEAKER_02: And it's the kind of thing he seems so off from my experience where he's like the problem with phones is that it's too hard to do things. No, it's too easy for me to order a bunch of junk food and have a stranger deliver it. That's been a problem for me, right? Oh, yeah. It's too easy for me to waste six hours on Twitter. Like that's what the issue is.

SPEAKER_06: Do you know how much the easiness of everything has made me wake up the next morning with cold food on my porch that I never touched? Yes, absolutely.

SPEAKER_02: It was too easy.

SPEAKER_06: So we need those, those, those like bumps in the road to keep drunk stone diffy from ordering a like triple-double burger that I'm hoping gets to me before I fall asleep. It never does.

SPEAKER_02: I've had so many problems with my phone, but I don't think I've ever had the problem of like this thing's just too hard to use. That's just not a problem. I know that anyone has. But Jesse complains that, you know, there's too much friction with smartphones and his device, the rabbit is going to let you like, you can just tell it. Book me a flight or a hotel on Expedia and you don't even have to know it like it'll just pick a hotel for you a lot of the time or like what flight, you know, it thinks is the most efficient. Jesse's goal is to basically create AI agents for customers, which like live in this little device you wear and act as you online to handle tasks you'd normally use your phone to do. So you can tell you rabbit to book you an Uber, you can have a book you a flight or you can have it plan your trip to a foreign country.

SPEAKER_10: That sounds really bad. It sounds so fucking bad. Rabbit's like, 'hey, how would you like a middle speed seat on the air?' Yeah, middle, sufficient as hell.

SPEAKER_02: Spirit so cheap. Now you can you can direct it more, but then that just seems like, well, yeah, that's what I'm already doing on my computer. Why is it easiest just to like work through a vocal chatbot that might not understand me? Or at least we'll be as much friction as like, yeah, when I click touch my phone and it hits the wrong thing, right? I just don't see that I'm saving much here. No one also seems to know how Rabbit's going to integrate with all these apps. And because that means their device has to have access to them for you. And that's kind of a big ask for all of these different companies. That said, and no one knows by the way how secure it's going to be, but no, what it's C.E.S. was listening either because the first 10,000 preorder models that opened at C.E.S. sold out instantly. That doesn't mean a lot of normal people are going to buy it. It means a lot of tech freaks wanted this thing.

SPEAKER_06: You know, that is the thing too is like, yeah, if you're at C.E.S. you're already you're already drinking the Kool-Aid. Yes. You know, and I'm very much it's so funny because I'm very much that guy or I was very much that guy who wanted to be on that cusp of technology. And I feel like, you know, in my early 20s, it seems so cool because you're like, yeah, I want to be Iron Man. I want to just have full control. And then you kind of get to the point where you start if you have enough self reflection, you notice that you're just kind of you're using technology to do things that you could just do for much simpler and cheaper. Like, for example, like when you get these apps and then I'm like sitting in a photo editor for like 50 minutes just to make it look like it was shot on film. And I was like, I can buy a disposable can like that exists. And what's great is when you now when you develop them, they put it, they email it to you so it can go on Instagram. It's going to take longer. But arguably, you might get some post flick clarity. Maybe that pick that didn't look good. Isn't now on the internet. Or, you know, you can like spread it out. But I am getting very, like, you are so much more thoughtful in that than like anybody involved in this product has been over the course of their entire life.

SPEAKER_02: Now, a couple of skeptics who have given reviews have already noted problems. Richard Lawler of The Verge was like, this thing is not built for left handed people to use like they forgot that left handed people existed. And so they designed it in a way that specifically a pain in the ass, which yes. There's also they brag they have this camera that can like move on its own so it can cover stuff in front of it or behind it. And a commenter on Lawler's article was like, it's a pretty fundamental design principle that you don't add moving parts if you don't need them. And there's plenty of space in this for a camera in the front and the back, which is one less point of failure, one less thing for shit to get gunked up in. This is actually bad design. They're bragging about this, but it's a bad idea. Yeah. Yeah, there's a couple of other issues in there. You know, we'll see. It looks like it's going to be a lot thicker than a smartphone. I just don't know the degree to which a regular same Google Glass issue, right? Yeah. Do you want a second thing that you have to keep on you? Like alongside your smart phone?

SPEAKER_06: That's just going to be my question. So I'm glad. So is this a something that should be replacing your phone or is this another thing that you are supposed to be holding on to?

SPEAKER_02: I think the goal is for it to eventually replace it, but at this point, you will still need to have both. So like carry another thing, you know, you know, walking around like a drug dealer. You have two different. It's kind of it's a big device and it's not tiny and it has a big screen. So it's just like, well, you just made a different kind of smartphone. Again, what am I gaining? Yeah.

SPEAKER_10: It's what he's holding in this video, correct? Yeah. Yeah. Yeah.

SPEAKER_06: Yeah. Okay. And what more is like that on top of tech person is definitely the person who carries a power bank with them. So that's a look and that's me. So that's three things you're rocking with. I know I have so many power banks and yeah. And ADHD has blocked every single one of them from being useful because I forget to charge them or I forget to bring them.

SPEAKER_10: It also just doesn't look comfortable. It's bigger than his hand.

SPEAKER_02: Oh, yes. It's not a whole lot. We'll get a better look at it in a second because I want to show you when he's talking, he's trying to make the case for this and one of like he spends a significant portion of his not very long keynote on him, Rick rolling himself in the most iffy. You've got to fucking watch this. It is so painful.

SPEAKER_12: Our one has an eye on board camera designed for advanced computer vision. It can analyze surroundings and take actions in real time. So activate the eye. Just double tap the button.

SPEAKER_02: Oh, funny seeing you here, Rick. It's a picture of Rick Astley that he points at at.

SPEAKER_07: Let me take a look. Never going to give you up. I know enjoy.

SPEAKER_12: What am I getting Rick rolling my own keynotes? Let's go on to the next one.

SPEAKER_06: I'm reading a free tech keynote speaker to take input class.

SPEAKER_02: I don't recommend that to normal people.

SPEAKER_06: Because you're like no selling your own joke. Just like what? You're not even laughing at it. Why am I going to laugh?

SPEAKER_02: I will say this that was scripted almost exactly the way Tim Robinson would have written it. Oh, man. But Sam Richardson could have delivered that bit better. Oh, 100%. So that is very funny. But I find this next clip more disturbing because it shows this kind of desire that the people that are the early adopters here have not just for more convenience, but to hand over like the power to choose to a robot that's basically just pulling the first advertised result from Google. Like it's kind of messed up.

SPEAKER_12: I can also use R1 to order food. Get me a 12 inch pizza from Pizza Hut delivered to here. The most ordered option on the app is fine.

SPEAKER_07: Ordering a 12 inch pizza from Pizza Hut. Since you mentioned that the most ordered option is fine, I will select that for you. I just created an order for a 12 inch pizza. It's going to be hand tossed with a classic marinara sauce and topped with regular cheese. Please confirm your order.

SPEAKER_02: That sounds really good. I just confirmed an order. He is to click it again. Just like on a smartphone. Look at the device and click it.

SPEAKER_06: Here's a freebie for any of even Richard Lawler who has been in the game for a long time. But any of your tech bloggers just do a side by side with this video and physically do everything he's doing real time. Because a pizza would have been ordered way faster if you would have just pulled out your phone.

SPEAKER_02: Well, and you could get the pizza you want.

SPEAKER_02: What? No way is that the most ordered product from Pizza Hut. I don't believe it.

SPEAKER_06: Who is definitely paying nonsense?

SPEAKER_02: Yes. Yeah. It's just like, yeah. And there's the next clip. I don't think we'll actually play it. But like it is, it's him saying, hey, playing out like a three day vacation in London for me. And as far as I can tell, the AI goes for like the first top 10 list of things to do in London that it finds, which was probably written by an AI and then makes an itinerary based on those. And it's like, first off, are you that basic second planning a vacation is fun. Is that not a thing that you like want to do?

SPEAKER_06: Yeah. You're so right. Why would you? Because I look, the reason you would go to like a travel agent is because they are experts at it. They're going to find the most fun thing for you. But outside of that, yeah, I want to play in the cool stuff I'm going to do, you know, and yeah, what about people with fears, you know, or people without skills, which is definitely going to be a large margin of people who do this. So you you're in London now. You're you're you're you're in a minute takes you on a trip to Malta to go scuba diving. You don't even know how to swim. And now you sit in there. You sit in there like you already paid for, bro.

SPEAKER_02: Yeah. You let your fucking thing work for you. It's on you. Yeah. It's silly, right? And I don't want to be I'm going to say this is not the most direct parallel to cult shit we'll have. But watching this, I couldn't help but think about a cult that was like the subject of our second episode for this year, The Finders. And it was one of those things. The guy, Marion Petty, who ran it was like running games is the way he framed it. And people would join the cult and give up their agency and he'd tell them go take a job in this city or like go follow this guy and take notes on him for a year or like have a kid and raise it this way. And this is stuff like this is really common within cults. One of the appeals of a cult to a lot of people is that you both get a sense of purpose by following the cult and whatever things that's going to do. And you give up the burden of having to choose a life for yourself. And this is such a common thing in cult dynamics that psychologist Robert Lifton, who's kind of one of the foundational minds in studying cults, described it as voluntary self-surrender^[2]. And it's a major characteristic of a cult. Many of the finders were not, you know, these are not dumb people. These are not like rubes. These are not hillbillies. As they're often portrayed in our popular media, a lot of the finders had Ivy League degrees, one of them owned an oil company. And these guys still handed their lives over to a cult leader. Haruki Murakami, writing about Aum Shinrikyo, which is the cult that set off a bunch of poison gas in the Tokyo subway, noted that many of its members were doctors or engineers who quote actively sought to be controlled. I found a lot of this really information on the fundamental characteristics of what makes something a cult in an article by Zoe Heller published for the New Yorker back in 2021^[2:1]. At the time, she was kind of looking at QAnon and trying to decide, there's not like a clear guy. And there's, that's the cult leader. And there's not like a geographic center to this. And usually there is with cults in history. Does this still qualify? And I think a lot of people would agree that like, yeah, it does. I think a lot of experts tend to agree that, yeah, it does. And when she was looking at the QAnon movement as a cult, Heller noted this. Robert Lifton suggests that people with certain kinds of personal history are more likely to experience such a longing. Those with quote, an early sense of confusion or dislocation, or at the opposite extreme, an early experience of unusually intense family milieu control. But he stresses that the capacity for total is submission lurks in all of us and is probably rooted in childhood, the prolonged period of dependence during which we have no choice but to attribute to our parents and exaggerated omnipotence. And I found particularly the bit where he's talking about like, yeah, if you have an early sense of confusion or dislocation makes people crave this kind of to give up this kind of control and responsibility. The people running these AI companies, and maybe not necessarily at the very top because I think those tend to be pretty cynical, realistic human beings. But like a lot of the people who are in them and a lot of the people who are latching on to AI as a fandom online are people whose childhoods and adolescence is like all of ours were shaped by 9-11, the dislocation and change that that caused. And their young adulthoods, a lot of these people like us will have come of age around the time of the 2008 crash. Many of the people who are younger in the AI fan base are maybe zoomers and stuff. And a lot of them are people who have really ugly ideas about like artists shouldn't charge for shit or whatever. But also, these are people who a lot of them came into their careers, went into STEM fields because they were told coming up, the tech industry is the same thing. The tech industry is the safest place to make a good living for yourself. And that all fell apart a couple of years ago. It started to at least tech layoffs began. So again, dislocation, chaos, the sense that like, what else am I going to entrust my life to? I thought I had a plan and it fell apart.

SPEAKER_06: Yeah, you know, this is where if he's going to get real philosophical and big player. But I just finished All About Love by Bell Hooks. And you know, she often talks about like the wandering life with lovelessness and that searching for it and not having it. And I feel like that goes hand in hand with what you're saying, right? Where it's like, I want a sense of belonging and I want to feel like I'm a part of people. And whether that is running into Target and trampling people for Stanley Cups or it's being a part of like what you perceive to be the next big thing. Like, I think that is the biggest kind of selling point for a lot of these AI people is like, this is the future. Like, that is almost every person who starts a 50 tweet thread with shitty examples of why AI rock starts it with. This is the future and you just got to get over it. And there are so many people who just want to be on the ground floor of that. They want to be the people who were on it because how many times, even I, you know, when you have that like time machine question, you're like, oh, stock and Starbucks. Oh, an apple, you just want to be there before it gets big. So, so when it really at the end of the day, it all comes down to commerce. You want to be at the top when it all shifts. And that is actually the danger in this for me is the commerce. I think about it often because, you know, like you're saying, it, it, like orders the top Google search. Google is currently in courts right now fighting against, you know, basically shaking down companies to see who would be the top one. So like the future of this being actually a, you know, a useful app kind of lives now in that case because if Google wins and they can put whoever's on top, that's only going to make it more valuable where they place who's on top because people are using these weird rabbits, you know, exactly. It's, yeah, it is to them. They see the beginning of the future. And I feel like to me, I'm just looking at all the ways it can be abused because if we just look at everything that has come before us, we have to think of the ways that it has been abused

SPEAKER_02: and all the ways it'll be a worse future, you know, and I think I really like that you brought up the panic they try to incite and the rest of us, the like the FOMO where it's like, this is the future. Get on board or you're going to get left behind. Right. That is, that is the cult recruitment tactic. Right. And what they're trying to do, I just brought up that a lot of the people who are most vulnerable to this are the folks who like, yeah, they have this sense of like insecurity, dislocation. And they see getting on board with this early. They feel like a sense of security there. And by saying, you're going to get left behind. This is the only way forward. You won't be competitive. If you don't embrace this stuff that they're trying to induce that sense of fear and dislocation to make people vulnerable. And I want to read another quote from that art from that, that New Yorker article. The less control we feel we have over our circumstances, the more likely we are to entrust our fates to a higher power. A classic example of this relationship was provided by the anthropologist, brought us La Milano whiskey, who found that fishermen in the trow B and islands off the coast of New Guinea engaged in more magic rituals the further out to see they went and I think we all feel like we're getting further out to see these days, right? Like that's, that's, it's not hard to see why like, yeah, I'm near the shore. What? I don't believe in anything but what's right in front of me. And then like you can't say anything but water and you're like, no, there's a God and I can keep him happy.

SPEAKER_06: Yes, yes indeed. It's, it's tough. Yeah, everyone's just kind of grasping at what they can to just bolster themselves and sometimes you're grasping at some weird stuff.

SPEAKER_02: Yeah. Yeah. Yeah. And it's, it's, you know, it's noted often by, I think a lot of particularly atheists on the internet that like church attendances down people who identify as part of an organized religion. That is at its lowest level, basically ever. And this is true. These are real trends and they have, they have real effects but I don't think the fact that less people are religious in the traditional sense means they're less superstitious or spiritual. Yes. It's just that what they invest with that belief has changed in part because they've seen the world dislocate so far out of what most priests and, and other sort of like religious heads are capable of sort of explaining or comforting them over. Right? Yeah. Yeah. Religion is less comforting in a world is advanced as ours. For most people. Now this may seem like a reach still to kind of call what's going on around AI occult and I get that I ask you to bear with me here. And I do want to note there's nothing wrong with the inherent technology that we often call AI or at least not with all of it. That's a, because it's used as such a wide banner term for stuff is very just like a text recognition program that can listen to human voice and create an on the fly transcription. That's an AI. That's an example of that kind of technology, right? Like it gets folded in there. That's one of the things an AI has to do recognizing language in like facial recognition to recognizing faces. If you're ever going to have an actual artificial intelligence, those are two of the baseline capabilities that it needs. Chatbots obviously are a big part of this along with like the sundry tools that are being used now to clone voices to generate deep fakes and fuel are now constant trip into the uncanny valley. CES featured some real products that actually did harness the promise of machine learning in ways that I thought were cool. As I noted, it could happen here. There's like this telescope. It uses machine learning to like basically clean up images that you take with it at night when there's like a lot of light pollution. So you can see more clearly. And I'm like, yeah, that's dope. That's great. But that lived alongside a lot of nonsense. You know, chat GPT for dogs was a real sign I saw. And like there was an AI assisted flashlight to help you not be a premature jacket.

SPEAKER_06: Because of course, that's the one that popped on my timeline is like, and it was like, and then they gamified it where you go to different planets. You defeat the planet. So I'm like, what do you keep talking about beating the planets? So how do I lose? Is it when I bus about bus you lost? Because you're now introducing shame to sex again. And I thought we finally got out of the way. I can't be level one.

SPEAKER_02: Those kind of bad ideas. That's all par for the course for CES. But what I saw this year and last year, not just at CES, but just over the year in the tech industry from futurist fanboys and titans of industry like Mark Anderson, is a kind of unhinged messianic fervor that compares better to Scientology than it does to the iPhone. And I mean that literally. Mark Andreessen is the co-founder of Netscape and the capital firm, Andreessen Horowitz. He is one of the most influential investors in tech history. And he's put more money into AI startups than almost anyone else. Last year he published something called the 'techno-optimist manifesto' on the Andreessen Horowitz website. On the surface, it's a peon to the promise of AI and an exhortation to embrace the promise of technology and disregard pessimism. Plenty of people called the piece out for its logical fallacies. For example, it ignores that a lot of tech pessimism is due to real harm caused by some of the companies Andreessen invested in, like Facebook. What's attracted less attention is the messianic overtones of everything Andreessen believes. Quote, we believe artificial intelligence can save lives if we let it. Medicine along with many other fields is in the stone age compared to what we can achieve with joint human and machine intelligence working on new cures. There are scores of common causes of death that can be fixed with AI from plane crashes to pandemics to wartime friendly fire. Now, he's right that there's some medical uses for AI. It's being used right now to help improve the ability to recognize certain kinds of cancer. And there's the potential for stuff like in-home devices that let you scan your skin to see if you're developing a melanoma. And there's debate still over how useful it's going to be in medical research. I've talked to recently some experts and I've read some stuff that like there are some reasons for caution too. For some of the same reasons we should have caution with this everywhere. There's also disinformation that spread medically with AI even to doctors. And some of the patterns that using this stuff gets medical professionals into can make them discount certain diagnoses as well. So I don't say that to like say there's not going to be some significant uses for some of the way this technology works medically. Some aspects of AI will save lives. It's just the evidence right now doesn't suggest it's going to completely revolutionize medical science. It's another advancement that will be good in some ways and there will be some negative aspects of it too, right? It's also very much not fair to say that like we're going to reduce deaths for human beings as a result of AI because right now the nation of Israel is using an AI program called The Gospel to assist it in aiming its airstrikes, which have been widely condemned for their out exceptional outstanding in many cases genocidal level of civilian casualties. It's just outrageous.

SPEAKER_06: Oh 100% and you know that's exactly what's going on is a genocide and you know the language and none of these speeches says that's much fair. Even more so, yeah like you're saying another thing I want to point out which you might have been about to say and I'm already jumping ahead is how I think it was chat GPT that has quietly switched their terms of service to say that it wouldn't be used for like weapons to hurt people.

SPEAKER_02: They're going to use this shit for weapons for sure.

SPEAKER_06: Yeah and now it has been quietly scrubbed from those terms of service and we do need to talk about that because there's just so many things that we have grown accustomed to with tech that I think is dangerous as we get into things that have more room for error because we're used to updated terms of services on our iPhone, right? Every time we grab an update it's like here's a new terms of service and you just kind of scroll through it and you go, yeah, because you're like, yeah, you know this is just a phone. It's not going to be used for anything weird yet. So you're comfortable but like when you're doing the same thing with these chat GPT machine learning situations where you're agreeing and you're agreeing, okay, I will help this thing learn. And now you are just actively helping it learn how to be an assassin, what happens there.

SPEAKER_02: Yeah and it's again it's this back and forth where on one hand there is some technology like AI enabled robots that can go run onto a battlefield and pick up an injured soldier. I have no desire to see some random private bleed to death in a foreign country. Fine with that or anti missile missiles right using AI to intercept and stop a missile from blowing up in a civilian area sounds fine like I don't I don't want random people to die from missiles. But it's also going to be used to target those missiles and to say like based on some shit we analyzed on Twitter or whatever we think wiping out this grid square of apartment buildings will really get a lot of the bad guys. Oh yeah, best one at least we should blow them up. Exactly right. It's just it's certainly it's certainly not fair to say there won't be benefits but it's absolutely unclear in every field of endeavor whether or not they will outweigh the harms right. And even if they do to what extent you know because a lot of what I'm saying suggests that even if the benefits outweigh the harms in a lot of fields. It's still not going because of the extent of the harms in part it's still not going to be a massive sea change right. There are a lot of reasons for caution but Mark has no time for doubters in fact to him doubting the benefits of AGI. Artificial general intelligence is the only true sin of his religion quote we believe any deceleration of AI will cost lives deaths were prevented that were preventable by the AI that was prevented from existing is a form of murder. And that's fucked up. That's really dangerous to start talking like that. Oh yeah. And this is the more direct cult comment here. I want you to compare the claim mark made above that slowing down AI is identical to murder. I want you to compare that to the claims the Church of Scientology's makes because the Church of Scientology they have this list of practices and the beliefs that they call tech right. And they believe that by taking on tech by engaging with it people can become clear of all of their flaws and by doing that you can help you can fix all of the problems in the world right. The Church of Scientology on his websites claims that its followers will quote rid the planet of insanity war and crime and in its place create a civilization in which sanity and peace exist. How is that in any way different from Mark Anderson saying all of the shit that he's saying right then it's going to like create this this this we're going to revolutionize medicine we're going to like end friendly fire we're going to cure pandemics we're going to stop car crash deaths. What is what is the difference right. And Scientology uses that claim that Scientology tech is so necessary it's going to fix all these problems so anyone who gets in the way of the Church of Scientology and the deployment of this tech for mankind's benefit is subject to what they call fair game. A person declared fair game quote may be deprived of property or injured by any means by any Scientologist. And again Mark Anderson has not said that in his techno optimist manifesto in fact he makes some claims about like none of no people are our enemies right. But if you're saying you're a murderer for slowing this down it's not hard to see how some people might adopt a practice like fair game eventually right that's how where else does this go is my what do we do with murderers what is I feel like the general rule across all creeds, yeah across all beliefs is typically murderers are bad and should die.

SPEAKER_02: At least be punished, right? There's a punishment for murder most people agree yeah

SPEAKER_02: So the more you dig into Anderson's theology, the more it starts to seem like a form of technocapitalist Christianity. AI is the savior. In the cases of devices like the rabbit, it might literally become our own personal Jesus. And who, you might ask, is God? Quote, we believe the market economy is a discovery machine, a form of intelligence, an exploratory evolutionary adaptive system.

SPEAKER_06: Oh, God.

SPEAKER_02: Through this concept of reality, capitalism itself and capitalize the sea there because it's a deity has chosen to bring artificial general intelligence into being. All the jobs lost, all the incoherent flots I'm choking are in and out. All the Amazon drop shippers using chat GPT to write product descriptions. These are but the market expressing its will. Artists have to be plagiarized. Children need to be presented with hours of procedurally generated slop and lies on YouTube so that we, one day, can reach the promised land of artificial general intelligence. If he isn't it worth it?

SPEAKER_06: Oh my God. It's so, I didn't know, you know, one of the biggest criticism with AI is that it, you know, it is one of the effects of when, you know, creativity and commerce meets, commerce will always try and kill creativity. Yes. Because it is commerce is more concerned with the buck than it is the outcome or what it takes to get said buck. And that was, that was going to be a whole thing I was going to drop at some point and they just said it for me. They just sent like, like they, I didn't, I thought it was more veiled. I thought it was more hidden. But you know, that is, that is why you can, you can try and say that it is ethical to take from artists because we're making it easier for you. Your fingers hurt doing all that drawing, but it learns from you. And, and, and, and, and, you know, don't ask how you're going to get paid, but if it learns from you, we can, we can give the people what they want without causing you all this labor, you know, but who gets the money? It's, it's always the dork behind the computer who did the code that is just essentially stealing from all of these people and learning from them and then just producing this amalgation of everything they've done.

SPEAKER_02: Yeah. Yeah. Absolutely. Also, I should note, I'm trying to be consistent about this. I wrote it down. And then I think slipped into it. It's Andreessen, Mark Andreessen, Andreessen Horowitz. It's just a weird name that I'm not used to saying. I, I wrote this down and then immediately forgot to correct myself at the start of the podcast. Again, folks, hack and a fraud, but you know who can own, well, actually, I won't say only humans can be hacks and frauds like that because the AI is absolutely mispronounced shit and gets shit wrong too. I guess that maybe they are getting conscious. Can they build an AI that's as much a hack and a fraud as I am? We'll see. So no. Thank you Sophie. I appreciate it. AGI is treated as an inevitability by people like Sam Altman of OpenAI who need it to be at least perceived as inevitable so their company can have the largest possible IPO, right? There's a lot of money on the line in the people with money believing all of the promises that Andreessen is making. This Messianic fervor has also been adopted by squadrons of less influential tech executives who simply need AI to be real because it solves a financial problem. Venture capital funding for Big Tech collapsed in the months before chat GPT hit public consciousness. The reason CES was packed with so many random AI branded products was that sticking those two letters on a new company is like, they treat it like a talisman, right? It's this ritual to bring back the rainy season. If you throw AI in your shit, people might buy it. There's versions of this like laptop makers are throwing AI in everything they do now just because laptop sales soared during the start of the pandemic but they plummeted because people don't need to buy laptops all the fucking time. Most people wear them out, right? Yes, exactly. And again, this comes in if you can get people in the code, if you get them scared that they're going to fall behind without AI, then maybe they'll buy a new AI and I have laptop because they're like, well, this is what I got to do to stay competitive, you know? The terminology that these rich tech executives use around AI is generally more grounded than Andreessen's prophesying, right? But it's just as irrational. The most unhinged thing I heard in person at CES was from Bartley Richardson, an AI infrastructure manager at NVIDIA who opened a panel on deepfakes by announcing, I think everybody has a copilot. Everybody's making a co-pilot. Everybody wants a co-pilot, right? There's going to be a Bartley co-pilot, maybe sometime in the future. That's just a great way to accelerate us as humans, right? And it's funny, he's named Bartley. If you know your old Star Trek and you can remember Barkley, the Sad Ensign, he sounds like that guy and resembles him.

SPEAKER_06: Oh, man. What's funny about that speech is it sounds like he's trying to convince himself to. He's like, he's like, he's going to accelerate us, right? Yeah, we're not wasting our time, aren't we?

SPEAKER_02: Yeah, yeah. Again, later in a separate panel in NVIDIA in-house counsel, Nicki Pope, who's like the only skeptic they let on, cited internal research showing consumer trust in brands fell whenever they used AI. This gels with research published last December that found around 25% of customers trust decisions made by AI less than those made by people^[3]. No one on stage bothered to ask Barkley. It was like, okay, you want to use this thing. We know your own company has data that it makes companies less trustworthy. Are you worried that if you use it, people won't trust you? Like, is that not in your head? And that was kind of the pattern at CES. All of the benefits of AI with some very specific exceptions. Most of the benefits of AI were touted in vague terms. It'll make your company nimble. It'll make it more efficient, you know? It'll accelerate you. Harms, though, while they were discussed less often, they were discussed with a terrible specificity that stood out next to the vagueness. One of the guys in the Deepfake panel was Ben Coleman, and he's the CEO of Reality Defender, which is a company that detects artificially generated media, right? Their job is to let you know if something's AI generated. And he claims that his company expects half a trillion dollars in fraud worldwide this year just from voice cloning AI. Not fraud from AI, just the fake voice AI, half a trillion dollars.

SPEAKER_06: Yeah, here's the thing, too, that I think is the scariest part of AI that I don't think we talked about yet, is that everyone can use it. You know, like, this isn't a thing that is like, oh, well, it's, you know, these companies, it's too expensive. People are priced out, and we just got to hope that everyone's good. You're getting like SpongeBob, wrapping Kendrick lyrics on TikTok. You know, my buddy, Benium, jokingly, like promoted his show and made it look like he was having a FaceTime with Obama, and it was like, pretty good. The only reason you knew it was fake was because of just the nature of the video, but like, at what point is someone going to stop and go, hey, we shouldn't have technology where someone can impersonate world leaders.

SPEAKER_02: Yeah, it's bad. And, you know, to be honest, that's not even just because like, I think everyone is ready for the idea that like, yeah, people are faking Obama or by because we've done little versions of that for years now. I think the scariest thing is people aren't ready for their loved ones to be imitated by AI, and that is a thing that is happening in 2023. And this has happened to a lot of people. There was a specific case that kind of went viral of this mother who got a call from what sounded like her kidnapped daughter. And like, the AI generated the voice of her daughter, and then a guy was like, give us money or we'll fucking murder her, right? And her kid was never kidnapped. She very nearly sent the money because who wouldn't, right? Yeah. Like, if you don't know that that's a thing that can do, who would not? Yes. Yeah. Like, that's a, that's a, and it's the AI was able to clone her daughter's voice because her daughter has a TikTok, right? It doesn't take that much, you know? And this is why, by the way, that people are talking about ways to mitigate this. I think one of them is like, have a family password or something where it's like, all right, if I'm fucking kidnapped, I'm going to say the password, you know? So that, you know, some random person with your TikTok won't know it, or at least has to try harder to guess it. So great. Thanks to AI. Now we have to have passwords for our families in real life. Cool.

SPEAKER_06: Yeah. If you, if you want to see that, thanks, Kim, you got to know the password.

SPEAKER_02: Yeah. Fucking great. Yeah. At CES and at the substacks and newsletters of all these AI cultists, there's no time to dwell on problems like these. Full steam ahead is the only serious suggestion they make. You should all be excited. Google's VP of engineering, Bishad Bizati tells us during a panel discussion with McDonald's executive. If you're not using AI, Bishad warned, you're missing out. And I heard versions, variations of the same sentiment over and over again, right? Not just this stuff is great, but like you're kind of doomed if you don't use it. And I will give Nikki Pope was the only skeptic really who had a speaking role in CES. It is not coincidental that she was an academic and a black woman because her background is studying algorithmic bias in the justice system. Yes. And so she, she, she had some really good points about like the actual dangers this stuff had. She was on this, the panel, she was almost governing AI risk and kind of her like the partner on the panel, the guy she was talking with was Adobe VP, Alexandra Coston. And she urged the audience, I want you to think about the direct harm algorithmic bias could do to marginalized communities. Quote, if we create a that desperately treats one group tremendously in favor of another group, the group that is disadvantaged or disenfranchised, that's an existential threat to that group. And she was specifically like people talk about the existential threat of an AI going crazy and killing us all, but like that's not as realistic as what we know will happen.

SPEAKER_11: Yes.

SPEAKER_06: Oh, yeah. When you have these banks that like automatically, you know, are using AI to try and approve, you know, loans and Deontay since his name and they're like, not approved.

SPEAKER_02: Yeah, exactly. It's like, no, good. And I am glad she was there. She again, she still works for a company that's going to make money off of this. She's not like a doomer on it, but like at least one person was being like, could we please acknowledge there are dangers?

SPEAKER_06: I know, because here's the thing is, and I truly believe this and you were basically saying this earlier that AI as a tool is fine. Yeah. I mean, when you when it is in a tool is something that is always held and used by a human that there's the checks and balances. It is only as evil as the person who's using it. And that is just any item, physical or digital will ever, you know, will always be under. But the moment you're like, I'm going to give you free reign based on information. And how many times has an article gone online that was like, it's scanned Reddit or it's scanned Twitter and it's racist now, you know, like there's. And we still like went full steam ahead with producing this and thinking we're right. We know, especially when you see a lot of these tech leaders being predominantly white men, and we know that in general, most white men don't care about protecting marginalized people. They care about getting their bottom dollar. They see it as a rare occurrence because they don't live that existence. They don't have the data—pun intended—to build something to defend against it because it's not a real problem to them because they don't see it. And that is beyond just them being them and more into as humans, a lot of times, if you don't do the work to see it and understand what happens to other people outside of your perspective, you're just going to believe that it's not real and it's or people are exaggerating or it's this and it's that. And when you are gung-ho and you have drunk the Kool-Aid, that is the AI Kool-Aid, and you are telling people that this is the future. We have to do it. You're going to push ahead. But like we've literally seen a clear-cut example of what happens when you push past safety and you just do what you want to do just because you have a whole bunch of money and a mad cat's controller. Like it gets dangerous.

SPEAKER_02: So as you started talking about like the dangers of certain tools, right? And how the value of the tools has you. I literally looked over at the gun on my table, right? And we all agree, even people who really like them, there should be regulation and I think the bastards are like, agree, more regulation, but they are like again, not to say that it's sufficient. But there are a lot of laws about like where you can carry a gun legally, how you can buy a gun, right? And because people understand that like, yeah, if a tool is that powerful, there should be limitations and things that you can do that get them taken away from you forever, right? I don't know how we do that with AI, but I don't think that's a reason not to try. Right? Yes.

SPEAKER_06: You don't do what they did in that movie Hackers. And it's like you're just ran from the internet till you graduate high school or whatever it was.

SPEAKER_02: Yeah, yeah. Now, Costen claimed that the biggest issue, again, so Nikki, Nikki Pope is like, yeah, I think this stuff could really hurt marginalized communities. And Alexandria Costen from Adobe responded that like, well, I agree, but the biggest risk of generative AI isn't fraud or plagiarism. It's not using AI, right? He claims like, this is as big as the internet and we all just have to get on board. And then I'm going to read verbatim how he ends this particular statement. I think humanity will find a way to tame it to our best interest, hopefully. Hopefully.

SPEAKER_06: Great. Cool. No way.

SPEAKER_02: Why? Awesome. Awesome. And the whole week was like that. Again, these really specific devastating harms and then vague claims of like, yeah, we're all just going to have to do it. You know, and I brought up Scientology earlier, but when I think about touting like vague claims of world saving benefits alongside, and it's going to hurt to when you have to accept the pain, I think of Keith Raniere, right? The next same guy. We all remember Keith, you know, like most cult leaders, Raniere promised his mostly female followers. You'll get all these benefits. I'm going to like, he'll you'll be extra productive. You'll be super good in your business, super good in your career, but you have to follow my commands because I have to retrain you on some stuff. And so it's going to be uncomfortable, right? And the end result of this is a bunch of them branded them their flesh and part took in sex trafficking. Yeah. You know, these techs, these tech executives are not Raniere, but I think they see money in [takin] some of his tactics, right? The benefits are so good. We just have to accept some pain. You know, I got to hurt you to rebuild you better. Now, all of the free money right now is going to AI. And these guys know the best way to chase it is to throw logic to the wind and promise the masses that if we take it, we're going to make it better. And that's the masses that if we just let this technology run roughshod over every field of human endeavor, it will be worth it in the end. This is rational for them because they're going to make a lot of money, but it is an irrational thing for us to let them do. Why would we want to put artists and illustrators who we like out of a job? Why would we accept a world where it is impossible to talk to a human when you have a problem and you're instead thrown to a churning swarm of chatbots? Why would we let Sam Altman, Hoover up the world's knowledge and resell it back to us? We wouldn't. And we won't unless he can convince us that doing so is the only way to solve the problems that scare us. Climate change, the cure for cancer, and into war, or at least into the fear that we will all be victimized by crime or terrorism. All of these have been touted as benefits of the coming AI age if we can just reach the AI promise land. And we're going to talk about some of the people who believe in that promise land and what they think it'll be like. But first, if you know what is the real promise land?

SPEAKER_02: So I want to talk about Silicon Valley's latest subculture emphasis on the cult. Effective accelerationism or E/ACC, EAC, I think is probably how you could pronounce it. The gist of this movement fits with Mark Anderson's manifesto. AI development must be accelerated without restriction, no matter the cost. EAC has been covered by a number of journalists, but most of that coverage misses how very spiritual, some of it seems. One of the inaugural documents of the entire belief system opens with the statement, accelerationism is simply the self-awareness of capitalism, which is scarcely begun. Again, we see a statement that AI has somehow enmeshed itself with capitalism's ability to understand itself. It is some way intelligent and can know itself. And I don't know how else you interpret this, but as belief in a God built by atheists who like money a lot. The argument continues that nothing matters more than extending the quote, light of consciousness into the stars, a belief Elon Musk himself is championed. AI is the force the market will use to do this and quote, this force cannot be stopped. This is followed by wild claims that next generation life forms will be created inevitably by AI. And then a few sentences down, you get the kicker. Those who are first to usher in and control the hyper parameters of AI slash techno capital have immense agency over the future of consciousness. So it is not just a God, it's a God we can build and we can use it to shape the future of our reality to our own whims. And again, some of these guys will acknowledge maybe it'll kill all of us, but as long as it makes a technology that spreads to the stars, that's worth it because we've kept the light of consciousness alive. Wow, that's not I don't think the mainstream view, but you can definitely find people saying that and they'll be like, if you attempt to slow this process down, there are risks and they're saying the same thing and reason is you stop it from doing all these wonderful things. But also, I do kind of view that as a veiled threat, right? Because if if AI is the only way to spread the light of consciousness to the void, and that is the only thing that matters, what do you do to the people who seek to stop you, right? Who seek to stop AI? I actually am fine with extending the light of consciousness into space. I'm a big fan of Star Trek. I just don't believe that the intelligent hyper aware capitalism is the thing to do it. Again, too much of a Star Trek guy for that. When I look at the people who want to follow Mark Andreessen's vision, who find what these EAC people are saying is not just compelling, but inspiring, I think of another passage from that New Yorker article by Zoe Heller. Quote, not passive victims, they themselves sought to be controlled. Haruki Murakami wrote of the members of Aum Shinrikyo, the cult who's sarin gas attack on the Tokyo subway in 1995, killed 13 people. In his book Underground, Murakami describes most Aum members as having deposited all their precious personal holdings of selfhood in the spiritual bank of the cult's leader, Shoko Asahara, submitting to a higher authority to someone else's account of reality was, he claims, their aim. Now, the EAC manifesto newsletter thing used the term techno capital in conjunction with AI. This is a word that you can find a few different definitions on because it's a wonky, like weird academia philosophy term, and there's a number of folks who will argue about how it ought to be described. But this is broadly kind of the same thing that Andreessen is referring to when he talks about the market as this intelligent discovery organism, right? And while there are a few different ways you'll see this defined, the EAC people and Andreessen himself are thinking about how philosopher Nick Land, who's the guy who's generally credited with like popularizing the term techno capitalism, defines it. Land is one of many advocates of the idea of a technological singularity, the point where technological growth driven by improvements in computing becomes irreversible, the moment at which a super intelligent machine begins inventing more and more of itself and expanding tech in a way that humans can't. As one of Land's fans summarized in a Medium post, a runaway reaction of self-improvement loops will almost instantaneously create a coherent super intelligent machine. It is man's last invention, the most notable of industries, AI, nanotechnology, femto technology, and genetic engineering will erupt with rapid advancements, quickly exceeding human intelligence. Now, obviously, the way Land writes is, again, kind of worth reading, but perhaps too dense for an entertainment podcast. So I'm going to read again from a substack called Regress Studies by a writer named Santi Ruiz, kind of talking about this idea of techno capitalism that Land has from a more critical standpoint. Quote, Nick Land who coined the term is a misanthrope, he doesn't like humans much, so the idea that there could be an entity coming, already being born, drawing itself into existence by hyperstitionally preying on the dreams of humanity, cannibalizing their desires, wearing the invisible hand is an ideological skin, he's into that. Techno economic interactivity crumbles social order in auto-sophisticating machine runway, as he would put it, and that's good, you're being colonized by an entity that doesn't care about you, except in so far as you make a good, we'll talk about hyperstition in a little bit here. So Land is the guru of accelerationism. You might not be surprised to learn that he is at a voted following among the far right, this is because he is quite racist, anti-democratic, and obsessed with eugenics. Now, his eugenics are not your grandpappies eugenics, for him it involves gene editing, which will be available to greater extent than ever thanks to AI. Land claims to dislike white nationalists and conventional racists because they don't see the whole picture. Quote, and this is me quoting from one of Land's publications. Racial identarianism envisages a conservation of comparative genetic isolation, generally determined by boundaries corresponding to conspicuous phenotypic variation. It is race realist, and that admits to seeing what everyone does in fact see, which is to say consistent patterns of striking correlated multidimensional variety between human populations or subspecies. Its unrealism lies in its projections. That's pretty racist. Land is listed by name in Andreessen's manifesto as someone you should read for a better understanding of the wonderful optimistic future he and his ilk have planned for us. He cites extensively Gregory Cochran, who posits that space travel, spreading to the stars, will solve our race problem because it's a natural filter, basically saying some races won't make it into space. So we don't need to be violent, like we just have to spread to space and that will do our eugenics, part of it for us. So that's cool.

SPEAKER_06: Yeah, you know, I'm stuck on this like, you know, journey to the stars through evolution thing, because I don't know, Robert, do you play Warhammer 40?

SPEAKER_02: Oh, my, oh, God, iffy, iffy. Of course I play. I've been playing Warhammer 40K most of my life.

SPEAKER_06: Okay, because this is on a very adeptus mechanics. And I'm like, yes, absolutely. And I'm like, I'm like, what is going on here? I'm deep and well, traitor. And now I'm like, no, this is them.

SPEAKER_02: What is fun is that like Warhammer 40,000, the Deep Fluff envisions a society that it's like a hybrid between the Federation and Star Trek and what these AI, Yak people dream of that is this utopia for like 10,000 years because they develop thinking machines and then all of the thinking machines turn on them and murder everybody. And so in the future, we just lobotomize people who commit crimes and turn them into computers for us because we can't have intelligent machines anymore.

SPEAKER_06: So we're just, we're going to, that's what it looks like we're marching towards with these folks.

SPEAKER_02: Yeah. Yeah. And obviously the thing that the Warhammer people are inspired by is like the book Larry and Jihad and Dune, which is the more artful version of that story with less orcs, which makes it inferior in my mind, but I do love Dune.

SPEAKER_06: So, you need more red because the red makes everything fast.

SPEAKER_02: Yeah, red makes it go faster.

X: Yes.

SPEAKER_02: So, Land concludes by imagining both racists and anti-racist binding together in defense of the concept of race. Right. That's what the result of AI is going to be the racists need race. You know, we're going to get so good at gene editing. The racist will get angry and the anti-racist will get angry because they're so in love with the concept of race. Well, we're just going to improve people annihilate racial differences through moving to the stars and the natural filter that that implies. By the way, the name of the article land wrote all this in is called hyper racism. So, cool guy. Glad Mark Andreessen cites him in his manifesto. Glad the biggest venture capital guy in the country. Yeah. He's like, yeah, read this dude.

SPEAKER_06: Yeah. Well, that's the sequel to flumes hyper realism.

SPEAKER_02: Yeah. So that's, yeah. And these guys don't tend to cite it as much. I think he could get in some of the deeper stuff. They're all talking about these techno capitalist concepts that Nick Land plays with. They don't talk about what I think is actually one of his most sort of insightful points, which is about a concept called hyperstition. And in brief, hyperstition is like creating things in fiction that become real and the process by which that happens. I think about that a lot when I think about things like the Butlerian Jihad, the war against the intelligent machines in Dune or, you know, what happened in the warhammer 40,000 universe. But I also think about how part of why these people are targeting creators, writers, actors, musicians, artists, like pen and paper, you know, painting artists and stuff, is because the only way out of this future they have envisioned is imagining a better one and then making it real, right? Like, and that is a thing that creatives have a role in doing. So if you can kill that ability, hand it over to the machines that you control, maybe you can stop them from this path of resistance. Motherfuckers.

SPEAKER_06: They're on. They're on to us. They're like, you want something better? We're going to take it away from you.

SPEAKER_02: So anyway, I think that's going to end it for us in part one. You know, this whole investigation in much more condensed form, just kind of really focusing specifically on the argument that there's cult dynamics to the fandom is being published in an article on Rolling Stone. I'll probably edit in like a the title or something here so you can find it. But check that out.