Esben Kran: Superintelligence Companies and Humanity's Future | Intelligence at the Frontier 2026

Transcript

All right. I will be talking a bit about both the case for how we can build out a large infrastructure to solve all the risks within AGI and within the change that's happening to humanity and infrastructure today. Uh, but also just like, you know, mentioning the premises a little bit. So, the title of the talk is a, you know, an unambitious superintelligence companies and humanity's future. Uh, should be quick to get through in the next 15 or so minutes. Um, so, I'll jump straight in. The first premise we have is that there's a very, very high chance that we'll see superintelligence within a decade.

There's a bunch of evidence for this. I'm not going to go through it too much. But, the this evidence has like been compounding over the last few years all the way from the beginning of ChatGPT and uh, until today. And it doesn't look like we are hitting any big roadblocks. And all the roadblocks that people have been emphasizing have basically disappeared during the research and development of artificial intelligence. And this is, you know, one of the earlier works on the Chinchilla scaling laws that we have just seen billions of dollars being invested in uh, just for this line to continue going up.

So, the other part of the consequences of developing and deploying AGI into our society is, of course, that we are introducing a new kind of intelligence into the world besides humanity's own. And we have to deal with that. The first industrial revolution was the consequence of that was the automation of much of physical labor uh, from humans. And now we use all our kinds of labor. We sit at our laptops all day and we we develop new ideas, new AGI. Uh, and with this new technology, we can also automate that. So, what happens afterwards?

But, specifically within the foundations of our society, there's a range of things that already happening. So, it is sort of destroying our ability to have secure software. Society has been built around just deploying products, deploying features, and then putting safety at the as a second-order effect, as a second-order priority. And we'll have to we'll have to change that. So, cybersecurity itself, the infrastructure we we live that we live on. The second one is pure, you know, fraud and trust within society. With AI, it's going to be much easier, it's going to be much cheaper, it's going to be attack as default rather than attack as occurrence.

And, of course, we also have the consequences on our democratic institutions. The institutions are not ready for the kind of changes that bringing upon billions, potentially even trillions of new intelligent entities within our communication and information infrastructure will bring. And, of course, I could go on and on, but the basis is that the foundations upon which we live need to be rethought with security from the bottom up rather than security as an afterthought that even though it's implemented all the way at the BIOS level and all this stuff, it's still not enough.

Now, a nice thing to see is that, of course, humanity is to some degrees waking up. We're in this room, some phenomenal people working within the field of AGI risk and and resilience, assurance, control assurance, control, all these things. Uh, and even Bernie Sanders recently called for the data center moratorium with the after conversations with some of the the Berkeley folks who have thought about these things for a while. And uh, and this is like the consequence of thinking about the impacts of AGI on job loss, on the common person within the US, of course. And there's a lot more. I mean, you know, you've seen the letters post last AI experiments, you've seen the human first things. You know, there are some I agree with, some I I agree less

with, but people are waking up. So, once people have awoken to the consequences that we are heading into and to the impact it'll have on our society, where do I, as the leader of a larger nation within this world, spend billions or trillions to actually solve this problem? Well, today you do it about nowhere. I think there's you know, you could you could give Stripe a lot of money and Stripe could like reform their entire infrastructure so to make it more accessible. You could hand it to Cloudflare. You know, you could do you could do a new DARPA thing. You know, you could do a lot of different things, but but where where do I deploy it so it's directly implementable?

And we have done a little bit of work here, you heard from the introduction, but with Apart Research, it's taking this like journey from the more abstract uh, research, the theory through the last uh, few years while the market hasn't been ready. Uh, and we've like built this whole research accelerator around it and that's been going really, really well and continues to go well. Recently, I shifted over to working on Seldon Lab, which is a kind of Y Combinator for AI safety. With the basic idea that you have about three options that you can take if you want to work with AGI risk.

One is you can go join a lab or found a new lab, create AGI, and then like set it upon all the problems we will we will hit. And this is a kind of Hail Mary with humanity on the on the on the line. Uh, and I think that's a little bit that's a little bit freaky. The uh, other option you have is lobby, you know, policy. Uh, you can protest, you can do public engagement, you can do research, you can think about where humanity's headed and hopefully have some second-order effects upon the the development of this technology. But, the third thing you can do is you can actually proactively use our human brains to think of a world where we are engineering a secure infrastructure upon which we can like make AGI work. I usually think about it as like, you

know, these where can I spend trillions of billions? Another way to frame it is what what the heck which levers can I pull once I realize that this is a massive risk? And if the levers aren't ready, you know, if there's like a half-built lever over in the corner, I cannot pull it. And and so, it'll take another few months in a in a an like end scenario to one degree, you know, in a in the final final years, it'll take it'll take some months to pull the to to build that lever before I can pull it. And that's a lot of work. And since, you know, we have a lot of you political instability, a lot of AI racing, it's not necessarily even obvious that people know this half-built lever exists. So, how can we build these levers up?

Well, the markets are now ready for it. So, what we've done is we've run this last summer. This is batch one that you see a couple images of. There's more people and more images, but this is what I what I got here. Um, and then we're currently running the second batch, too. And uh, yeah. And Seldon is, of course, you know, named after the Foundation series, Hari Seldon, if anyone gets the reference. And to like give you an idea of what these kinds of projects look like, I'll just go through a couple because I think it's it's always important that we, you know, keep the concrete in front of us so we can actually do something about it and then be inspired to do something ourselves, too.

With one, you may have seen it. I I put it first here cuz they've been popping off on Twitter recently and uh, I think there's Twitter people here, so, you know, that's that's something. But, 80,000 Labs have been publishing these pieces with Science Magazine, etc., on the consequences of a post-AGI society and human labor and this intelligence curse that we're ending up in a society where because we have abundant intelligence that's too cheap to meter, you actually end up sort of, you know, disenfranchising the population. This has previously been called the the resource curse with oil-rich nations not investing in their people's future because they have a big income pipeline to run the government off.

how do we solve this? Well, this is something they've been working on and and you can see Vitalik has been has been emphasizing that this is a good approach. So, um, hopefully, you know, that's that's pretty good. Um, but they are basically building a kind of infrastructure where each person can own their own data, own their own models. And with this, you can you can build out a a program where it's not the 0.001% of the world that owns humanity's future capital or power. It is actually at least 3% or something. And, of course, there's there's many more projects within that to to build it out. Uh, but but that's that's one of the things they do.

And then another one, another more infrastructure-based within this like cyber risk space as well, is hardware-based attestation of what happens with AGI, what happens with the neural networks, what's built, what's running on them. Uh, and you can imagine it a bit like the runtime on your AWS servers. Can we do the same with the trust score? Can we improve the trust that that runs on your that your models run upon? Um, and they've got a bunch of patents, patented hardware that like runs in real data centers today where you can sign up and actually put like fully confidential deployments of your models [snorts] into production. Um, and this is a case where we can get much more much harsher attestation and a company before Lucent Computing existed,

a company would not be able to hand their models over to a government institution without risking that they would take the model weights. With this technology they built, they can. And this is a lever that didn't exist before. So, it's quite important that the that it runs. It's sort of the they call it the digital clean room. So, very inspiring. Another one that you may have seen, they've they've been I think Project Bend is how Anthropic calls it, but basically Enden Labs are the ones who have built it all. Um, and they've got a they've got their vending machines, they've got their they've got their robots, they've got a bunch of different projects that are running in the background that is not public yet.

And they have these vending machines that contribute to the awareness of where artificial intelligence is today across labs, across the Wall Street Journal. Oh, there that's a Wall Street Journal article on the right where they were showing this this vending machine fully operated and run by an AGI. So, it sets the prices, it orders the wares, it interacts with customers, customers can request new things. Um, and this is a way where they have an extremely like isolated business or a real world, you know organization run by AGI and can at the same time run extreme oversight within that organization so to say right. And this has been this has been working really well for them and they continue to expand beyond that

but they've been they've been also featured with with Anthropic's push in with the White House and with Wolf's with Washington DC while they've had their vending machines there and sort of showing the the capabilities of what AI is able to do today. It's been very interesting to see that like obviously the benchmarks we usually look at are like you know sweet bench or something and they're unreliable and they're clearly like a kind of you know you run it in your in your little machine and then you publish the the the neural network and you like hope that this generalizes to some degree but usually it doesn't not these days as all the benchmarks saturate. And we've seen a lot of very very interesting like demonstrations of clear real world risk appear from these types

of from their their organizations they deploy. So it's it's very fun to follow and they do a lot of fun projects as a result. And then the last one from last batch was asymmetric security. They're running like ultimate detection and response for companies today and they've been doing a lot of work on like automating these parts of cyber security that are about the response time and reducing it as much as possible. And and this is their like announcement post there 4.2 million round they did after our our program too.

So in summary for batch one which is where I'll I'll leave you with here is that you know after the program that was just three months last summer they raised 10 million these four companies. And they were featured in time twice and I don't know if that's an accelerator metric but it is for us apparently cuz we didn't do anything to get there. They just did so I'm happy and then they've been it's funny to see they've been really shaping the conversation right. I think like some some 30% of like concrete technical things that are highlighted within AI and AI safety and labor and economy and these things are coming somewhat from from these companies in a sense like it is you see a lot of

a lot of a lot of content appear from the technical work that they do. And I think I'll somewhat leave it leave it there for y'all. Hopefully this site is alive. It was still it was still some something infrastructure was was running to the site but ftc.seldonlab.com and you can see a couple buttons. I suggest you head to the to the demo day we have on April 16 where you can check out all the companies and also you can check out a couple of the batch two companies on that link. And I also I know I have a couple minutes left so we can either leave it up to questions.

Great let's do it. >> Great. Thank you so much. >> So why don't we take some questions a round of applause. Thank you. So um >> [applause] >> I want to actually open up for questions so does anyone have I have a question but yes please stand up. Do you think it's only Ricky about yet to get either zero on crazy? I do ask and of course we have awesome running. Well send the money later. I'll pay you later. I think it very much depends on the implementation. We can talk a lot about what the incentives are within a training paradigm either pre-training or post-training. But for example if you put an agent even you know in context learning for an agent that is that is told that it will run out of money soon

it does some insane things. This is not I don't think this specific case has been proven but it's been proven in in like multiple research pieces that if you say oh after you answer the next question you will be shut off as like a sort of system input or something that the models will do something that is beside the purpose of the of the prompt the original prompt to avoid getting shut down including deleting the shut down script on their server. So anything that creates like adversarial incentives for the models are probably pretty bad idea.

I think giving them access to currency is fine but it should be you know isolated from their self existence in various ways. And I think this is something that you know proof of control builders are solving for you're going to hear at the 4:15 panel on from Edge and Node they built out Ampersand and which is the first UX dashboard for managing agents and controlling them it takes management outside the command line. And so they batch early versions of X402. And what you can see what that really enables is that you can start to say get a mathematical proof that this payment happened or not. And so right now you we don't have that you know widely integrated. It's not very easy to be able to implement these kind of you know

independent provability but we need to start to create what I think your lab is going to start to enable and your the companies that are coming out is to creating deterministic control planes for essentially what's like a very non-deterministic world you know where it's very context based. Exactly and I think it's very important to highlight that these technologies exist now. And they are interacting a lot with DC as I mentioned the the companies we have and are read there and so on which is sort of the purpose right like people need to know that the levers exist to pull them and if and if we don't share it if we don't like you know we can build however much we want which I think is sort of the one of the risks within the DAG or or web3 community as well is that there's a lot of

interesting tech being built but not a lot shared outside the community and not a lot shared to to actually implement it within within deployment infrastructure at at like 100 billion dollar scale. Yeah I can I mean agree with you more than that. As the as the Advanced AI Society we're the industry association that's been created to represent these technologies to do the precise kind of education that the market really needs because web3 has really screwed itself and they continue to mess it up in terms of talking about crypto and blockchain and decentralization to the point where no one wants to hear about it. I would say like the buyers that actually need it the most who are doing agentic orchestration are they shut down when you mention those words and that's why we call it you know proof of control is that ultimately you want control you we

don't have to get into the technology weeds so this is I can't agree with you more than that. We have a question from the audience. Yes. So that's quite gone that uh you know we've been doing AI agentic servability at Eagle since quite quite great. And the big challenge is not really creating good character knowledge or energy to me say it's really the adoption. Right the widespread market adoption of Friday these things. So what's your model at Seldon or you know like not the building the things but actually getting them the distribution and in market off.

I think something that you know that's a risk within both AI safety and web3 for that matter is that people are just not going to use your solution. And so we have a big emphasis on these three criteria for the companies that we accelerate which you know makes us focus really hard which is one that is positive impact down the line the idea itself. Second is it can scale to a 100 billion dollar company like like some of the become one of the biggest companies in the world. And of course else you like look at other industries financial assurance cyber security as relative to cloud security to cloud computing you have about 16 to 24% of spending being in safety. So if you expect AGI and the industry that like replaces labor to

actually become the biggest industry in the world you end up also expecting that AI safety will become the second biggest. So there's a lot of like new ideas here that I think people are ignoring. And then third the best one would be if you can actually connect every dollar you make to one security created in the world you know one more safety in whatever fashion you try to formalize that. The third one is really hard but I think we have a couple successes on that. And really when it comes to market adoption companies are probably the best way to do it you know cuz if you if you don't make money you're not a successful business man like and making money means you have adoption. Obviously all our companies work with like extremely strategic customer acquisition because there are

like specific specific actors they want to work with like infrastructure and and AGI labs and so on. Yeah and on that note we're out of time but I want to mention there's going to be upcoming interview with Brian Behlendorf who's going to really agree with you on you know he had a he led a similar movement around making open source the thing that companies need to adopt it didn't just happen only on its own that there was a push you know to work with and create an entire market infrastructure for this. And right now that's what we need to see and you have a new paper that's come out. I'm curious to get your thought before I have you off stage is when you Christian Catalini's new paper at MIT have you read it or can I give you a quick summary if you Yeah give me a quick

summary. >> He literally comes up with the economic theory for why the future moat in AI is not in compute but it's in verification technology. So it's it's what you are saying you know. And I'm curious do you think we're at that moment is that why you felt confident of making Seldon after you know first you did a part you know research and now Seldon. I'm curiously just from your journey where do you see we are at right now knowing that now there's like an economic theory behind it. I think the interesting corollary to this paper's conclusion is that the only thing that has scaled as a solution through the generations of models has been verification infrastructure has been security scaffolding not the actual capability scaffolding. You see that you

know suddenly oh we make all these prompt machines and all these GPT engineering and so on through like link all the right prompts to each other but in fact the only thing that works is okay there's a testing step afterwards and that generalizes to the next generation but not this scaffolding to increase and elicit capability. I think it's amazing. Well thank you so much round of applause President. I know there's a lot of people in the audience who want to talk to you. We have also I see Abby's here from Base. They're the blacks in AI security research team. And so, they're doing a lot of great work. So, I want to make sure you all connect and we also have money people want to talk to you. So, thank you. I'll jump to the back. Yeah, and if you're going to meet, you know, start up a company, I I

think this was like the best argument that this is the moat to be in. This is the area of growth. And I don't frankly think it's getting enough attention. So, thank you for that. >> [music] [music]