Adam Russell & Tom Kalil: What Do We Want AI to Do for Us? | Intelligence at the Frontier 2026

Transcript

Tom has crafted a lot of policy. Um, as you told me could you tell me a little bit what you did with the OSTP? Sure. Yeah, so I worked for two presidents, uh, President Clinton and, uh, President Obama. And for two different parts of the White House, the National Economic Council, which coordinates domestic and international economic policy, and the Office of Science and Technology Policy. And when the President in the State of Union would say, "Today I'm pleased to announce that we're going to do X." Uh, my job was to try to help come up with the X's.

Uh, so examples of things that I worked on uh were uh, the promotion of the internet in the 90s, the National Nanotechnology Initiative, which, uh, President Clinton announced in January 2000, which has resulted in $40 billion in funding for nanoscale science and engineering, getting first DARPA and then every agency the ability to do incentive prizes for up to $50 million, and then the, uh, BRAIN Initiative. Uh, where our early goal is to try to do for neuro science what the genome project did for genetics. So it was broadly interested in science, technology, and innovation.

Um, and as a direct, uh, beneficiary of the America Competes Act, that allowed, essentially, uh, incentive prize challenges for the government to put out. So rather than having to fund people to do the research, you would actually set this ambitious challenge and then people could respond. And I'm sure you all are familiar with like what what that means. Uh, but when I was at IARPA as a program manager, it was because of that work that we were able to launch, uh, you know, our first challenge, which, I think actually provided tremendous ROI. Um, that meant, obviously, that you were thinking of many different ways that the government could, uh, essentially achieve its mission, uh, without just sort of relying on sort of the standard playbook of of funding and policy. Um, what other Well, so where you are now, like what were other, uh,

sort of new mechanisms of of funding that you did in government, and what are you most excited about now? Uh, so broadly speaking, currently the US government has a widely utilized mechanism for making financial commitments that are contingent on failure. So, if you were to look at the balance sheet of the federal government, what you would see is trillions of dollars of loan guarantees. So, that is the government saying, "Hey Adam, if your business goes bankrupt, the US government uh will assume your debt obligations." So, it's an if-then statement, but the if is predicated on failure.

And so, what I think is interesting is increasing the capacity of the government, but also other institutions as well, to make financial commitments that are contingent on success. So NASA uh when the US retired the space shuttle, the NASA had to give the Russians $50 million every time we wanted to send an astronaut to the space station. And no one thought that was a particularly good idea. So, NASA said to SpaceX, "Build us a rocket that can go to the International Space Station, deliver and retrieve cargo, and we will buy rides on the rocket, but we'll also provide you with milestone payments for intermediate progress towards that goal."

During the Trump administration, the way Operation Warp Speed worked is the government said to Moderna and Pfizer, "Um if you get a if you develop a COVID vaccine which is safe and effective, then we will buy 300 million doses." So, there's an if-then statement, but now the if is predicated on you being successful as opposed to you failing. So, you know, if I were king for a day, I would create a marketplace of outcomes that has three participants, the entity that's willing to pay, like NASA was for the Falcon 9 rocket, the team that believes that they can achieve that goal, and the investors who are backing that team.

Um so, that works primarily in those instances where you can define what winning looks like. So, for example, for the DARPA self-driving car competition, the finish line was literally a finish line. There was an obstacle course. You had to get to the end. The second time they ran it, a Stanford team won. That's right. Larry Page was at the finish line, and he promptly acquired the winning team. Uh so, that's where Waymo >> Hello, Waymo. uh Google X came from. Yeah. Um I I know you've got here, but the conversation today, the ideas are just really incredible, robust. Uh you have some of these ideas well there.

Let's say uh somebody here is like deviates from what he believes like if there were an outcome that we achieve in some particular area of either yes, Jerry, or or privacy, example, how would they go about like trying to seed an idea like that in the government uh to to do this or trying to go negotiate discuss? Yeah. So, uh when I worked for President Obama, people would come visit me, and they'd say, "Tom, you're never going to believe it, but it turns out that the thing that I work on is really important." And I would say, "Gosh, I'd never heard that before. What is it that you would want the way else to do?" And that shockingly I have to say that at the time people did not have a good answer to that. So, the thought experiment that I always gave people is

uh you have a meeting with uh President Obama in the Oval Office, and he says, "If you give me a good idea for, you know, whatever, AI of human censorship, then I will call a lot of climate. Uh it can be a conference call, so they can be more in the one person of all right. If that person works for you right to hit by by being directed will be something cuz I've been boss. If it's someone outside of government, I could challenge to do something. So, you add a Now, wait. Do you have to tell me what your idea is, but in order to make your idea happen, who would I call alone ask to do? So, that that is the White House version of the Hillel Mark House. Okay.

If you could call anyone, who would you call, and what would you ask them to do? It's So, the premise of the thought experiment is that complex problems require building coalitions, and it is very difficult to build a coalition if you cannot articulate the members of the coalition and the mutually reinforcing steps that you want them to take in the form individual organization A takes action B in the pursuit of goal C. Check. Yeah, uh he mentioned the Heilmeier questions. Has anybody heard of those before? Oh, wow. Okay. Um yeah, if you've ever been through an ARPA, uh it makes you instantly sweat. I mean, my hands are sweating here. Uh cuz the George Heilmeier was a director of DARPA who came up with what's called the catechism. There are eight fairly simple, like deceptively simple

questions that essentially every program you see coming out of DARPA has to answer these questions. And they basically load on the same factors of like, what are you trying to do? Why can't we do it now? What are the limitations? But then, who cares? What's new in your approach? >> What's new in your approach? This is always the And by the way, the answer can't be uh well, I'm just smarter than everybody else. Um which rarely is the case. Um But you the story you just told is people come to you with ideas. Um one of the terms I used, which one I would use again, was harvestigate. Like, how do you How do we How do we use ideas that are here now, but also can seize your can seize your investigate, right?

What's the way for us uh to to find and identify like new new results of public goods? How How does the government do that? How should it do it? How do you do it? Yeah, so I think one thing that is underrated is unbundling idea generation from execution. >> Mhm. So, the way we usually do research funding is that we assume that the same person who has the idea should also be the person to execute the idea. And I don't think that's necessarily the case. So, I I think there are some instances where you just want a lot of ideas. And so, why why might you want to uh unbundle those? Well, in part because the person who has the idea may not be in a position to actually execute the

idea. So, for example, there are huge number of uh of opportunities at the intersection of uh AI and health where uh the you know, the the challenge is that a number of organizations are hoarding the data, right? So, you may have a researcher who could articulate, "This is what I would do if I could get access to the data, but I don't personally have the political juice necessary to make this happen, but you know, if you give me an opportunity to send my idea into someone, I can tell you uh you know, what what I think the the opportunities are. So, that's that's that's sort of one thing that I don't think that we do enough of is unbundle idea generation from execution.

Interesting. So, in in terms of the idea execution again, um or sorry, uh idea generation, um where we sit at DARPA is we are used to because there's a large budget, you're used to people coming to you, uh which makes you lazy in some cases, right? In particular, a lot of complaint, as you can imagine, is like, "Well, the talent in AI is very thin." And that's probably because the talent they see is the stuff that's going to you know, the the large tasks. Uh when you actually get out there and start hunting, you realize, "Oh, actually there's more than you think." Um what would you recommend for like a field strategist sort of approach? Is Renaissance Philanthropy doing that? Um Yeah so uh our vision is that in the same way that wealthy Italians fairly supported the Italian Renaissance by backing intellectual activity, today's

philanthropists are positioned to support a 21st century Renaissance that is fueled by these big five bases in science, technology, and innovation. And one of the ways in which we do that is we design and launch thesis-driven funds that have uh you know, the problem that we're trying to solve and why we think philanthropy can play an important role, the team that would do a world-class job of leading that effort, and the philanthropists and foundations that are backing that. And we can and do start with any of those three. So, sometimes someone comes to us and says, "I really want you to launch an initiative in this area, and I'm willing to be the anchor donor."

Sometimes we have the idea and we're trying to convince the rest of the world that we're right, so uh another DARPA alum, who was our chief scientist, Joshua Elliott, believes we're under investing in climate emergencies. So, those are areas where we're nearing an irreversible tipping point like the impact of the West Antarctic ice sheet on sea level rise. And in that case, we're finding that it's the rest of the world that we're under investing in this area that we should be doing more. And then sometimes someone unlocks in the in the forum with a brilliant idea that we get excited about and then apply support. Yeah.

I mean, can I ask about how crazy you can get? Um how how crazy can we get when it comes to like uh sort of All right, I'll You can pass. Uh especially in terms of like the funny mech business for example. So, here here's a my crazy idea is figure out some way maybe string is a mayon this. Figure out some way that you can have outcome-based challenges or or funding approaches that also reward the people who have already done the work and didn't even know they did it. Right? People who have already data for example to to the database so it was used later on for example PDB and alpha fold and they get chance to win awards.

Of what I was supposed to just the warm's low of the of knowing that they contributed to the public good? Well, that's I think that's super important. Uh won't won't buy you any altruism. Yeah, so I My risk hearing is that that's one of the levers that this community has which is retro active funding of public goods. So, uh So, I mean, they already thought of it? Yeah, they already thought Uh So, I think the issue is that uh I I I was talking to someone about this um is that there's this concept in people who who study social networks, not as in Facebook but as in, you know, relationships with between people. And there's this concept of structural

holes. So, if you were for example if you were look at a social graph of a high school, you would see blocks and nerds but very few people who are members of the both the the frog click and the and the nerd that maybe you were one of What called jerds. The uh Uh and so similarly, I think there's a danger that if you have this idea that sort of rose up in the crypto watching web3 ecosystem, it never makes it out of that because there's not enough people who are in position to play this bridging role between this uh concept like retroactive funding of public goods. With other communities that like Idris and Central supporting it. So, you need more

ambassadors and people who can engage in that sort of boundary spanning. So, for example, the idea for advanced market commitments grew up in the global health community because they had this very specific problem, which is left to their own devices, drug companies will not work on vaccines for poor people because they have no money. Um and so, I found out about it in the context of that market failure. And that what I discovered is no one in any other field other than global health had heard of that idea. So, when you find one of these ideas, you need people who are going to engage in sort of proselytizing and boundary spanning, otherwise it will just get stuck. And the only public

goods that will be funded are the ones that are of direct importance to the web3 blockchain crypto community as opposed to saying, you know, how do we get more people interested in this idea of decentralized funding of public goods. Yeah. Um and again at Darpa, we we uh talk about a good problem and you can tell a good problem because the people who solve it are not the people that you ever would have thought about solving it. Um uh my first program uh had social scientists, sure, but evolutionary biologists, applied mathematicians, physicists uh brought to the table, which means, you know, you're going to get a different perspective in some sense. Is there any any place where advanced market commitments doesn't make sense?

Yes. Um well, it doesn't make sense if there's no market failure, right? So, you don't need an AMC for the next version version of the iPhone. Uh you know, you don't need an AMC for the next and you know, Nvidia GP GPU. So, it has to be an area where there's something is important, but there's some reason that the private sector will underinvest. So, that So, that's one thing. The other thing is it doesn't work well if you cannot come up with a performance-based description of what success looks like.

And so, um if if beauty is in the eye of the beholder, uh then it's it's not an area where it makes a lot of sense. But it But if you if you know what you want like I want a self-driving car, or I want a rocket that can go to the International Space Station, or we have a proxy for it. So, like warts and all, we use the FDA when we say we want a vaccine which is safe and effective. Then I think it it works a lot better. Yeah, interesting. Um I do want to touch once more on on the sort of data question. Um and in particular, what what sort of data do you would you if you could wave your wand, would you collect uh to really kick start So,

um if you look at the areas where we've made progress in AI and machine learning, um it is often the case that we have a well-defined problem that is leveraging the ability of AI and machine learning to serve as a universal functional approximator. We have a large, high-quality, diverse data set, and we have a benchmark. Um and so, that was ImageNet for image recognition, the protein data bank and CASP for protein structure prediction, and common crawl for a large language models. There are the the protein data bank is the exception rather than the rule, right? So, we sort of lucked into that because in 1971, that community got together said, "We

want to have a well-curated data set. Uh we want to have a set of QA/QC standards for what you have to do before you can even submit it. And we're also going to get the journal editors to agree that you cannot publish your paper unless you deposit your protein structure in the PDB. So, there was Why did Demis Hassabis wake up one day and say, "I'm going to work on protein folding?" It's because we had $20 worth of structural biology in one well-curated data set. We have We have a well-defined problem, which is if you know amino acid sequence, can you predict uh protein structure? And we had CASP, which was this blind competition to know whether or not you were making progress. So, in

lots of other areas, uh we don't have that. So, if I working for a day, what I would be asking the research community is the following thing. One is, what are the important problems that we want to solve uh where we believe that AI machine learning could play a significant role. Number two, is there anything that is preventing us from doing that, right? So, the answer may be no. It may be, you know, you put You You ask the frontier models the question and they give you the right answer, right? But if the answer is yes, there are there are many instances in which um you know, what is limiting our ability to solve that is we don't have the right data set, we don't have the right benchmark,

we need to invest in a platform technology that dramatically lowers the cost of generating the data. Yeah. We need to have the self-driving lab. Um we need to take numerical simulation codes that are still written in Fortran and port them over to a more modern language so that it can support uh lot of differentiation. You know, so In other words, people are really focused on the AI appropriately, but what is not getting enough attention is what are the complements to AI? So, economists say there are things called complements. So, like hot dogs and hot dog buns is always their go-to example. So, you know, in in my in my view there should be more a lot more

attention devoted to what those complements are. And then we have to figure out who's going to pay for them, right? Is a venture-backed startup going to pay for them? Is the Does Do we need philanthropic dollars or or the government to support them? Um because the my view is a lot of people are saying, "Oh, what is AI going to do to us?" Um and the question that I think is a little bit more life-affirming and interesting is what do we want AI to do for us? What is our vision? Yeah, what is our vision? What is what what kind I mean, you know, uh what is the civilizational tech tree that we want to build? Um and for the problems that we think are really urgent,

uh is there anything that is preventing us from using AI and machine learning in combination with other computational and experimental tools from solving that problem? And if there is a bottleneck, then we should figure out how we mobilize the talent and the financial resources needed to pay for it. So, for example, um in addition to being CEO of Rethink Priorities, I'm also chair of Convergent Research, which supports focused research organizations. And this is supporting these public goods that are accelerating progress in different areas of AI, like AI for math.

Uh one of the focused research organizations is called Lean. And uh this allows you to figure out whether a proof is correct or not. And so, if you're doing reinforcement learning with verifiable rewards, having a ground source of truth is incredibly valuable. So, that's why a a of the AI for math startups are all building on top of Lean. But again, someone has to fund that public good for all the startups to build on in the same way that all these companies build on TCP/IP, but like none of them would have developed it and and put it in the public domain. So the how how are we doing for time? Sure sure, we okay?

4 minutes. Okay, sorry. That's what that number. I thought that was a Okay, so not an explosive. We're going to be okay, people. Um Believe me, I would let you know if we had 4 minutes to live, okay? Um Well, at least 3 and 1/2 minutes and I'd leave and then you guys are Okay. Uh So here here's um something I I think a little bit about in terms of data. Uh cuz I totally agree with everything you said. Uh you know, is is after I I helped stand up a another ARPA I'm an arpaholic if you can't tell. So I ARPA DARPA and I helped stand up ARPA-E and they asked me to write an article afterwards and uh they said, "Well, you know, what are you sort of most worried about with new ARPAs?"

And I said, "It's that they will forget that their mission is intelligible failure." Like they're meant to fail to some degree. But there's a difference between failure and like oops, that didn't work and intelligible failure, which is we know why it didn't work. So we are better off, right? We know something we didn't know before. Uh when I think about intelligible failure in the world of AI, where are the null results? Where are the things that didn't work? Where are the all the hiccups and the boo-boos that are just as important for science that are not being published? >> I'll give you a classic example of this. If you talk to chemists and you say, "Why haven't you had an AlphaFold moment?" Yeah, they say, "Our data is behind a paywall, number one. One. Uh number two, the only thing that is published in the

literature and patents is successful high-yield reactions as no one publishes on "Oh, I tried mixing these two chemicals together and nothing happens." So their data is fundamentally unrepresentative. And if they were generating data for the purposes of training a machine learning model, they would do it totally differently from legacy data sets. So they would have more For they would have more data on intermediate reactions. The you So, you might say, "Well, well, why isn't this happening?" What are the important things to know is academics are rewarded for novelty as opposed to utility. Yep. So, they would get a publication if they said, "I created this novel data set."

But if 3 months went by and what they'd done is double the size of the data set, they wouldn't go back they wouldn't be able to go back to the journal editor and say, "Hey, here's my other publication." They just would just reject it. They'd say there's nothing new here. All you just would just generate new data, right? So, there are things that move the entire field forward that the people that we were counting on to solve these problems uh are not incentivized or paid to do. Um so, that's why there's an Octant logo logo here. So, one of the things that ARPA-H is doing is they are using Octant as a contract research organization for open science.

So, Octant is being paid to to uh you know, generate data sets that could potentially transform the way we do drug discovery by being able to predict in advance whether a given small molecule is likely to be toxic or have the other elements of a small molecule that are essential for it to progress successfully in a clinical trial. So, we need to be the the whole premise of meta-science is that we should be applying the experimental method to how we fund, organize, incentivize, and staff research. And this ARPA-H open ADMET Octant collaboration, I think is a is a really interesting example of that.

Yeah. Um speaking of institutions uh that probably need to call evolve a little faster, um modern publication uh institutions. Okay, last question uh with a minute left to go. Um so, uh again, I sort of invent these concepts out of nowhere. One I have is is called LQ uh wording quotient. Let me say again learning because I spent years in England so I say my L's like W's. What's a warning quotient? Learning quotients. We have all these fun experiments and important experiments that are being done. Are we collectively learning from all these things better and if not how can we? Yeah, I think the hardest part is having people be explicit when something doesn't work, right? So

I think the problem is is that something you'll something will just go away, right? And then if you like ask around people are like yeah, it didn't really work. But like no one is motivated to say yeah, we tried this and it it like blew up in the launch pad. So I think I think that's I think that's the issue. I think there are some areas of science where we can create these like at a technical level these really fast feedback loops between computational experimental approaches, but I think the hard part is the sociology of rewarding people who raise their hand and says yes, we tried this and it was a complete disaster.

>> True of science, also true of government, right? That there are all these experiments that are occurring everywhere and are we are we collectively smarter for that? Yeah, I think that all be fine. One thing that the government did was it created these programs where we would provide small grants for new ideas we're trying medium sized grants to validate promising approaches and large grants to scale things up. So I think that is a mechanism that we should be doing more of because I would say that is still the exception rather than the rule. Totally. And if I have to remind you as an anthropologist, we're all in this together at the end of the day, right? So hear ye, hear ye. Thanks Tom for your time.

>> Thank you. Thank you very much. >> Ooh. Ooh.