AI beyond Claude and GPT: How to use AI and not kill privacy trying

Transcript

Okay. To kick start the day, please give a warm welcome to Paul Lenski CEO at Dappnode speaking on AI beyond cloud and GPT. How to use AI and not kill privacy trying. >> Thank you. Good morning, everybody. Welcome to Berlin. Uh I was telling Armand that um I always talk last at the presentation or the last day. So, it's such a pain in the ass cuz you're the whole present the whole um uh the whole conference you're thinking how's it going to be? Am I going to do it properly? So, I'm going to do it terribly now and then uh I can be relaxed. I can relax or I can beat

myself up for doing it so poorly for the rest of the conference. Right. So AI private. Why do we need private AI? Um in the 1st of January 2025 uh it was New Year's Eve. It was after midnight. Everybody was celebrating. Everybody was happy. And in a trailhead in California, there was a man. He was wearing a cap. He was smoking a cigarette. He had arrived there with his car.

He came with a mission and he did what he needed to do. And 7 days after a massive fire was raging through the Palisades in California. It burned 23,000 acres, destroyed more than 6,000 buildings and killed 12 people. The guy that did it asked ChatGPT to generate these images right before it happened.

And he called 911 to report the fire. And while he was doing it, because we have the time data, while he was on the phone waiting on the phone, he asked ChatGPT, "Am I liable for if I cause a fire with a cigarette? Can I pretend that it just the cigarette just fell?" So, they caught him. He's facing 30, 40 years in prison, and he They caught him because one of the reasons is because he used ChatGPT. They used many other things. They used um cell phone tower data. He was the only person in the area.

Um but OpenAI willingly provided these documents and saying, "Hey, he This is This is That's how we have access to this picture. This is his private history with OpenAI. He generated these pictures." Luckily, the picture quality on ChatGPT has improved a lot. There are only two ways of getting private inference. Um you either run it locally or you get a confidential compute API. You can run it locally in your hardware, then it doesn't leave.

And you can If you run it in somebody else's computer, you need some verifiable assurance that they are not keeping this because if they don't If you cannot verify it, it's basically as if they're using this data, right? Okay, why do we even need private AI? I think the fire example might be a a good example. Um but you're thinking, "Hey, I'm not I'm not a criminal. I'm not using um ChatGPT to ask for how to commit crimes." Um but you might be asking, "But I like to think about why do we need privacy uh in AI in terms

of inputs and in terms of outputs. What you give the AI and what the AI gives you. Basically, some information must remain confidential. The AI is a black hole of data. It will suck all of the data up and if you have private code, uh you have client data, you have health records, um if you send it to a model means that you have lost it. You have no privacy guarantees. Anybody can use it. Um if you regulate it, the rules have already decided for you.

You cannot share with uh OpenAI certain data about your customers. If you're not regulated um well, all the data that you give that you give up, it can be used and it's not yours anymore. And in an in a time where you can just point an AI to a product to a website and say, "Make a clone of this." The only thing that makes you different is the data that you use to make that product product better. And there are people worried that AI is going to take their jobs. Well, you've been telling the AI exactly how to do your job since ChatGPT started. How do you want it not to take your job?

Um there's a one example that we were talking with Sebastian right here yesterday and we were talking about blood results, for example. Um and I was thinking blood results are or like health data, um personally uh touches us all, right? And you can share this with ChatGPT and say, um "Hey, what are these What is this blood panel say? Um am I okay?" And it will give you a really good answer. But, the next year, when you go to renew your private insurance for those who are lucky to have private insurance, you might find your premium has increased and you're like, "Hey, how does How is my premium increased?

8 40%. Oh, because you're in a risk population group. How do you know I'm a risk population group? Well, because your uh cholesterol is that much. How the did you know my cholesterol? You just gave it away. You gave it to ChatGPT. Um so, do you know this movie? This movie is Minority Report. They use uh they predict the future with all the data that they have. Um and we're going to a world where you can say, "Hey, how can I start a forest fire?" And there is some Minority Reports type style uh style type that um they caught this guy.

You can say, "Hey, what is the what does this blood result the this blood test result means?" And there is a world where insurances have access to this data. And there is a if you're in a country that uh is autocratic and you or you want to do something against the government, you can say, "Hey, how can I rebel against this dictator?" And the dictator will detain you before you can even start doing something. How can I organize a demonstration against the government? Um and you can be detained even before the demonstration starts. You're giving this data away.

All right. So, this is in terms of inputs. What do you give to AI? But in terms of outputs, LLMs act as our brain and they give us superpowers. This is all these agents that we spawn, all of these brains, they they do amazing stuff for us. And not using them kind of puts us behind. You're a lot more productive if you use one of these, right? Um but how do you know that they are acting properly? If it is your brain and you don't control this brain, you basically have a brain parasite.

It's doing whatever it wants to do, but it's it doesn't work for you. It works for another company. It works for this alien. It works for a company that does not have your best interests at heart. You pay $20 a month for your ChatGPT subscription. I'm sorry, but you're not making bank for them. They're you they are not working for you. Um There is the open-source models as well. Open-source models, this is a Quen 35 23.6 30 billion, I think. And if you asked what happens in Tiananmen Square, sorry about the typo there.

If you ask what happens, it will not respond respond to you. It'll not say certain things. There's some parts of the brain that you can literally not use. It This does not exist. They're rewriting history. Um Fable Fable from Anthropic used um is used to code, and everybody used it, and it was fantastic. But if you asked it how to do LLM research, frontier LLM research, it would cap the results. It would deteriorate the response on purpose, so you cannot use it to compete against Anthropic because Anthropic is a private company, and they're not going to give you the tools to compete with them.

So Andrej Karpathy has this thing, not your keys, not your crypto. That's what we say in the crypto world. But then not your weights, not your brain. If you don't know what's in that brain, it's not your brain. Um so basically, this is it. Um this is us. This is you looking at the shadows, looking at the response, looking at the inference. And these are the models that will basically give you this this truth. It's like Plato's cave. This is not the truth. You take it as the truth, but it is lobotomized. It can be degraded. The results can be poisoned, or it can just change halfway. They change a flag server-side, and it they just it just changes the

the behavior. They can just cut the access like they did with Fable. So, basically, I like to think about it in this way. What happens to your data? Well, your data in a public API is logged, and if you use a verifiable inference provider, it is encrypted, and you can verify that it's only accessible to you, even though the computation the inference happens somewhere else. Um and if you run it locally, then of course, it never leaves your your machine. These you have the strongest guarantees. Who controls the weights? Well, if the model lives somewhere else and is not open source, they. If the uh if you run it in a confidential TE, you choose. And if you run it local

locally, well, you choose as well. Where does the inference run? Their servers. In a TE, it's still their servers, but at least you have these verifiable guarantees. Or again, if it runs in your hardware, this is it. And here's the trade-off that we need to talk about, right? State-of-the-art models. Everybody wants to use Fable. It's the most intelligent one. It's the it's the best of the best. Um but how bad are open-source models? They're 4 to 6 months behind the state-of-the-art.

And here's why local this is also the reason why local is also not the best option. Cuz you can usually a lot of hardware very very expensive hardware. It's not It's not cost-efficient for you to have this huge model. So, you can run local, but it's going to have to be smaller models. Um let's see how behind those models are. And you can see here the open-source models are in the in pink, and the close-source models are in uh green teal color. So, you can see that it's 4 to 6 months behind on intelligence. It's not that bad. It's

actually very good. And it's extremely good if you consider the next slide, which is the cost. The cost is incredible. Deep Seek V4 Pro with max sinking, it gets you a 0.06 cents per agentic task. Whereas Fable can go up to 3.25. So, you're going to spend a lot of money if you're using Fable. And where And if you use something that is just four months behind for most of the stuff that you can do, it's so much better to do it like this.

Um all right. So, what do you want? You want You have regulated uh you're regulated in your industry. You have client or health data that you really absolutely cannot let go. Run it local. Run it in your own hardware. It does not leave your premises. It does not does not leave your hardware. That's the way to go. You need agentic use. You need uh things where speed matters. So, speed matters a lot and cost efficiency as well. You can still get really really really good assurances with verifiable TE inference if you run it in a confidential API.

But you need to demand that you can test this. There are products out there that say private inference, but it wasn't a trust me, bro model. Like you have to trust them. You have to trust them that the How do I know that you're not keeping these logs? And they cannot provide assurances. You have to make sure that whatever you use, they provide assurances that this data is verifiable verifiably does never leave the TE. And then if you will have something low stakes or you want to like do a quick prompt or a quick question, then public API is fine, of course.

All right. So, we're back here. We're back to running it locally, um owning your own hardware. It never leaves, or you can use confidential confidential compute API. So, as you know, uh many of you already know DApp nodes, and we have been helping people run complex things. We help people run Noesis chain validators, Ethereum validators, um with a couple of clicks. Very very easy, very nice UI. You have a DApp store, you just click here, and you download whatever you want. We're doing the same with local models, local inference. And here, I would like to separate where the inference happens and where the agent happens, right? Because the agent can run locally, but the brain

needs a lot requires a lot more The The brain is the LLM, requires a lot more computation, right? So, you can run both. The piece of hardware that you can see here is a DGX Spark from Nvidia. You can run pretty good models in there, pretty good models. Um and fairly fast. It's not state-of-the-art models, but if you combine a couple of them, you get very close to state-of-the-art. But the agents that decide, you can run them on DApp node, and any DApp node. It doesn't have to be a DGX Spark in your in your uh 3-year-old DApp node that you might have, you can still run those agents.

But then we're missing where's the computation happen if I have a 3-year-old DApp node? The computation needs to happen in an API. Oh, sorry. Um so, in the DApp node, we've added a chat. We've added an MCP. Now, the DApp node knows how to use DApp node. You can just ask DApp node to use DApp node. Download this package. What's happening with the logs of this? And all of these can happen locally if you have a uh hardware that can run models locally, or it will happen by using a private confidential API that will do the LLM will do the thinking.

And that's Nexus. Nexus, I like to think about Nexus like the Tornado Cash of inference. It gives you privacy in a certain way. There's locally, the first box is your dev notes, and you can run your local prompt there, you can run your tools, you have your data in there, and before you send it out, you can run a personal identifiable information anonymizer. You can remove the PII data and then send it out. The second box is our servers.

We can also run a PII anonymizer there, just in case you're running And then we have a router, and all of these runs in a TE, so we don't know what's happening, and you can verify that we don't know what's happening. The only thing that we're sending outside of the TE is the number of tokens, so we know how much to charge. And then we send it to either an inference provider, that's what we call anonymous models, because they know they can read the prompt. We can't, but they can, because they need to process it. Or you can run it through a confidential inference provider.

Why do we offer both? Because the one above is much cheaper than the one below. And for some other things, it's okay. And why do I say it's like the Tornado Cash of inference? Because it doesn't come from your IP. Even if they can read the prompt, it doesn't come from your IP. They don't know they have no link to you, and if you've used a PII anonymizer and your data is removed, they don't even know that. They don't even know any of the data that you put in there. Right? And then if you really want the maximum assurance, you use a confidential inference provider, that's also happens in a TE, and you can verify the whole chain of trust from beginning till the end.

Right, those are an example of the proofs that you can get in Nexus, and I you have to require to see the proof, to see the attestation, to see the verification. It looks something like this. We offer it in Nexus, but many private providers do not pass these test. They cannot provide you with this verification. All right, so what can you do when you have something private? We have a new teammate. And I am okay with giving it all the data of the work. Many of you are testing with this. But how many of you are not willing to give all of the data of your company to a model? I can do it.

There's the TabNote brain. I give it meetings, the Discord history, Grafana data, errors, support, and it even does PRs. It can automatically receive an error from our Grafana observability front end, and it receives the error, tests it, and sets a PR, and a human has not been involved. It will try to fix it. And it does task list for humans, for agents, it does everything everything you want it to do. And I am okay with it because the data is always mine.

There's nobody else that knows the data of my company. Right. So, TabNote, you have plug-and-play hardware. You have open-source software. It's easy to install all these agents to make it work easily. And most importantly, private inference API because sometimes local is not enough. Right on. I like to finish with this um I I actually say this in every talk. Scalability isn't important until it is. Decentralization is not important until it is. And privacy isn't important until it is.

Until you cause a fire. Until your insurance price goes up. Until your company does not have an edge anymore. Thank you. We have 1 minute for questions, I think. >> Thank you, Paul.