LLM inference verifiable on-chain with SnapVM

Transcript

CTO of Rumblefish. Rumblefish is a small development shop in Poland. We have 30 something developers and I've been in crypto for nine years. Uh and especially in EVM wanted to be here but he is from the other part of the globe and like couldn't travel. Um so what is this project about? What are we trying to

solve? We are trying to solve a problem of prediction markets which currently need accurate data and as things are um you know being done today then today the resolution of a certain event is just you know centralized and these markets are prone to manipulation by whales or even outro fraud and corruption by just people who have admin access uh proving uh you know the opposite of the uh actual resolution and this limits the usability ility of these systems uh you can only trust them until a certain TVL because uh there is always enough money uh that if it's on the line it will be

enough to corrupt individuals who are admins and just like you know rackpool and or do something. So what we are trying to solve is shift where the translation is uh from uh you know admins to well I will talk about where so uh yeah uh so what we want to do is we really want to allow Ethereum to read the news right this would be ideal that just like uh ideally we would introduce a new op code which is like shut GPT and you just put in the prompt and get resolution and this would be making our project you know unnecessary. But obviously this is

not going to happen because um calculating LLM inference is a huge computation. Uh if it were done by all the verifiers um like this would just blow up Ethereum. This is not going to work. So this can be done in a sort of like a L2 fashion where on L1 we have sort of like a registry of events that we're interested in and like some information about the process of resolution but uh the computation itself so the LLM inference has to be done offchain and proven in a cryptographically sound way. Uh so this is a little bit how we envisioned

the process. So there's some kind of registry uh like we have in existing systems like in poly market in numa there's a registry and people just uh produce uh information that they are interested in a certain word event in this example there is interest about u you know football game uh of Barcelona and uh then this event actually happens and there is some reputable source of news published on the web and this is really where we want to shift the trust relation uh so that if there is an article on the web and it's on a HTTPS so there is a possibility to grab

information about this sess session using TLS notary uh I don't know if everyone is familiar with this project this is a project from material foundation and they are like making very good progress uh so uh this is not something that would have to be done by us this is the most important thing like we wouldn't have to be a trusted party uh notoriizing the TLS sessions because there is a widespread um effort to you know make this work and we can you know build on top of giants um so uh what we really want to do is that there is sort of like a network of uh let's say nodes and they observe uh the news specified by the registry of word

events and they uh like when the event actually happens and the news is published they notoriize TLS session and upload it to data availability layer so that other notes in the network can actually you know verify the data and for the layer we chosen Celestia which is project you may know about or not but this is sort of like a another blockchain and you can verify on Ethereum that a certain um blob has been published on on Celestia uh in a smart contract And then the idea is that like using this uh TLS session we can actually uh combine the description of word event with a text um extracted from the news

article and we can construct a prompt and then this prompt can be run through LLM and LLM uh if like instructed correctly can give you a yes no answer or can give you a more let's say detailed answer. Maybe you're interested in a score or something more you know particular but this is all uh just LLM engineering and not you know subject of this talk but much easier than uh the the cryptographic uh part of it and uh after this is all done we just want to produce an attestation on chain so that other systems can just use this at a test station to resolve their you know events. This is the big picture of the project. So this is what we want to

solve. Now I will tell about how so how proofs can be done and how they have been done historically. So there is approach of using interactive uh protocols and non-interactive protocols. So example of interacting protocol for example is arbitrum like how they resolve blocks on L2. Uh essentially they have a full history of execution and um you when you want to resolve something you produce uh optimistic resolution and other participants on the network can challenge your resolution and when they challenge your resolution there is always agreement on the initial state and disagreement on the final state and then through interactive protocol you just bisect you make the

range of disagreement smaller smaller smaller smaller smaller until it's only one step big and this one step you can replay on chain. This is this is the typical implementation of interactive protocol and it's cool because it's very flexible and allows for proving complex computations but I mean you may ask how complex like really scale is the limit like you can make many inter uh inter iterations and really every iterations you add you double the size of your computation. So this really can be used to uh prove very complex and long uh computations. But obviously this interaction is um is

complicated from the cryptoeconomic point of view because you have to have some time slot for every interaction. And then if uh you are a party who want to derail the system, you can just stop interacting and uh we have to wait until we decide it's a timeout. So if there's like many interaction steps, it may take weeks to you know resolve and this is just not ideal. If you want to have predictions markets which should react fast, we cannot have like you know a week of waiting. So some aspects of it are very cool but some not so much. And then there are starts and starts are great and really if it were possible to just import lama CPP into risk zero and just run it like we we wouldn't be doing

this project and I wouldn't be doing this presentation but I think it's clear for everyone that this is way too much computation for a start to be able to uh you know sustain like this is probably in terms of like years of centuries for computation like this and and weakness to be calculated. So this is not just not going to work. So what we are doing in this project we are combining these two approaches and sort of finding a middle ground. So we we will using techniques from both uh and we call it a naysayer protocol. So in naysayer protocol we are kind of using a divide and conquer um approach in which we uh from from the interactive protocol we also have a settler produce an optimistic resolution and

what is addition is that we also have him upload to data availability layer a set of intermediate snapshots and I am taking a little bit of liberty here because I only have 20 minutes and I don't want to get into details. In reality, this is heavily compressed. But for sake of this uh presentation, let's just say this is just intermediate snapshots. And what we are able to do is that we can use the non-interactive approach to have a naysayer so a challenger uh prove that the transition from snapshot number n to snapshot number n plus one was invalid. And this invalidates the whole computation. So instead of running entire computation in

a snark we only uh have a challenger run a tiny part of it and uh I will tell how this is accomplished. Uh so first of all what do we really run? Uh so it is all based on llama CPP. This is an open source project published by meta. uh and uh like the llama name is confusing because there is a name of a model which is llama but also llama cpp is just sort of like a utility that you can load a ggml model and it's not only llama model which is um uh you know published in this way also deepseek is published in this way so really any model that is um

published as ggml file we can load and run inference through it. Uh so uh what we do with this, we compile it to risk 5 architecture because it's much easier to work with than Intel or ARM because it has much smaller instruction set. And we get an L file like you all probably use Linux. So you are familiar with L files. And then this is our piece of uh software a snap VM. This is based on unicorn which is a well-known uh CPU emulator. And uh we just you know load it as a elf loader in uh you know Linux KER would load it. Uh and then we run it. Uh and when we run it like the first thing that Llama CPB does it opens up the GGML file

and loads it into memory. And GGML file is kind of big. I mean it can be anything from 1 GB to 10 GB. So we need to run it for a while. And we also needed to implement uh system calls uh so that it can grab uh you know model data. Uh but then you may say okay but if it needs to read model data like it is not going to be deterministic because someone can provide a different model data and yeah of course this wouldn't work. So what we do after loading model data we just pause the uh virtual machine and we take a snapshot of it and pretty much the entry point to this computation is a llama CPP which was sort of pre-run only to the moment where uh entire model was

loaded into memory and just represented in memory as GGML uh objects and uh at this point we are ready to make the inference and this is uh this serves as the layer uh actually I want to do this slide first. So this is as a layer. So we sort of have our memory in initial state where at some region of memory we have a llama CPP with a model loaded and then we overlay on top of this data of word event and the uh artifact of TLS session and uh all in all like obviously this memory is way bigger than this Merkel tree and in reality we are not using Merkel trees but something more

complex but I wanted this to be clear but what uh is possible is that we only to validate the initial state and this needs to happen on chain because there needs to be agreement on the initial state that this is correct. Uh we only need to validate certain sort of sub trees of Merkel tree. So only a handful of paths needs to be uh validated to show that a snapshot to entire memory holds exactly what we needed to hold at the at the beginning of computation. Uh and we do the same for word event and and TLS notary. And then uh going back to this state uh state uh the nice thing about VMs is that when we have agreement

about initial state and we have rules of VM like we know that the initial state determines completely the next step which determines the next step and at infinitum it also um determines the final step and there is nothing indeterministic in this approach. There is no multi- threading, there is no reading time, there is like all the things which typically introduce nondeterminist has been just removed. So this is as deterministic as solidity code running in the VM. Uh so yeah so so pretty much if we have agreement on initial state and disagreement on final state like we can just show where the error was and there is always an error there there cannot be

you know any disambiguity and how do we prove a single step like uh this is this is a virtual machine you are probably all more familiar with uh EVM but this is pretty much a same approach uh there are um the the program. So, so the so the uh emulator has some internal memory which has program counter all the registers of the CPU uh and uh then the program counter points to a region in memory which has the instruction which is being correctly run and every time you run an instruction you just include the in uh increase the program counter.

some instructions can just jump the program counter to some other location. But the the point is that what we need to actually do in a stark proving a single step is very very simple. Like we need to for example here um instruction being done is just that you do a bitewise and operation between register A5 and the value 15 and then you put the result in in a zero. So the point is that if you wanted to run a risk uh zero stark demonstrating that some snapshot has correctly transition to another snapshot, we only need a very small um data points. We need a value of program counter. We need a value at this um

memory address. We need these registers. Five minutes left. Perfect. Um and uh then uh so so we need to prove them and to prove them we are using Merkel paths but we also it's not exactly a Merkel tree it's a Merkel tree uh sort of done in a way that we know which regions of memory are updated more often than the others because registers are updated all the time. So they are very shallow in a tree and we also know that the stack uh is updated all the time. So it's also kind of up the tree. But then and another important optimization is that

the uh program counter always points to just a code part of memory and we already validated this in initial state. So we don't need to validate it over and over and over again. if we introduced another rule in our VM that just program cannot update its own code which you know is is an error if it were to try. So with this small you know corner cutting we were able to really optimize it in terms of how many uh paths we need to you know prove inside the stark. And when you have ability to uh prove a single step like if you want to prove multiple steps you just do the same but

in a loop. Um and uh yeah. Okay. So this was a lot and I didn't make a pause for for questions. Uh so maybe that's a good moment like uh maybe there are some questions about like chart. How long does it take that initial snapshot when you get everything loaded in memory? Well, initial snapshot is kind of fast for for us to uh generate initial snapshot. It took like I think four hours and it depends on how big of a model we choose because there are like many different choices of parameters and quantizations. But it's a little bit irrelevant because we only do this once

and we uh produce uh you know like a like a snapshot and then it just loaded as initial state. But then the entire computation currently it took us 17 hours to complete and this is something that we are working on and it's really about how unicorn implements extended set for vector operations. And I know it's a mouthful, but this is something we need to work on to like bring it back to a sensible uh you know times. When you're loading up real data, uh do you take the previous snapshot and then load more data in or do you have to do the whole process again of initializing model? So let's say you

know you have your snapshot from you know let's say two weeks ago there's been some football games you want to add that through no so to make one computation you take a snapshot and you just overlay it with the data you you want so so there is no transition no like learning no no remembering so it's it's it'solated uh computation every single We have two minutes.

Okay. So, so the summary is that what we are doing like we want to enable close to oneshot uh validation of LLM inference and um uh with this approach we can use uh LLM models which are big enough and smart enough to actually resolve uh real world events and um yeah like this is uh this is also important that what we are building generalizes to any computation like we we sort of are interested in LLM but this is from our point of view this is just extensive computation. So uh what we will do probably can be used for

you know some other uh things uh which are you know comparable in complexity as as inferencing LLM. And I thank you very much for coming to my talk and for your attention and uh I hope you learned something interesting today and it will like inspire you to uh do something interesting on your own. Thank you. [Applause] Great. I think we go with Is it here? Gahn here because that guy is the next one. He will start. Oh yeah. Yeah. Pleasure man.

So you already have your slides. I gave it the slides.