ZK Podcast: Monsters in the Mempool with Dan Robinson from Paradigm

Transcript

[Music] welcome to xero knowledge a podcast where we talk about the latest in zero knowledge research and the decentralized web the show is hosted by me anna and me frederick in this week's episode we chat with dan robinson research partner at paradigm we talk about his recent blog post entitled ethereum is a dark forest a story about how he and some colleagues try to save a transaction from the monsters that lurk

in the mempool but before we start in i want to say thank you to this week's sponsor leased authority leased authority is a team of security researchers cryptographers open source developers and privacy advocates they are a security consulting company known for their dedication to pushing the limits on how to build privacy respecting solutions mostly known for their security audits like the eth2 specification protocol labs gossip sub protocol atomics library wallet and smart contracts for the tesos foundation block stack's investor wallet centrifuges tin lake 3.0 and more the leased authority team is passionate about advancing the security of projects in the blockchain cryptocurrency and defy space

importantly these independent reviews improve the security of the technology and what's more when the reports are published the broader community can benefit from the shared knowledge this is also why leased authority decided to sponsor this podcast they recognize the potential for zero-knowledge proofs and protocols to improve the scalability of projects and the privacy of users so if you're looking to further improve the security of your protocol and its use of cryptography email leased authority to book a no obligations consultation appointment the email address is contact us at leastauthority.com i've added the link in the show notes so thank you again leased authority now here is our interview with dan robinson

so today we are talking with dan robinson a research partner at paradigm and also someone who co-wrote a recent blog post called ethereum is a dark forest it's been doing the rounds it's about the mempool and we're going to be talking a little bit more about that so welcome to the show dan thanks for having me on yeah welcome to the show great to have you um i guess before we we dig into the memphis stuff which is an interesting story i think that i'm looking forward to let's talk a little bit about what you do what paradigm is so you know what is uh a crypto vc i guess and what is unique about paradigm sure so paradigm is a crypto asset investment firm and here i should mention that um

i'm here speaking and representing my own views and not those of the the firmware of the fund and nothing that i say on this on this will be investment advice if parents are crypto asset investment firm we've act kind of like a hybrid between a a vc and a hedge fund in both making private investments in the cryptocurrency space and buying public tokens um and holding them um tokens like bitcoin and eth we're entirely crypto focused so everything not only is every investment that we make crypto focus but we really sort of everyone on the investment team it tries to be as technical and sort of as deep into it as possible and we think that gives us an edge in terms of uh the kind of investments we can make what makes it the crypto vc versus a traditional vc what does that mean

i know you mentioned okay so you're also partly a hedge fund but just on the crypto vc side of things how is working in this space different yeah so this this is my first job as a vc previously i was a i was a lawyer actually and then i became a crypto engineer at a company called chain which was acquired by stellar so i can't necessarily compare it to what it's like to be a general svc i think it would be pretty different in that a lot of what we do revolves around very sort of technical either due diligence or value-add so the reason i think we're we're so focused on crypto in addition to thinking that it's a really exciting space for investment is just that personally for the for the whole investment team so the firm was founded by matt

wong from sequoia and fredo some who's one of the co-founders of coinbase and we've got a few other researchers and uh investors on the team now um everyone is just sort of personally passionately into crypto and we think that you know that kind of obsession and that and that amount of technical knowledge kind of gives us an edge when investing in these companies and when helping them succeed and that's something where like generals investors they sometimes invest in crypto stuff but they sort of don't know what to make of it sort of a non-sequitur but uh something i've always been curious about and your um your intro triggered it for me who needs to say that they're not giving investment advice because i always feel like it's either people on youtube who aren't

investment advisors and feel like they seem more legit by saying that they're not giving investment advice or it's people who are investment advisors and like want to make sure that it's like this is not investment advice yes in this case it's the latter um so it's a paradigm a registered investment advisor i do think in general it's kind of like when someone launches a smart contract and then says you know smart contracts are all insecure you're going to lose all your money this token won't have any value everyone's like oh that means i really should invest in it um so i worry about the about just sort of the reverse psychology of it yeah georgios who works with you was on the show

also said it recently and we we haven't said it i mean i'll say it now to anyone who's listening nothing we say is investment advice i hope that's obvious but i wonder who can really give get in trouble like if i were to give someone investment advice like on this podcast not that i ever have but if i did like could i actually get in trouble for it yeah that's a good question if it was if it was fraudulent then yes and i think a lot of what people when people do that is they're trying to cover their own ass in case something they say is wrong either legally or i'm not i'm not actually sure about the law around yeah i mean uh there's there's not really a law that says like you you have to have a license to give to give investment advice like legal advice

but it's um but yeah i think they're trying to cover their own asses yeah so i think now we should move on to the topic at hand or one of the topics we wanted to cover in this episode and that is the article that you wrote called ethereum is a dark forest and you wrote it and published it like a week after i think dark force version four was released or like right when it was coming out um and i obviously i i mean good good piggyback there but tell me in your case you're using the term dark forest like it's it's a similar i don't know how to describe it you're describing something vaguely similar but it's quite different from the the video game uh zk videogram which by the way we've had on the show a few

weeks ago if you haven't heard it we'll put the link in the show notes it's great yeah so i think we both i've talked to the to the dark forest guys and we're both just fans of i guess chinese science fiction or particularly this this one series and the whole series is great i recommended the three body problem yeah the whole series is a very rich sort of fertile ground for for metaphors and there's a lot of other ones that i use in day to day conversation that are actually more obscure but the dark forest metaphor um which is from the second book in the series is a really powerful one for thinking about these environments like crypto where it's kind of a war of all against all so a dark forest in the context i was

using in and which is also the inspiration for the game is an environment in which there are such advanced predators out there such monsters out there that if you're visible you'll be killed almost guaranteed and so the only solution to this is to try to hide and then if somebody wants to destroy you all they have to do is reveal your position publicly and so in dark forest in the game they depend on these snark proofs right to and then sort of the the grinding to basically have private information about areas of space and so that's a way to basically maintain this fog of war without which otherwise you just you just sort of have like constant

constant battle totally by the way did you play this version this last one i have not yet played dark fires i'm really looking forward i have some friends who just spent all all their time on it no i'm not i i did it i did play last weekend for the first i mean i had heard about all the other versions and i had not yet jumped in and then i did and then it turned out to be a dangerous game for me because i love things like sieve and i love map games i love resource management games i have been addicted to like these grinding games these games where you have to like you know mine something for a long period of time and then wait the timing and then on top of it built with zkp's and i think i don't know how many people

were playing maybe 400 but like it's the zero knowledge proof community mainly there so it was amazing and except i only played like for right near the end so i didn't you know rank but next time anyway you guys should totally try it frederick have you tried it yet no way i'm not i've not gotten the time but i i'm yeah it's on your list it's one of the things you know obviously i want to play it but it's also one of the things like i want to have the time to play it because i know that once i start playing it i really want to like hack around probably buy a super powerful machine start grinding out hashes and like actually just go gung-ho on it but i i

can't really do that now one of the best things about my job is that i will be able to credibly say that i'm playing this game i felt the same way [Laughter] very cool now going back to what we were just describing you know you're using the term dark force based on this book to describe the mempool and we did do an episode on the mempool which is quite technical we'll put a link to that in the show notes may be worth listening to just to get some context for what we're talking about here and we want to kind of hear this story of what you tried to do to basically like you know save save a transaction or save someone from losing a ton of funds

but before we go into that story let's do a few quick definitions so what is a mempool so the mempool on ethereum or on any blockchain is the set of transactions that have been published and are known by all miners or stakers or whoever's constructing blocks but haven't yet been confirmed into a block so the mempool is a technical detail of how nodes are implemented but it's also a necessary one at least naively because you don't know where the next block is going to come from so you have to basically tell a lot of people in order to make sure that your that your transaction gets included and so one of the the known issues with the mempool is that your transaction is

public and it's possible for somebody to create a transaction that gets in before your transaction with the knowledge of what's in yours that's the dark forest environment that i'm into there's a way for someone to front run it or someone to take what's in that transaction and sort of use it themselves i don't actually understand how that would work like i do know that there's you could maybe kill a transaction maybe by like throwing up the same one with the same nons i don't know if that actually works but like with a higher gas price but like is there like how would you take what's in it so by default ethereum miners order transactions in a block based on the gas price descending order

of gas price and so if you see a transaction that's pending someone has published it it's in it's in the mempool and we use the term the manpool although of course every every node has their own has their own individual one but basically by that i i mean it's just out there in the world waiting for someone to add and everybody sort of knows about it and is waiting to be added to a block once it's out there somebody can submit a transaction with a slightly higher gas price and then if a miner sees it they'll give precedence to that transaction and so for most transactions possibly this won't matter so for example in a bitcoin transaction if i'm sending bitcoin to somebody there's no way for you to get a transaction submitted early that takes advantage of that or that prevents mine from being

executable um and this is true for a normal like ethereum east send as well is it you know that's my you can't you can't actually stop me from doing that when it becomes an issue is when there's more complex what's a term called minor extractable value which is when you have something going on in this transaction where somebody knowing that you're doing this means they can get in and cause you to get a worse price on a trade or possibly cause a transaction to fail and the fact that you've published this basically gives them valuable information and doing this so the classic example is front running on uniswamp so if somebody makes a really large trade on uniswap you can do what's called a sandwich attack on this trade where you put in a transaction before which pushes the price up and then and

then they end up buying at a worse price and then you put a transaction afterward which sells and basically locks in your arbitrage at a higher price because because the user has pushed the price up for a large enough user trade you can make a profit just sandwiching someone's individual trade as a pure arbitrage profit damn and because it's public you can actually see the transaction you can see what they're trying to do you can look at the price right now the way to do this would be to do something to push the price up and make the gas more expensive so that it happens earlier on your transaction you set you submit the transaction with a higher gas price so you choose the gas price of your own of your own tx and so yeah so you just basically submit a transaction that has a gas price that ensures that

that transaction's included before the the the one you're trying to attack and then you you include one that's after are there two like is that happening that must be happening all over the place and i can't imagine that it's like someone individually going and looking at this thing but are there bots that just do this this happens all the time and i think on unit swap which is one of the most popular applications on ethereum or encryptor generally there's a lot of real sort of really specialized ecosystem around arbitraging unit swap specifically and this isn't just doing like the sandwich taxes also getting the first trade in the block after the price has moved in the real world so you can get an arbitrage on it or rb between unit swap and some other on-chain uh trading trading platform but there are all kinds of bots out there

for extracting all kinds of mv anytime a protocol and d5 does basically does almost anything quite often there's there's some way for smart people in the mental to take advantage of it and this isn't always a bad thing like with uniswap the the arbitrager who trades as soon as the real world price moves they're actually providing very valuable service they're an essential part of uni swap working because they're the reason that uniswap always has an accurate price or a reasonably accurate price so in some ways good mechanisms are designed to take advantage of the fact that there will be parties out there who basically pick up these these free pennies um on the sidewalk but then there are other ways in which in which it can be a bad thing so these are

there's a lot of you know a lot of really smart people i kind of picture them as you know sort of like russians or um just sort of in a in like a yeah like a basement somewhere just sort of like hacking on this like you don't you know you don't have to be in the ecosystem i probably don't know the best black hat hackers or men for people because um yeah i'm sure they i mean i think they make crazy amounts of money from what i can see on the and what's happening in the mempool wow but there's a more generalized form of this this this is what i find really fascinating on ethereum there's a particular strategy you can do that doesn't it for a particular type of mev it can like always take it on any contract and the the programmer doesn't even have to

have any idea what the contracts are doing like could be a new contract with new code but for a particular kind of mev this bot can detect it and pick it up and this was what this is what led to this uh story that i that i published with georgios a month and a half ago where in which we we called ethereum and dark forest we have an episode a previous episode that explains sort of the technical aspect of memphis um but back then i mean this this was years ago this episode and even back then we were seeing something like a thousand transactions per second just from bots outbidding each other right so that's also an interesting thing is like these bots get into war with each other so like if there is minor extractable value then one bot

will detect it inject their transactions and then another bot will detect it say i'm gonna bid one way higher gas than you and inject their transactions and then the first bot replaces there so like a node is actually turning through thousands and thousands of transactions per second sometimes just between like two bots fighting each other to get the final thing in and then eventually one bot gives up because they say well this is not profitable enough for me anymore that's right so there's this thing called the priority gas auction which is when a bunch of sophisticated parties see some mev or one of them sees it and this triggers other ones to start to go after it and you know i don't think i'd be able to successfully participate in a priority gas auction i think these are

highly specialized predators here figuring out how to how to win these against against other bots absolutely like them and how to minimize their costs if they lose yeah so that's part of what makes the mempool really scary i think is that not only are these are they picking off um individual users but even they aren't safe from other from other predators and there's another interesting angle too is i would assume at least that basically all of them use gas tokens so they're not actually paying the same gas price as you are so they have a completely different playing field as well in how they engage in these transactions that's right so they can they can set a high gas price on their transaction but they don't pay all the gas that the transaction would cost because they're exactly they're redeeming a gas token in the same transaction and by gas token

do you mean like gas token the thing or is it a type of thing there's a few there's a few instances of gas token um the original one was designed by phil diane who's also one of the researchers who coined the term mev and is one of the experts on the at the manpool and front running and all these things but gas token for those who don't know it's it's a trick that takes advantage of how ethereum measures gas costs to be able to basically to get a refund on gas at exactly the time that you want it so you can basically prepay for gas by storing some junk state on the ethereum um blockchain and then later get the gas back in the transaction where you want to use it potentially one with a much higher gas price is that kind is it kind of like an

averaging product like it sort of means that gas prices won't spike but you may sometimes be paying a little more or you always pay less you typically probably wouldn't use a gas token if you could buy one at the at the same price that you'd be paying for the gas so it's sort of like almost an option on gas at a particular price right at any given time it's one way to think about it or imagine pricing it but i think it does probably tend to have a leveling effect on the price of gas although i'm not sure if that's played out in practice and because it's it's mostly used by these by these weird games like possible could have the opposite effect i think it averages the the bots costs for gas but uh for the general public

it's probably not that big of an impact so i think we've we've walked through the the basics here and the the primitives so tell us what the experiment that you did was what was the what was it that you did in the blog post and and uh what did you find so this was a situation that i found myself in which i'm not usually in because i'm i'm mostly not a white hat hacker i don't spend a lot of time trying to break other people's contracts so what happened was i heard about somebody having sent uniswap liquidity tokens into the uniswap contract itself and this is one of the most common mistakes that people will make on ethereum is sending erc20 tokens to

the token contract itself or to some other years to 20 contract or some contract where they can't recover it and so this was basically a fat finger error and usually this is unrecoverable but i realized that after thinking about this for a bit that in this very particular instance the tokens that they'd lost were worth around twelve thousand dollars could be recovered not just by them but by anyone there was a function on the unit swap contract that they could call which would actually withdraw the liquidity for them and give them all the all the liquidity that's worth you know around 12 000 and since anyone could call this i could call it and i could just be a white hat hacker and and pick up this money

and give it to them but i knew because of the nonsense that goes on in the mempool that this actually wouldn't be that easy so i was lucky that i'd heard about this from phil dion a few years ago the same researcher who created gas tokens and quantum mbb he told me a story over beer about being an encounter with with something that he called a generalized front runner so a specialized front runner is the kind of frontrunner that we were talking about before which is written by a programmer to sandwich a trade on uniswap and make a profit and this depends on some human intention around looking at a mechanism figuring out that it's that it's broken in a particular way and constructing a

pattern matching bot that will automatically take some action in response to saying something a generalized front-runner doesn't just look for one particular kind of mev it looks at every transaction in the ethereum mempool and then simulates what would happen if they sent that transaction instead of the user wow and i guess for most of them there for most of them they're just like this is useless it wouldn't do anything exactly but once in a while right so for most transactions this will fail for some transactions they just run this and they see oh my erc20 balance in this token increased and i didn't pay anything and it was worth it was worth the effort it would be for gas and when this happens they just front run your

transaction they just submit a transaction with a slightly higher gas price that get that is generally guaranteed to get in before yours and so this is only an issue for a very particular kind of mev and this is where there's like a function that anybody can call and that's that's not every kind of mev by any means you know it's it's not probably not most kinds of mvp but it was the situation i was in because there was a function on on the uniswap core contract that anyone could call and get this money out do you think though is it aw is it always mistakes like those things that one could front run is it always an error or is it sometimes purposeful i would think it would usually either be a mistake a result of mistake as it was

here yeah or a bargain and a smart contract so like relatively often when there is an exploit one of the problems is that exploiting it yourself could potentially get front-runned by something like this because it's in the nature of an exploit not every exploit but in a lot of exploits that anyone can do them yeah it would probably be a bad idea to design a mechanism on chain that really depends on people not being able to do this because it's it can you know anyone can call this could have called this function at any time and pick up the money so you shouldn't really design it so that like anyone could pick it up because then anyone might but the funny thing is in this case the money had been sitting there on chain for eight hours

before i had this thought that i that i could go pick it up and i went and checked and it was still there so there was this function the documented function on the contract that anyone could just call and get and get the money out no one had but i knew that about these generalized smart contract front runners in the mempool and as a result i knew that if i tried to pick it up which is like transaction sending to it it would get sniped yeah and so the mempool is even scarier than the ethereum smart contract state which is pretty scary already wow so okay so you you were there you saw this problem you you realized there's a solution but then kind of luckily also realized that if you just went for it blindly someone would see it and then front run you

aren't you attempting to save it yeah it's sort of like like you as a human see this right but the bots only like they're not going around analyzing contracts they're not actually smart they're not actually finding these things so no one else was looking at like the the actual contract and then this in depth but you have a bunch of these predators who are just looking for like how are you moving and then based on how you're moving like determining where you're going and then going in the same direction exactly damn i'm so seeing some similarities now to the dark force game anyway that's cool exactly exactly and so it's a really fascinating situation but then it was it was a somewhat scary one to be

in there was also a ticking clock because if anybody removed liquidity from this pool the normal way then either they would get the liquidity they'd get this this money out or the front runners would see that would see the call and they would actually and the front runner would end up getting it out either way it wasn't likely to get back to the person who'd accidentally sent it in so in order to try to recover this money i talked to some smart contract security researchers including samsung scott bigelow some experts on the mempool and on smart contract security and we formulated this plan which involved basically a lot of unnecessary steps needlessly complicated scheming to try to hide from the the bots

the call that we were actually trying to make and so i obviously couldn't call the contract directly with my transaction but i also couldn't just put up a contract and call the contract um that would that would make the call because you know you wouldn't necessarily be able to tell that just by looking at my transaction that i was even interacting with uniswamp but if you ran my transaction and simulated it you can see every internal call that i make wow and what these bots could do is run the transaction do an execution trace of it and uh simulate not just if they sent the transaction what would happen but if they did every call internally what would happen and so that wouldn't be enough obfuscation on its own basically they could extract that i had done this call simulate it and then just do a

transaction that makes a call doesn't even go through my contract and there's a belief that these generalized bots would be doing that they would actually run every call i didn't know exactly whether they would but basically in this in this scenario we had to sort of think like an attacker and say okay if we built a bot like this and we had a lot more resources and we're a lot smarter like these uh people are this is how we would do it and so we figured that wouldn't be enough to prevent to grow this this call for being from being front run our step actually involved deploying two smart contracts and splitting the transaction that activated them into two transactions which we tried to submit into the same block and the idea was hopefully a generalized front runner would

simulate these transactions separately if they simulated them in in sequence then they would see that the second one actually did the rescue call but if they simulated them separately then the transaction the second transaction would fail and they wouldn't act without calling the unit the call to uniswap so that was the hope is that the bots weren't smart enough to actually detect us and was the the individual like the first entity calling it guess your address or something like did you also split that or did you just split the contracts yeah we called from two different and uh externally owned accounts got it and uh made these calls to two separate contracts one of which would like activate the contract that would actually do the rescue and the other which we would call a contract that would then call that

contract which would then do the rescue and we hoped you know these transactions don't look at all related cool um different gas prices and everything but uh the different senders called different contracts but yes if you ran them both then you then you'd see this so unfortunately we didn't manage to get both transactions in in the same block the strategy was not itself fully tested um we didn't see whether it would have worked if we got in both transactions into the same block the second one ended up in a either the next block or maybe a couple blocks after that i can understand the architecture of the action that you're trying to perform but like why in the same block do they have to be sequential you don't want anything in between or well so what actually happened was we got the transaction the first

transaction in one block right okay but as a result the second transaction when run against the current blockchain state when said when somebody when some generalized front runner was trying to look at what it would do the second transaction would make that internal call to uniswap yeah when they when they ran it against the current state and so the reason we wanted it in the same block was on the hope that the generalized front runner wasn't running all the transactions in that block which would be which would be somewhat more computationally intensive than running all transactions of the mempool in sequence or constructing candidate blocks and running them and seeing what happened in each transaction in them we were hoping they were running the transactions individually so this didn't work we actually got

front run um which part got front run though the second transaction the second transaction was basically it was a naked call it was it was routed through two contracts so it was like a nested internal call but nevertheless when you ran the transaction against the current blockchain state it resulted in the unit swap burn call being made the call that could that could be used to pick up the free money and so we were basically sitting ducks with that transaction in the in the mempool by itself and so the transaction did get did your front run some crazy bot came in and burned a bunch of gas tokens and and picked up the picked up the money damn did your first one go through correctly and then it was the second one that they front runned and then they because the second one was the one where you actually got that

okay that's right so you know this was a strange experience because while we were we were spending like hours trying to you know write these contracts and and write the script that would that would submit them and test it and this whole time you know we it felt a little like being paranoid or like we were like why are we does it actually make sense for us to be to be doing all this and we knew intellectually this is a huge risk that this that this would happen but it wasn't until it actually happened that you could really believe it you know like in the in the beginning of a quiet place where the the kid gets eaten by the monster and like in like the the flashback or the sort of the pre-history so i think the reason they do that in the movie is otherwise everyone's gonna look

really stupid being super paranoid about saying anything i don't know i don't know the reference but okay this is a movie where where aliens are on earth that that detects sound and so you have to be really quiet and you can't talk so there's like very little dialogue in the whole movie yeah but like that it's it's a situation like this where basically being detected means death but until that happens you can't see the monsters around you just sort of know they're there wow and i guess you were you not sure exactly like were you exactly sure on how the bots work is there a deep understanding about how these work or are is it sort of a mystery monster like uh a big unknown when this post was published i think it surprised a lot of

people and it surprised me when i first heard about these bots um before because this is a case where this and like white hat hacks are sort of the cases in which this particular kind of mev comes up and where this kind of generalized front runner happens and most people don't encounter that on a daily basis there's there was some lore in the white hat hacker community around what these bots could do it's not all public like not even everything that's known about how they work is public in part because that would leak alpha basically where if white hats actually do have to get one of these transactions included um the tricks that would work in this case because our trick didn't work we we felt comfortable publishing um publishing this and thought that it was

worth scaring everybody but um we've we've considered and we actually have we have a research budget that hasn't yet been used to kind of map out this dark forest and figure out what these bots can actually pick up on and what they can't it'll be interesting to see because like it's sort of the same thing as in the the hedge fund world or like the the general trading world where there are a bunch of open source bots that can do automated trading in in various ways but they get published because they don't work anymore because they're not not profitable anymore and then but you know that whatever a real hedge fund is using is this at its basis but with some variation that now makes it profitable again

and so you kind of know what this scene looks like but you don't know exactly you don't know what exact models they're using because then they wouldn't be profitable right and to be honest i think this obfuscation tactic is doomed there's there's just going to be no way that that just even like with a sequence of different transactions and hash precommits and um all kinds of like and ways to get it to get in the way of these being detected it's ultimately it's unsolvable and like the the bots are going to win this arms race and the reason you can say this is because for our rescue to be successful a transaction had to be included in an ethereum block that made this call at some point internally to unit swap and that is the necessary

condition for us to for us to win right and as a result because we were submitting this to the to these miners memphis some miner had to be able to construct a block that did this and if a miner can do it then the attacker can do it yeah and so if if these you know i'm not sure whether they have enough computing power right now to really be like to be to be constructing blocks fast enough to always to always get this but that's sort of inevitable but yeah i think so like it they certainly pretty close so i mean going back to what you were running your your whole thesis was that if we have these two separate transactions and they both get included in the same block the theory is that they're not constructing and executing a whole block because then again they would see

that call being made but just running one transaction against the current state and you know if they did this for each of them individually they'd run the first one see oh this only activates a contract doesn't give give me any money the second one oh it just calls this contract and nothing happens from that so it doesn't give me any money but constructing and executing a whole block takes on the order of like 100 to 300 milliseconds and the block time is like 15 seconds so you have a lot of time to construct and execute full blocks and this is with like a a mining client and there are many clients and and like evm evms that aren't in production but are way way faster so if

you were like on a hyper experimental like i just want to simulate you could do it way faster than that probably down towards like 10 to 50 milliseconds that gives you a lot of simulated blocks per like block round chance that's that's right and you can imagine this running as like a cluster of huge high memory nodes on on aws like there's there's really sort of no limit on on how parallel for example this could be right yeah and so so that's that's why i say like i don't think i don't think there's a way actually to obfuscate your transaction when ultimately at some point an internal call is to be made that is that you're trying to hide there's no way to hide it because ultimately yeah they can just do whatever a miner does and do it maybe like 10 times in parallel with different combinations of

of transactions of the mempool who knows and i think that's that ultimately is why that's a dead end the way to do it oh right now the cutting edge on this is submitting the transaction directly to a minor so it gets included in a block without ever touching the mempool or being published to anyone else yeah i was actually i was gonna ask like if the miners run this themselves then that would skip this problem i guess i don't know if that's actually how it works but like yeah so okay so yeah so so first i'll talk about i think um whitehead miners and like miners being able to be to be really helpful in actually being able to do this um and this is something that actually happened uh subsequent to subsequent to our pose and then i'll talk about the the future and why that why it might be worrying if

miners start to essentially try to extract this value themselves so the best way to ensure that your transaction doesn't get sniped by any of these bots is just not to show it to them and to have it be included in a block um without ever being published to anyone with only being in the memphial of this of like a single mining node so if you happen to be an ethereum miner or you're like close friends with one you can basically get them to include this manually in there in their blocks without propagating it when we were in this situation like we didn't know any miners we so we were we had to had to make do in the public mempool a couple weeks later sam sun the person who we

who paradigm just hired the contract security researcher actually found a bug in a defy contract that put him in a similar situation except with 10 million dollars on the line instead of 10 000. whoa and the way he got out of it was pulling an all-nighter with some other people in the security community and minors to be able to actually get this included in a block without ever being published and so this this is a great story it's a it's a follow-up sort of kind of sequel to my to my dark forest post this post called escaping the dark forest and that's actually what they did they got a transaction included that skipped the memphis get therefore skipped all these monsters is that actually published i don't think i saw it yeah yeah yeah so we'll definitely include that then cool

this is in general a very controversial topic because it's it's been often discussed of should miners have like their own private network among amongst each other because that would lower uncle rates and we could increase blocks and you know a bunch of other things but then you're obviously centralizing between these just these miners yeah exactly like you're forming this this strange thing but in in an interesting way um you can sort of have the best of both worlds if you go proof of stake because then the validator set is known it will change by the protocol but the set is at least known so you could submit something to just this set that's true although although then you

have to so in that case you have to trust that the stakers that you're submitting it to aren't going to take the mav themselves and this is true of my news too if you give this to the memphis in this case it was spark pool and and they they were honest and published a transaction uh in the block without without sending it to the man pool if they're honest then you're then you're fine this is a great situation if they just chose to to extract it themselves there's nothing you could do about it and so you do have to reveal this to somebody now i think it's a better situation than having to reveal it to everybody and generally i think miners so far have tended to act in a kind of rationally altruistic way with respect to to not kind of extracting any of them

themselves in the future i think that's likely to change in part because if you're a miner extracting a movie you can get so much value potentially from that mev that that can offset your your computation costs and basically and make you a more competitive miner and so miners that don't try to do this in any way will get competed out by those that do oh no but wait when you say a minor like it can't be just one minor can actually do this better right because you need a lot of minors to accept this too it's like so i don't actually understand like do you have to have a pool how does it work that you'd actually get these miners to do it well it's always a it's a it's still a valid block so like by the protocol the other miners

would accept it sure they could run their own analysis and reject it but that's then you're breaking the protocol right right so exactly so it's one mining pool that we that that they submitted this to spark pool yeah and you know they might i don't know what percent maybe 20 of ethereum blocks so that ensured that this transaction will get included at least like relatively soon and then once it's included in a block yeah the other miners build on it um same way they normally do but but this is a this is a risk and potentially one that could destabilize the consensus process itself because if one of those other miners was really malicious what they'd do is they'd roll back they'd fork out your block they'd look at your block see the mev and then

fork back build on top of its parent block and include the include the transaction and extract the ebv from it and then and then continue and this means that potentially you'll have this war not happening not between front runners but between miners whoa where they're all trying to fork each other out and all trying to destroy each other and that that the war between front runners is mostly only shows up for the end user as being like just really high gas costs um and sometimes sometimes transactions do weird things this would show up as quite often having having rollbacks and conductors so that would be quite troublesome and we're not in that situation yet but it's it's possible that maybe gets gets too um big without without other ways of dressing it is there any way to prevent this i guess somebody's got to be doing some research

in that direction a lot of people are if you are working on this um please reach out to paradigm you can reach out to me at dan at paradigm.xyz um or find me at dan robinson on twitter because we love make we love talking to people who are interested in this space and we're interested in making investments um in this a few of our portfolio companies are working on ways that could that could potentially um that could mitigate this i think there's a lot of i've heard a couple of different proposals i mean there are zk versions of these proposals to avoid front running but there's also like some some companies have reached out to us to use like the parity secret store which is like the miners could deploy a distributed key generation

mechanism so that you can actually encrypt the state of transactions but miners can still decrypt it run it make sure that it's valid include it all this stuff so you're it's kind of like this idea of just sending straight to the miners but you're sending it to the public just encrypted and the miners can decrypt it so of interest to your listeners there's a lot of ways that potentially cryptography could be used to ensure to ensure that transactions can't be read by the miners that you they won't know what's going on in them until after they've been included in a block in addition to encryption another way you could do it is vdfs so verifiable delay functions if for example every transaction itself had to have a vdf on it that was like 15 seconds that would mean that

nobody would be able to create a transaction in response to one that's already in the pool if every transit every transaction every transaction had a time lock encryption on it that prevented it from being seen for 15 seconds then it would be included in the block potentially not decrypt until after you don't know what the state of the block is until after the block has already been constructed but these are generally ways of of separating either separating um transaction inclusion order from transaction execution or blinding whoever is executing or sequencing the transactions from what they're actually doing hmm yeah they are they all have various trade-offs in terms of like centralization complexity you know ux if you have a vdf on everything and like oh yeah it starts getting really complicated

uh yeah it's i think it's an interesting space i haven't dug super deep into it but there's a lot of interesting stuff that you can do there is this an ethereum problem is this a only for ethereum problem is this a problem that every blockchain is gonna have or has bitcoin has mostly avoided this problem because there isn't all that much mev on bitcoin so if you like we talked about before if you submit a transaction uh just spending your bitcoin no one else can submit that transaction nobody else can can basically do anything to interfere with it as we start to see more things built on bitcoin like lightning for example a payment channel provides a way to get mbv it's it's a it requires quite a lot of work from the

miners it's much more than just front-running transaction it requires censoring for potentially a long period of time but that's one way in which miners could potentially extract value by manipulating transactions in the pool in a way that could ultimately both mess with lightning and mess with with bitcoin consensus generally another way is just the transaction fee itself is a kind of mev including a transaction with a very high fee is is obviously minor extractable value right and actually there's a paper called on the instability of of bitcoin in the absence of the block reward which talks about how if transaction fees are the main way that bitcoin miners are are paid then it becomes much more profitable to do weird strategies where you roll back

other people's blocks in order to steal the high transaction fee transactions that are in them okay and that this could potentially and then then you get into games where leaving you might want to leave transactions in the pool so you don't get rolled back and ultimately there's there's not really an equilibrium to this and so it results in this instability of the consensus system itself so i do think as miners get more sophisticated and competitive even on on simple chains like bitcoins this will be a problem and it certainly would be a problem on any smart contract chain that uses roughly ethereum's model of you know um the contracts are public the transactions are public and pool well this also i mean i guess this will carry forward onto any sort of eth2 or sharded blockchain as well would it yeah and

and i think uh the the only thing that that we've talked about on this show that i think starts to address this is like sexy where contracts are private the calls to those contracts are private and like everything down that line but then yeah completely different model of programming and interacting and everything else it is also in theory the miner could just say to users you have to unblind your transaction to us if the user's capable of unblinding a transaction so this is kind of a hard problem maybe you could do something with denial um encryption i don't know but if a user's capable of unblinding the transaction of showing the the miner then the miner potentially extort that out of them this is just a really hard problem there's it's just like game theory

in what's in some ways an iterated game but in some ways as a free-for-all and this competitive dynamic what scares me most is the is the issue that it really doesn't matter how honest and proof of work like miners are because ultimately the least honest ones the most extractive ones potentially will win well um and we'll we'll win not just like like when these races but also just like just like out compete the honest ones and so that's a that's a very worrying situation this is all under the general metaphor of malik which is a popular metaphor and an ethereum of basically coordination failures and and competition leading to what ultimately is kind of like a really scary uh environment like the ethereum is now yeah yeah because like even in uh if you paint an extreme

scenario where if i'm submitting a transaction to a dex then i also have to be running a bot to make sure that that gets in at the price that i want and etc even in that world the miner still has the advantage and can just like he doesn't have to pay a fee because he would just be paying himself would this also affect pos systems because like their miners are taken out and i know you just mentioned you gave this example like validators but like has there been has that been considered in a pos context so pos is almost the opposite problem where in proof of work because anyone can show up and compete that's one of the best properties of it

it's also one of the scariest because it means ultimately that you know some malicious party might actually end up being the best able to pay for electricity with their mev profits right and be able to essentially take over the chain proof of stake is sort of the opposite it's much more entrenched in that the large owners of steak and and those that they've delegated to ultimately are more they're they're harder to display this you know you can't just have some random party come in um with with a better strategy and beat them because they don't have all they don't have all the stake and it's hard for someone new to come in and buy up all that stake from them they'd have to sell it but as a result this does mean that if ownership of your of your coins does get captured that potentially the whole chain is

basically locked down and ultimately if that if that turns out to be a corrupt oligarchy that's running the the chain then trust is lost completely i guess yeah yeah and decentralization is lost and potentially and censorship resistance and so i do think in the in the medium run proof of stake will likely work a little better than proof of work um and that's because it doesn't have the competition just isn't as fierce but ultimately i don't think it's an ideal situation to have basically trust that these that these large whales have the chain's best interest at heart yeah no i think i mean they they still have to be considered malicious right i mean that's how you always have to build a protocol you

have to consider all validators malicious or yeah like at a base level and then you hope that two-thirds of them aren't but it's still interesting because like i don't know in the minor world the miners are known like if spark pool gets a bad reputation because they're doing this band yeah you'll have the same kind of effect there like the the people who use that pool is going to move away and so you've same thing with validators who are getting a lot of nomination like if they behave badly maybe people nominate someone else although we aren't really seeing that we're not seeing nominations being that sophisticated we don't see nominations that have the chain's best interests in mind and so i wonder if we get to that point or like

if it's currently all nominees are just basically people who seek higher interest rate than their bank will give them yeah so i think that that's one of the big structural risks for proof of stake long term it's it's also a risk for work and i mean right now proof of work partly works because the mining pools tend to be um yeah long-term rational where basically they they know you know and this is true on bitcoin ethereum they know that doing things like like double spending attacks and rolling back the chain or or really aggressive extraction of mev could basically break the chain itself and mean that like their all their mining equipment is worthless right and this is similar for steak where yeah there's probably some abuse

you can get you can get and do at the edges but if the consensus set gets too dominated by someone and they start extracting uh too much from it then it kills the chain and ultimately their coins are worthless and so like arguably this this has maybe happened to eos where i think the the perception that block production is is centralized or or uh cartelized um and earned by a small group has meant that the chain itself has been less competitive yeah so it sounds like this is a a space that you know there's no conclusions yet there's still a lot of research to be done and we have to keep our eye on it it's been kind of fascinating to even visit this part of the the blockchain world i had not spent

much time thinking about it until your blog post and and even more so now on this in this interview to sort of switch gears for this last point you're in this role where you do research for this vc firm you're you're spending a lot of time kind of like communicating this outward and i wonder like what your feelings are about that like do you find it is that do you sort of see that like this blog post that you wrote like creating the story out of it do you find this like something that's important for you to be doing or your firm absolutely and it's really what me excited about joining a firm in the first place is well first there's this intake step where basically i get to a very sort of broad overview because you know we people coming and pitching

us i'm looking i'm out there always looking for for potential new investments or developments that are that are meaningful to our existing investments and so i guess i feel like i get this kind of perspective where i get to see a lot of ideas and then process and pick out some of them and then try to communicate them to a broader audience basically in order to make it easier for for more projects to to form that are working on these things so this is a lot of what i've done in d5 is sort of publishing out papers with like defy ideas and this happened i published a paper called the yield protocol and then ultimately an entrepreneur came they wanted to build it and we and we're actually incubating them now and so this has this is meaningful effects on for the fund but with the

mev post also we were basically trying to seed startups out there to be like look this is a serious issue it's potentially going to threaten um crypto we think there could be both promising investments and potentially really important ones to be made in this area and so it's kind of a call for startups to to do this and so trying to to some extent set the agenda on it or or get these ideas out there we think is really valuable and in these cases much more valuable than like keeping this alpha to ourselves would be because ultimately paradigm really depends on the crypto space working that's the most important um factor for for us being successful and so anything we can do to kind of advance that including including inspiring these new

projects um is worth it even if we don't capture the games even if we don't actually make investments in them do you sort of see though the strength in adding that emotional side of it the storytelling the metaphors like i it takes it a little bit away from the technical but i actually think it's incredibly useful it's not done that much i feel like it's kind of missing absolutely and you know i used to read this there's a fantastic book called the cuckoo zag which is about tracing a hacker in like late 80s you know using on like on the computer's ghost in the wires by kevin mitnick there's like great sort of hacker fiction already sorry packer nonfiction that's actually quite thrilling to read even though it gets into technical stuff that you know you you have no idea about but when you explain it really in terms

of this story you don't really have to yeah in fact it's much more interesting than like the fiction about hackers is um where you know they got a million screens and they're just like typing numbers right down the screens vertically i think and one of the most rewarding things for me of publishing this story was the people who who didn't know anything about crypto at all maybe weren't even interested in crypto even after reading the piece but thought it was a fun kind of cyber thriller cool yeah i think it's one of the few crypto blog posts pieces whatever that i saw on hacker news actually end up there and get somewhat popular were normally hacker news is really crypto averse yeah hacker news has crypto and and most of the comments were like

crypto is stupid but this this is kind of fun um so i thought that i'll take it well on that note maybe we wrap up and say thank you so much for coming on the show dan yeah thank you very much thanks so much for having me and looking forward to your next story of visiting some part of the dark forest that is ethereum and to our listeners thanks for listening thanks for listening