ZK Podcast: Digging into Mining with Kristy-Leigh Minehan

Transcript

[Music] welcome to xero knowledge i'm your host anna rose in this podcast we will be exploring the latest in zero knowledge research and the decentralized web as well as new paradigms that promise to change the way we interact and transact online this week i chat with christy lee minahan previously of core scientific genesis mining and the co-author of the progpow eip she now works on symbol protocol we discuss her road to crypto mining mining on ethereum and how the hardware

industry around this developed progpal or eip 1057 and the subsequent controversy that it generated in the ethereum community the challenges of off-chain governance signaling and more but before we start in i want to let you know about the upcoming zk jobs fair it's happening on july 29th we will be hosting some of the top ck projects who are looking to build out their teams and hire most of the jobs will be technical focused on software engineering and cryptography roles but there are usually some other positions as well so if you're looking to work in blockchain and the zk space this will be a great way to meet the teams and get to know them as you apply we host this on gather.town which is an awesome video game style

interactive online space it's a lot of fun this will be our third edition if you're interested and you want to come check it out i've added the sign up link in the show notes in the meantime you can also check out the zk jobs board which is a list of open positions in the zk community as a last related bit of housekeeping the zk validator is actually hiring right now zk validator is the other project that i spend my time on these days we are currently looking to fill a devops position so if you have the skill set and want to start working with us exploring new pos and l2 blockchain networks do get in touch so i've added the link to the job fair to the job board and to this particular job at the zk validator

in the show notes i also want to thank this week's sponsor ey blockchain ey is one of the founding members of the baseline protocol group which is an industry-wide open source initiative to build privacy tools and scalability technology to make it easy and secure for enterprises to do business with each other on the public ethereum blockchain find out more about it at blockchain.ey.com or visit the baseline protocol site at baseline minus protocol.org so thank you again ewy blockchain now here is my interview with christy lee mynahan [Music] this week i'm here with christy lee mynahan known on twitter as

oh god a girl a long-standing participant in many crypto ecosystems i want to say welcome christy thank you so much for having me anna and i think in today's episode the idea is to explore your background we're going to talk a lot about mining we're going to talk about prague pow maybe like how that started and where it's at now we're also going to learn about what you're working on today so i'm very excited to jump into all of these topics with you before we start in though this is the first time we're meeting so i really want to hear like where did you get your start in the space and what led you to working as a minor oh geez uh that is quite a story you know i say that i officially got started in in 2010 so we all have that moment where

we either brought bitcoin or we heard about bitcoin and decided to mine a little bit we all have that exact moment where we got captivated and for me i had a few chat groups i hang around in and a few friends had told me about bitcoin and had dropped the white paper and i was too young and too uneducated at the time to understand what any of that meant but all my friends were getting super excited so i naturally got super excited as well and it wasn't until 2010 when the first initial release had been dropped that i actually started tinkering and my friend group at the time made a huge game out of it we were all competing for the highest hashrate hashrate to me was almost like a bit of

a video game high score and in the early days when there wasn't a lot to do you started optimizing and hand tuning the memory on your laptop or the cpu and overclocking it to get a higher score was incredibly addictive and there's something i don't know quite how to express it but there's something it's a bit like bragging rights when you get to take a screenshot of your current performance your current hashrate throw it into a chat group and everyone's like oh my god how did you do that it strokes a little bit of your ego but it's also incredibly addictive in the same way that you'll you'll take screen caps of like a fantastic pen to kill on league or your high score in some other video game and just share it on the internet it's bragging

rights and that was my addiction point that was how i got hooked in crypto it wasn't the tech it wasn't the economics i was way too young for any of that it was all around the high score of hashrate were your friends at the time gamers was it like a group more of gamer types yes okay like that not even that um the the friend group we were we were all security enthusiasts is what i would call it there's a lot of overlap between the the cyberpunk communities and the security communities and the video game communities and it's something that you naturally do online to bond with people there's not a lot of ways where you can build relationships with online

communities like really close relationships and spend time together outside of chatting you can sometimes work on projects together but games are just a natural thing that a lot of us default into so you know it was a it was a diverse group and we had started competing until one day my best friend at the time drops a screenshot and he's got double the performance of what any of us have been able to achieve and we're sitting here the rest of us are sitting here trying to think okay how is this possible like this isn't at all possible we've all maxed out our machines what what's going on turns out he's rewritten part of the client for performance and he won't tell us what he's done

he won't tell us how he's achieved it he won't share and he has the hide to say i'll give it to you for two hundred dollars and at that moment i got so outraged so angry just so furious that i was like you know what i'm not gonna pay for this i'm gonna teach myself how to do what he's done just to go and one-up him and so that set me on a three-month rabbit hole of teaching myself everything there was to know about c and becoming incredibly obsessed with c programming everything there is to know about bitwise optimizations and hand optimizing for dedicated processors and how you target x86 architecture and just

staying in a little box and working on this until finally i had a optimized bitcoin client that was two percent better than his and that was all i needed and that was that whole experience was so addictive that it's what i've carried on with me for close to 11 years now as a career wow and then finally in the tail end of 2013 i actually got recruited to do the golden code uh which is like the optimized c code for an asic and that kicked off my entire career in learning about floor planning and asic and hardware development

and that was incredibly uh incredibly addictive so i had that nice little progression path of software cpu gpu fpga until finally i hit the hardware side of things the a6 when you talk about this golden code was this optimizing for existing asics or was there like actual hardware changes being implemented through something like this yeah so your golden code is basically your algorithm optimized in assembly so it's got all of the inefficiencies stripped out of it until it is the most performant um iteration of an algorithm so in the case of bitcoin it's your sha256d algorithm with all of your uh rolls and unrolls baked into there

um and any inefficiencies removed and then for at hash you know your golden code is is separate so on and so forth that becomes the basis for the actual hardware design and from that you go into the architecture and floor planning stage where you now break that down into actually circuits which is really really cool and so golden code it's it's kind of a the hand optimized code you use to build a crypto asic it is what sets you up for either success or failure and in those early days when everyone was neck and neck competing and there were still a lot of secrets to be uncovered golden code was incredibly valuable

so nowadays however everyone basically has the same golden code sha-256d has been studied enough there's enough reports out there there is enough optimizations out there that everyone has the same floor to compete with and now all of the optimizations are actually just a silicon level so now it's at your process level where you're now um tweaking and tuning the five millimeter process and getting your simulations right and then taping out to your multi-project wafer and optimizing hand optimizing and refining there it's more of physical optimization versus a software optimization like it was originally cool what you're talking about here were you still was this still very

freelance kind of like you're contributing but in your own capacity or had you already joined a company professionally no uh i actually did not join a i was entirely freelance and in fact i was entirely anonymous um so i had a male identity i used online i mean i didn't outright claim it was male just people assumed that it was male and so i just ran with it um and it wasn't up until 2015 that i actually joined a company full-time which was genesis mining and i decided to you know just be myself which was a shock to a lot of people who had thought nope this is definitely a dude that that this is definitely a dude there's no way

and along with that came the moniker of oh got a girl because i was like you know what if everyone's so shocked about my gender i might as well just own it and throw it out there so it's not a shock to anyone else did you get a little bit of a kick out of that i did i would of course yeah of course i like being underestimated and then keep showing them or in your case actually like i guess it was more like masked identity and then reveal very cool i think that the us talks a lot about gender discrimination but i feel it's not highlighted as much in other parts of the world it definitely was prevalent in the early 2000s and so i got a huge kick out of that

because um to me it didn't really matter what gender i was how does that how does that have any sway how does that have any influence on the skills i have as an individual and in a way you proved that yeah you removed that from the equation and were able to compete properly therefore you disproved that there was actually a need for this like gender gendered view yeah did not expect to dive into uh gender politics in this discussion but well actually let's bring it back to your story i want to talk so the first job that you had what was it it was genesis mining it was i i originally joined as a cryptographic

engineer which meant i was responsible for optimizing the uh doing a lot of things actually mostly hand optimizing a lot of their mining infrastructure so a lot of their mining software which at the time was predominantly ethereum and z cash and monero and then i think three three weeks into the job my boss rang me and he was like hey you know hardware right and i was like yeah he's like great uh we've got something we want you to look at in china so i was on a plane uh to china to go and see the spender and we've been trying to build

a evolution to our current gpu rigs so genesis mining for those who aren't aware is a cloud mining provider and so cloud mining is essentially where you can purchase a hashrate contract which gives you access to so much hash rate over a period of a year and from that you know you pay up front so you know you pay say 300 and now you get 300 gigahash worth of hash rate for the rest of that year um so it's almost like a extended cloud computing contract it's a pretty uh gray area because there's still a lot of laws being built up around this on whether or not it's a security or not but nevertheless it was an

incredibly big business i would say pre-2017 and it was a major way where people actually got exposed to mining so not everyone had the capabilities nor access to the capital nor even the knowledge of how to buy a asic or a gpu but everyone could go to genesis mining and purchase a hashrate contract and so in order to be the most efficient in that business you have to ensure that number one you have the latest and greatest software that is hand optimized that gives you an advantage over the competition but also ensures that every unit of energy you're expending is giving you an equivalent hash rate or like a optimized hashrate output

so ensuring that you're better than your competition in that regard the second thing however isn't optimizing for density so you want to ensure that you have as many gpu cards as possible in one rig or as many asics as possible in one kind of physical footprint so optimizing for density is pretty important and so gm had been experimenting with a new design and there were plenty of vendors in china um that we're also looking into this problem and that kickstarted my two two and a half year stay in china and it was it was an experience i lived and worked in the middle of fujio jiangxi

and we built close to 850 megawatts worth of power infrastructure and when i finally left china we had close to 370 megawatts of that running gpu uh gpu infrastructure running and that that's an experience like living in china during 2015 to 2017 that is that is quite an experience so those numbers don't i don't know what that means exactly but could you like do would you be able to put that in like node numbers or something that maybe would help me to better understand that size yeah not not quite node numbers so megawatt is it it all depends on what gpus you're using

but 370 megawatts was close to oh this is gonna be a scary number for people um that was close to about eight to nine percent of the ethereum hashrate at the time probably more um and then we we progressively added to it at one point you know we were close to 17 to 20 of the hash rate before the rest of the market kicked in and it was close it was a lot of gpus uh a lot of gpus but that you know that was genesis mining's business they didn't uh necessarily mine for themselves they threw up this infrastructure and then people would purchase hashrate rental contracts

and those then that hardware would be mining for the individual this is sort of a random question but like can you run multiple nodes on a single gpu at the same time is that usually what is happening or is it sort of like per gpu you tend to like focus i know i'm probably focused too much on the node client thing but that's always how i think about this yeah maybe help me understand how that actually connects of course so imagine you have eight gpus in a rig and this is essentially just hash rate you know it's a hashing machine and all it does is it gets work packets from a pool and then sends those work packets to the pool and what we think of as nodes don't really exist in mining per se we do have the concept of mining

nodes which is an individual miner so think of that like that eight gpu box is still an individual that that could just be like one node in a way like you'd have one node running at some point but it's the hash rate of all eight kind of uh because there's not really the concept of nodes so what we think of in ethereum right with with nodes we think of a full full client so a node holds all state of all smart contracts you don't actually have that in mining it's not required to participate all you need to have is mining software which essentially gets work packets and then sends work packets back to the pool and all it's doing is crunching through numbers essentially that's its whole job

and so all of the consolidation into blocks that actually happens at the pool level which is where the nodes are uh ran and maintained up at that level and so in the case of something like cloud mining infrastructure it's not quite that everyone has an individual node that they maintain um it's just that everyone has a certain amount of computing power that's allocated to them and then they get the rewards for that computing power and the pools actually maintain the nodes now we we do maintain nodes as farms in order to make sure that our transactions and our blocks are actually picked up it's more important if you are a large percent of the network

if you are just an individual yourself and running you know a machine at home in the us chances are your transaction is probably going to be picked up quite close but you only need about one node per 1200 rigs whoa so holy yeah yeah you're right you don't count in nodes i get it yeah it's um you only really need one per facility so a good rule of thumb is you have one per physical data center facility and that just you know helps propagate your transactions and communicate with the network that's about it if you're running your own pool that's an entirely different setup and then you do need to maintain quite a few nodes in diverse regions

but for the average miner no all they're focusing on is the hardware itself and this is because nodes don't provide any sort of incentive to running miners are very cost sensitive that means that we try and optimize for the lowest cost possible to squeeze out the most profit and so in a lot of cases running a node it's good will it's just something that you do for the health or the benefit of the network but a miner isn't necessarily incentivized to run a large collection of nodes now that being said many miners and many pools do run lots of nodes for the benefit of the network because they do deeply care about ethereum or about bitcoin but definitely you know it feels like we should have

put some sort of incentive for running and maintaining a node in there and that was the original design ethos right of the first crypto miners your wallet your node and your miner was all coupled into one and we kind of killed all of that off and i think about 2012 when we started having light clients and independent wallets and then independent miners got it that project that you're talking about where you went to china was that an ethereum focused project like that specific mining thing that factory that you just described was that mostly ethereum i know you mentioned it was ethereum c cash and monero generally but yeah what were you working on exactly or is it shared between all these

it's it's shared between all coins it's uh it's whatever clients have purchased in terms of hashrate contracts and it's distributed over a wide variety of coins normally you know cloud mining facilities would predominantly maintain a large percent in ethereum uh so in at hash and then they do a percent in zach and a percent in monero which were the hot coins to mine at the time and then a small percent in uh alternative currencies but it was whatever was you know most profitable at the time where the demand was in uh gm's case yes it was almost purely ethereum and yeah it was it was a very fun time to be a miner

back then you know in 20 2016 and 2017 we started having all of the memory shortages and the chip shortages and that can be directly attributed to the huge crypto boom that was happening especially in 2017. it was insane that memory was actually being purchased from the grain market by some of the oems in china just in order to have enough memory to fill all of their demand that they had and miners just flocked to purchasing gpus straight from oems in china versus what had predominantly been done which was purchasing gpus in say um on amazon or in-store or in you know a mom-and-pop store so it

was definitely a turning point into that kind of industrialization and part of that was also that amd was starting to get really interested in mining and you can't blame them they see they see these clients that have never showed up on their uh retail radar before suddenly purchasing three thousand four thousand five thousand ten thousand gpus in one stroke and they're like what is going on are you running a gaming cafe and you have to sit down and explain to them the process of crypto mining and no this is this is actually just you know we're mining this magical internet money it's called ethereum here are all the cool things you can do with it well at that time you couldn't do a lot

with ethereum but here are all the cool things in the future you could do with it um it was a whole new market and hardware oems were completely mesmerized by this and when we think mining a lot of people still think bitcoin so they still think asics and the fact that gpus at the time could be utilized for crypto mining it was mesmerizing so it was a very exciting time and there was just a lot of innovation that was also happening around this point spurned by ethereum mining innovation in hardware so for instance your actual server design how could you get the most efficient server with all of the high cost components stripped

out of it and something that would allow for optimal density and so these designs that were pioneered in 2016 and 2017 actually are still utilized today and they're utilized in markets outside of crypto which i find incredibly uh incredibly cool what was the next phase so you did genesis mining and you also did core scientific was this a similar role or similar job or was this a big shift core scientific was a bit of a shift for me personally so genesis mining was random maintained by a bunch of passionate cryptocurrency enthusiasts but of course scientific was the complete opposite you had

traditional corporate pedigree and i met the core scientific team when they were called mineco when i was actually looking to um i was running my own little startup at the time and i was looking for hosting space and i came across these guys uh in my in my search and they were in north carolina they called themselves something different and they were very passionate about the crypto space and bitcoin mining in particular but i could tell that they had just started to get into it and they didn't really know what they were doing quite but i loved their enthusiasm and i fell in love honestly with the characters and the individuals at the company and i met kevin in early 2018 and

we had a discussion and it was right after i'd signed a hosting contract actually with core scientific with my startup at the time that i was offered an opportunity to be a cto and come grow core scientific and so core scientific specialized in blockchain infrastructure so specifically in data centers in building deploying and maintaining data centers selling out hosting capacity for interested participants and uh really hyper focusing on the crypto side of things and then we also had a ai business which was specialized in high performance compute and that's because there's a lot of overlap in the designs not necessarily

in the thermals but definitely in the initial physical design of a data center um between hpc and crypto most importantly that both favor density uh you know we want to cram as much heat and as much output into a tiny form factor as possible so we had these two sides of our business and the entire team at core scientific was traditional fortune 50 executives you know you had kevin turner former ceo of microsoft and ceo of citadel securities we had bill humes former cfo of ingram micron one of the largest largest electronics distributors and components distributors on the

planet among other things we had christy barwick one of the most fantastic treasures i've ever had the pleasure of working with from intellectual ventures you just had this absolute rock star team that had come together and said you know what we believe in this cryptocurrency thing we want to come together and build the infrastructure to support it and it was uh it was a big growth in both my personal career getting to learn from that rockstar team but also um being able to teach them all about you know how it actually works in this space how do you market to cryptocurrency miners how do you talk to them you know the there was still that dynamic between centralized corporations

and decentralized autonomous groups that clashed heavily in 2018 and 2019. it's only in 2021 that we've kind of become okay with the industrialization of of crypto and the enterprises coming in in a big way so it was very interesting similar businesses just different different teams uh and different approaches to building out infrastructure you also just mentioned a startup that sounds like was in between those two roles what was that that was that just something you did for a shorter period of time yeah that was i had a small start-up that actually started in ukraine because i was living in ukraine at the time from china to ukraine oh yes yes

or is there some other stops along the way there were quite a few stops there were quite a few stops um yeah i had been actually recruited to do some cbdc work before cbdc became a buzzword in ukraine so i was camped out in ukraine for a good few months and uh during that time i had wanted to try my hand at actually doing a startup so film to start out with two other good friends at the time one who i met in ukraine one who's actually based here in seattle and went on to try um and build a cloud mining company that was run entirely through ethereum smart contracts so instead of a business that was um you know selling

selling hardware and selling these contracts it was all entirely run on ethereum everything from the gpu and its uh its gpus actually all hardware has something called asic ids which are these unique identifiers that are actually stamped into the silicon but are also read by your machine and are kind of used as a unique identifier and so we tied these to smart contracts so that your asic id actually became almost your unique token and it was uniquely yours as a user that was your gpu you could check it you could modify it it was yours and the thing to understand about miners is a lot of us care deeply about like what memory our cards have

because different memory gives different performance characteristics or different power characteristics and we care about what our gpu card looks like and what fans it has and what are the manufacturers of the individual components on the pcb and so we took all of this we built tools to scrape all of this data and wrap it up into a smart contract so that miners could pick their perfect gpu card and get it at uh cost plus plus a small amount of profit which contrasts how things are actually sold in the crypto space and it was fantastic it was a great business early on and i learned to tone as both you know as both a founder and a ceo and it was really great to build a

successful business on on crypto but what made you walk away from it for me as an individual and it wasn't a decision i made lightly either it took me about close to two-ish months to come to terms with this getting to learn from industry giants about how to properly build a public company is what was needed to serve the next generation of infrastructure for blockchain we'd gone through this evolution right where in up until 2013 it was all scrappy individuals in our homes and then some small businesses and then 2013 to 2016 that was when you kind of got your semi

semi professional businesses starting to industrialize but it was all very hush-hush in secret and this was incredibly true of the cloud mining space we never talked about how much hashrate we actually owned or operated uh we never talked about machines it was kept very much under wraps it was almost dirty in a way then you in 2017 and 2018 you had the big ico boom which was full of scam artists and charlatans and people who claimed they were you know blockchain consultants and blockchain experts and once the crypto winter kind of hit all of that got rustled out which is kind of nice

it is it was very nice and all that was left was now your professionals and the companies that wanted to be completely transparent wanted to be out in the open wanted to you know go out there in the space and um just do good and so it was a hard choice for me but it was the right choice to kind of move forward with that and it taught me a lot of things that i needed to know in order to be successful in my own career but most importantly it was the opportunity to get to be part of a big company that for the first time in history was going to go and take a company public and actually build a public company around blockchain infrastructure and basically be like look where

we're here to be a partner to clients with with no um no politics involved with no bias we're just here to do what is right for cryptocurrency what is right for bitcoin what is right for ethereum and that to me fit perfectly in with my my moral compass and i was like i am i'm on this train you're in yeah cool one of the things that i really wanted to ask you about was a lot like around that time so i feel like we've done a great you know history of where you started to 2018 2019 maybe so i remember on twitter and to be honest i didn't follow this very closely but there was like a lot of conversation around prague pow

and later a lot of conversation around eip1559 both i gotta be honest like never i never fully got inside those and i am curious to hear maybe you can tell me a little bit about like what's the origin story for prague pal let's start there and what were you doing at that time and i know it's probably a very big topic because that's actually where i remember first seeing you on twitter um so progpal it's important to understand the context at the time when the first ethereum asics were leaked there were a lot of discussions on the ethereum forums as well as on the ethereum reddit

about do we change the proof of work algorithm in the white paper one of the defining characteristics at the time for ethereum was that this was a proof of work chain that was optimized around gpus and it stated that asics were a bit of a scourge they were a plague on a network and it went into the design reasons and the design rationale for edhash and so at hash as an algorithm which it's the hashing algorithm for ethereum was designed to to target gpus and it was designed to be efficient on gpus and so there were a lot of proposals coming up about a alternative proof-of-work algorithm

that would either break the asics or make asics impossible and so myself and uh two other two other friends we'll call them deaf and else so we became if deaf else we're sitting around reading these proposals and we were realizing that a lot of people had no idea how hardware actually works and a lot of people don't think about how the hashing algorithm actually translates down into the hardware because the easiest way to explain how you make an asic it's actually that you start with gpu and you strip off all of the components that are unnecessary until you're left with the base silicon that you need and then you go make an ace account of that it's a very explained like m5 uh explanation but it helps to kind of

visualize and so in the case of ethereum there was 40 of the gpu card that was left idle actually in the hash algorithm and everyone was going about it the wrong way and so we decided well let's actually just go and contribute something like let's go put up a proposal of how you should do this right and see if they're interested in it and at the time i was actually between i was between my startup and an offer for core scientific so it was kind of in the middle okay got it and we decided to do that we actually approached vlad zamfir first for a review and he told us because we had no idea what we were doing

it's one thing to go and to do something in the ethereum community like be a miner into home forums it's another to actually go make an eip and the rules around that process aren't quite clear or they weren't clear back then and so we honestly thought the way that you make proposals is you go pitch them to vlad and to vitalik because in you know in our minds there's your ceo and cto right even though that is not at all how ethereum works and vlad was incredibly patient with us and actually guided us through the appropriate path he was like you've got to go create an eip here's how you go do that you've got to make sure it meets all these requirements everyone was incredibly supportive about

and and helped guide us around making sure that our ideas were formulated in the right framework to be classified as an eip and uh help get us to the point where it was actually up for discussion and so we did that it was eip1057 and it was essentially a very small tweaked at hash that optimized the operations to completely saturate a gpu and so when we talk about saturation that means it's trying to utilize as much of the silicon of a gpu as possible in order to prevent someone from coming along and actually making an asic and while you can strip components of a gpu away you're also competing with the

cost basis of a gpu and you're trying to say that you can get those components far cheaper than a company like amd or nvidia can who have global supply chains so you're doing one part kind of protocol design one part hardware design but you're also tapping into the economics of the supply chain around gpus and using all that to your advantage to create a highly optimized proof-of-work algorithm still at hash based the change is actually minor it's five very small changes and the entirety of prog pow it's actually only 37 lines of code maybe 42 now so it's very very small it went largely ignored up until devcon 4 when someone had reached out i forget

who now and said that hey christy there's been a lot of discussion about prog pal and a lot of the community is really excited and it would be great if you would come and present it at devcon and so i'd i'd taken my role of course scientific and i was like well i have to go tell my boss about this so i explained it all to uh to my boss my ceo kt and uh he was really supportive and he's like go on go do it so i went to prague and we did our presentation which you know is now on youtube and everyone watches it and likes to take screen caps and make memes out of it and it was meant to just be a quick primer over the changes that we'd done

at hash and why we had designed them as well as the behaviors and incentives that certain hardware types create so it's really important to understand that when you're designing a consensus algorithm you need to be thinking about how all of the users in your system are going to engage with this consensus algorithm and about the kind of behaviors and incentives that are created and the one thing we don't think of as protocol designers is we don't think about hardware mostly because that is a big gap in most designers understandings and knowledge how does hardware work how do the supply chains work how does that entire space work and so asics have a unfortunate history

um up until probably 2020 of causing centralized forces it's important to understand context again so i'll reiterate that it's about the time period so up until 2020 most large asic purchases were done under the table you went directly to the company and you were able to purchase the entire supply for yourself and you would leave none to the smaller operators or the smaller miners it was a first come first serve kind of thing and it was where money talked so you could pay you know a premium to completely lock out your competition manufacturers didn't really have restrictions or regulations like they do in the traditional space so as a vendor i cannot come in and just

buy out all of intel's supply of cpus no matter how much i pay because they actually have contracts and obligations to other vendors retail stores and other clients in other parts of the world so i can't just use money and throw my money around and centralize that situation whereas in the asic space you easily can in crypto you can um so that was a big issue and then any time that a6 had felt threatened historically operators of asics would 51 attack a chain and so we have a huge history of altcoins suffering 51 attacks the moment that there was even a discussion of the proof-of-work algorithm changing so operators of asics at that time and

what the evidence had shown us were incentivized to number one protect their investments number two incentivized to stay as small and asylum as possible and number three were incentivized to do whatever would make them the most profit which was keep at hash running at the expense of the network and there was a lot of interest and excitement and it wasn't until i think the community started rallying for it to actually be adopted when the next generation of asics had been announced and released which was um e3 started to be distributed at large that the community started to lobby for pro pal and so then we went through that whole process of getting it going through the different stages

getting it accepted and championing it this is some great background into like why it existed and like who who the groups were that championed it but who was against it who was the group how would you define the group that that had issue with it so amusingly at the start for about a year no one was against it and part of that was because a lot of people did not have an understanding of why at hash the hashing algorithm was designed in the way it was so it was hard for people to form unbiased decisions around praguepal and be able to have a logical logical debate about it the second reason was it didn't really

affect dapp developers or anyone except miners so the mining community was the only one who cared about it and the gpu miners cared quite a bit about having this and getting asics off the network and uh so did some of the ethereum core team uh per the white papers you know the white paper's reasoning and per the initial ask for community contributions to get a proof of work change what happened was an asic company at the time called lindsay mining started what i will say is one of the best lobbying campaigns i've ever experienced to try and get this eip shut down and they were quite voracious in their

efforts and that caused a lot of chaos and it did a few things it taught the ethereum community about how we need better tools in order to actually get real sentiment because the tools we had at the time could easily be civil attacked your only tools were reddit posts which it's really easy to spin off a reddit reddit account for a few thousand news articles it's really easy to buy a news article you know uh twitter it's really easy to you know make fake accounts on twitter those aren't good tools for sentiment collection and then you think about well hang on what if we have voting through blocks through the extra data

field and so we actually did a vote for prom pal yes no um with blocks and an overwhelming majority of the community voted yes but then you've got to think about the dap users who don't actually mine how do they vote and how do they contribute so now we've left a part of our ecosystem in our community completely void what about the people that are still holders of ethereum but aren't technical enough to either engage in dapps yet because the ux hasn't been built up or people that are you know still participants of the network and contributing in their own special way but don't hold a lot of other how do you get their sentiment their views and so a few things happened at the time there was the formation of the ethereum

cat herders which helped actually get a lot of this sentiment into some digestible format and there was people that built out other kinds of governance and sentiment tools to help collect all this sentiment and pro power ended up becoming a topic on every single bloody acd call that i think it just burnt everyone out not enough people cared except for the mining community the mining community felt they weren't heard they kept voicing they kept trying to almost lobby for this but a lot of miners aren't able to articulate why asics are bad beyond asics or bad and that's okay not everyone has the

best communication skills and you had at the end of it what ended up happening is all of these groups were pointing fingers at each other saying you're only saying this to protect your profits miners were saying well you're only saying this because you don't care about us we're also participants in the network it was politics awful awful politics i have followed a few of the very very open governance i don't i don't want to say processes because they're sort of like events let's say governance events or attempts to sway governance in one direction or another and it's very interesting because there does seem to be this attitude of like in order to win you have to convince the community but if you try to

convince the community you're kind of like a scammer or shell we don't like to be convinced and so some people will say present very rational arguments and then people can make a decision you do that and then maybe like the redditors can't they're not paying that much attention to it and they can still manage to sway or scare some of the actual decision makers and i think we've seen that over the years i i believe years ago i did an episode actually like looking at governance from various different perspectives in ethereum at the time to try to understand like how could this be better what's needed and i think a lot of times there was this drive for a signaling tool i actually worked on a signaling tool like while

ago a little bit for a short amount of time but that also got squashed [Laughter] like the signaling tool itself became politicized in a way bingo so that was that was a very i mean i was really just helping like on the side so i wasn't in the thick of it but what i could even see was like wow so everyone is saying you need signaling and then when somebody tries to build signaling there's this this feeling like the signaling is going to be corrupt somehow even before it exists and doesn't matter how open you are to feedback and that's where you wonder like what are the forces at play really who benefits from maybe not not having any signaling no change i

don't know exactly you hit the nail on the head there that as champions of an eip at it the onus is on us to educate and to convince people why they need this but with progpow in particular we were just contributing a solution to a problem that there were there obviously was and asks for contributions had already been put out in the space and this was our contribution to be honest we just got frustrated that a lot of people who didn't know what they were doing were proposing solutions um and i think that is the same for any developer you know at some point you just get frustrated that someone is talking and you want to be like you know what

here's how you should probably do it but the onus is on the champions to actually go and well champion their proposal go out and communicate but not everyone wants to do that because it is interpreted incorrectly and so because progpal was associated with mining uh it gave an easy opportunity for the opponents in this case the asic manufacturers to hyper fixate on my background who i was what i was currently working on oh is core scientific somehow incentivized to push or lobby this you ask anyone at coors scientific what prom pal is their eyes will just like glaze over they'll be like that's that thing christy was doing in her spare time

right um there's there's just a lot of skepticism and naturally we need healthy skepticism in our community but at some point it turns to toxic skepticism and yet it culminated in just so much drama that ultimately i was unmotivated to continue the core devs were unmotivated to keep having this damn discussion every week everyone was unmotivated and at that point you just want to walk away but prague pow you know now it's a meme it cannot be killed because to this date it keeps being brought up every now and then and it still makes its way into calls i i stepped down as champion i think in officially in 2020 um okay right

before i got covered actually so i i was hands off said nope not doing this anymore uh greg greg colvin actually stepped up to champion it and to the state it's still there in accepted status and every now and then it still gets brought up as you know as either a meme about governance or as a meme about controversial eips or heaven forbid when we're talking about you know the merge or about phasing out proof of work prog power is still there as a battering tool about what we can give this to miners in order to make uh make the merge easier so it still has not been completely killed off yet and it's

been adopted by a few altcoins um so ravencoin has their modified version which they call core pal because their mascot is a raven so okay and z coin which recently rebranded to firo has adopted it as well there is a small privacy coin called vale which has adopted it so a lot of altcoins are adopting it and one of the design parameters around prague power is the ability to have programmable almost dials which allow you to create your unique flavor of a proof-of-work algorithm optimized to a specific gpu type so all of these

coins can actually benefit from the same security an asic would provide by tuning to a different gpu card which is kind of cool so just to recap though what the difference was because that i never really got so this is the first time i actually heard like i knew it had something to do with a6 something to do with gpus but what you were saying is asics come along and you know optimize in a very narrow way to do something but in the case of ethereum most miners are using gpus but that are not fully used and the option here was either asics are going to come along or potentially you could increase the usefulness of the gpu that is already being used

and your argument was since miners already have these gpus it would like keep that mining community in in a way healthy whereas if you only do asics then those are hard to get and could be almost like front run in the real world with like big entities purchasing a lot of them correct and also a majority of the infrastructure for mining at the time it was just gpus and the white paper specifically called out asics as a plague and the gpus themselves had been purchased knowing that this was going to be a gpu friendly chain so miners felt that there was a social contract that they had entered into and they didn't want that to be broken now the white paper has been modified

now uh it sits on ethereum.org to remove a lot of those new answers but the original white paper at the time was a source of almost truth as well as controversy and you mean the white paper for ethereum client development correct got it and specifically even uh i think there was a part in the yellow paper that talked about at hash and the design rationale of the algorithm so there was almost a social contract between between minors and the community miners were going to provide this hashrate it was going to be on gpus even on the website at the time it specifically called out do not mind with a6 you know a6 can't exist on it if you see an asic it's probably a scam

which was hilarious i remember hearing and i don't know if this is like later maybe it's a different thing but i thought that there was sort of talk of creating some way that like the proof of work would always change slightly so that asics wouldn't work well is is that a separate proposal that came later or what what was that that was inside of praguepow because it has a random math element so um one of the things of prep power it's it uses kiss99 for its randomization algorithm and every x amount of blocks and you can define what it is so in the case of ethereum it was uh every minute the algorithm changes so that an asic that gets created is forced to have some

form of programmability and that makes it a little bit less like an asic right it moves it further on towards the gpu the best way to think about it is the gpu is the asic asic just means it's the optimal hardware for that task and so gpus gpus are asics in and of themselves when you buy the little core individually it's called a gpu asic and all we've done is said okay well if gpus are the asic we're going to go and design an algorithm that targets that as the asic so what happened to the like you sort of mentioned that there were a batch of asics that were released right around this time so so given the state

of ethereum as it was without progpow being included did asics just arrive on the market and then have they been taking over shares share of the of the network like where where are we at yeah so what happened was the e3s for a time were close to 30 of the network and they were silently mining um so no one knew where they were who they belong to what did happen was covid and the supply chain and the hit to our supply chain so it was kind of quite an opportune moment you had at the time three participants in the ecosystem who were in the business of building ethereum asics you had bitmain who had just built their e3 and was

looking to optimize that and specifically the e3 was built because they had an oversupply of memory because remember i mentioned that at the time i was in china we were going around and purchasing memory bitmain was doing the same thing they were one of our competitors so they were also hoarding and stocking up on all of this gddr5 memory and ddr4 memory and ddr3 memory and they had lots of excess surplus and so they used a lot of their old ddr3 ram to actually build out these asics because if you've got spare components what are you going to do just go sell them no turn it into a machine so that was why the e3 started to exist and then research had been continued on

other ethereum asics but then bitmain was having some internal politics entering lindsay lindsay's ceo was the former cto of canaan uh which is another large bitcoin and altcoin asic manufacturer she left to go and build an ethereum asic and they were in the research stage and what they proposed was to do this uh fantastic heavy duty massive kind of dye connected to a lot of small memory chips and it would have outperformed any gpu on the market by a factor of 20 to 30 x but it had incredibly high power consumption which they weren't transparent about and they never made it to market or mass

distribution so there was that issue as well and then you had you know silicon so inno silicon has the s master and a few other asics and they're the only competitors still in the ethereum asic game today alongside bitmain and they uh were in the process of building and researching and innovating because their main business is actually in producing memory fi which is utilized in gpu cards so they already have a lot of the innovation and the ip and the knowledge necessary to compete but all of this discussion about prog power had everyone skittish combined with discussions about well we're going to be transitioning to proof of stake yeah so the reason we never upgraded to

prog pow was ultimately because well proof of stakes right around the corner i kid you not it was said in march it's going to be deployed in july this year july 2020 was hilarious so it was always a case of we don't need it because we're going to get the merge soon basic manufacturers don't want to go and invest and do any research or development and then covet him and all of a sudden the world stopped and everyone stopped doing what they were doing and just sat in their homes and this is why there's such a constraint on supply even to this day so it was a time and place thing a very opportune time

yeah and you know now maybe we'll get proof of stake and the merge this year and eventually those gpus will be able to you know transition off into other coins and asics will just be stripped for parts and the only part good in most of the ethereum a6 is the memory huh what do you think the miners themselves are gonna do like do you think that they'll fork over to something like what do you have any sense for their behavior you think they're gonna sell no miners might threaten to fork but honestly a lot of miners do not have the capabilities to self-organize fork a chain and then

maintain that chain so at the end of the day value follows where the gaps in development are and that will always be the ethereum main chain and so miners might fork initially but it won't fizzle into anything and for it to have value it has to be listed on exchanges yeah and maintained on exchanges and there has to be development that happens so you would need a lot of miners to self-organize into that and the mining pools themselves have said no oh wow and will follow the longest chain what about something like etc though like that's a chain that uses it that has a very similar thing the switch would be easy have people

thought of going there the problem with etc is partly in culture they lobbied hard against prop pal so they've already left a sour taste in miners mouths the second part of that is they're pursuing a proof-of-work change called ketchup which favors asics so miners wouldn't flock to that chain because number one um it has to be profitable but number two it's right back where they started going to an asic favored chain where you know their days are numbered as an investor you don't really want to invest a ton of capital into something that you know only has a two to three month life span that's the best way i can kind of explain mining you want to make a two to three year investment not a

a few months investment so it's really really difficult for miners so a few of them will flock to altcoins um there's ravencoin there's uh firo there's a few others but a majority of the hardware in china especially now in the current conditions is just going to sell or liquidate wow or it will be repurposed into the hpc space which again is going to put pressure on the oems because now they can't continue to sell out their orders at their current price point what's hpc what does that stand for uh high performance computing so anything to do with cloud computing rendering artificial intelligence would you say like what what's this i mean i had a question as you were

telling the story of like these mining communities but what is the mining community like and how are they feeling about this like are they are they sad is there anger is there threats what's going on they're they're diverse so at the end of the day the the mining community is just as colorful and vibrant as the dap community which is it's a bunch of really diverse opinionated people that have come together in love of mining and there's natural skepticism over when the merge will happen and if it ever will happen and that is fair skepticism because we've been told that it will happen since 2018. um so that's pretty normal but there's also

you know the the prague power debate kind of made it clear what position miners are in and eip 1559 has kind of reinforced that so a lot of minors feel they aren't actually heard and that their voices aren't heard and unfortunately that is a reality of a chain that wants to move to proof of stake it is a whole community is being cut out um there's nothing that we can do about that what we can do is encourage them to come join the staking community and run validators instead and sort of transform and a lot of the miners will actually do that and so pools have started to build staking businesses and have started to prepare for that

transition and the larger infrastructure in the larger more professional firms have actually diversified into that but your individual hobbyist or smaller miner definitely there's that bittersweet anger and i think a lot of people you know when when the dust is set settled and when the merges happen a lot of those miners will still be users of the ethereum ecosystem ethereum is more than just mining it is it is development it is uh use it is stores of value it's this vibrant melting pot of communities and ideas and applications now and so the other that they've worked so hard to create has a purpose

and has a use and i think a lot of them will transition over into the nft space and participating in that way transitioning into the next step even a lot of miners are starting to get into d5 through way of mev minor extractable value and that is kind of the onboarding into the decentralized finance space for miners wow totally i didn't think about that but it very much is wow it's like onboarding them into this more financial ways of playing you know in in blockchain oh that's so cool and to be honest flashbots has done something phenomenal in that they've managed to self-organize the mining community

around a drive a common drive and vision for profit and they've shown them a different way of utilizing ethereum outside of just mining and it's great and yet flashbots right now is doing a fantastic job for the ethereum ecosystem and it will continue to exist uh when we go into the merge as well so mev is still a will still be a component of f 2.0 and i have no doubt that a lot of the people that have been onboarded into mev through flashbots will continue to maintain some sort of presence in f 2.0 cool i mean this has been such a journey from your start through these

you know mining and the communities and the companies and then these controversies what i want to hear about now is what are you working on today what are you interested in what's what's your focus these days you said you stepped away from being the prague champion in early 2020 what have you been up to so i took a a bit of a sabbatical very short one from the cryptocurrency ecosystem to go learn how mobile phones were manufactured and actually go launch a mobile phone so that was cool i did that check that off my bucket list um and then i'm back in the in the crypto space and uh doing a lot of advisory and consultancy on the expert witness side so actually uh being an expert witness

for legal cases and also working on something called symbol which is a blockchain protocol that's seeking to kind of disrupt markets so every every cryptocurrency seeks to do to disrupt something right bitcoins disrupting payment infrastructure ethereum's disrupting cloud computing our weave disrupts cloud storage and helium disrupts internet connectivity but symbols all about disrupting markets and it's a learning experience for me because symbol is a proof-of-stake coin they use a modified proof of stake called proof of stake plus which aims to reward uh users of a chain preferentially to hoarders of steak cool it's really cool

it's been a learning experience i've been working on that a little bit in uh in stealth mode and um just mainly getting back into the uh into the ecosystem still definitely a big part of my advisory is still in the mining space but you know i'm very curious about protocol design in and of itself and proof of work just happens to be the largest but i think that tide is going to tip in a few years uh with ethereum moving to proof of stake is there some zk in this symbol project the reason i ask is this is this is a bit how we've crossed paths i know that you've started to like be a little bit more in the zk space so

tell me a little bit about that yeah so in 2019 and 2020 i was actually doing a lot of consultancy around optimizing the matrix multiplication of zero knowledge zk starks on fpgas and i put all of that aside while i went to run off and learn about mobiles and then finally with symbol one of the things we're actually exploring is zero knowledge proofs and zk roll ups you know in order to ensure that our chain can support all of these different markets and so the last three weeks it's been nothing but research on the current state of zk roll ups and zero knowledge proofs and zk starks

and understanding um the various implementations be it z cash or ethereum and all of the innovation that's happening there are you focus more on starks or snarks starks stars oh interestingly on starks okay i am a stark maximalist um you can blame eli for that so here's here's the thing right if you're gonna use zero knowledge proofs it feels like you should go for your trustless environment and not have any of that toxic waste and while i understand that snarks are kind of where all of the uh both innovation as well as development is at the end of the day when you're looking forward starks are the superior technology there

where just most of the development hasn't been focused on that path from my perspective it seems it seems like if we're architecting this from the start we should go the stark path because that is going to be the successor in five to ten years out but it's it's all still in the the research and the architecture phase um and there could be a nice way to eventually transition but yes i am personally i'm a bit of a stark maximalist because trustless trustless environments for the win nice i think the starkware crew might might enjoy hearing that um so i have one last question or or thought i want to explore

with you which is like after all of this journey and now working more in the z case side of things like did you feel burned burned out or annoyed by all of this stuff because like it sounds right goodness the fact the fact that you yourself got kind of held up and like you were attacked so like yeah how did you sort of go through that survive it maybe if you have any advice to people uh goodness yes and the advice should not be don't try right that's not good advice yeah that is not good advice definitely i got burnt out the burnout was twofold one it was that there was such pressure on revealing the identities of deaf and elves

that i wasn't comfortable with that because uh as i mentioned you know having pseudonymity is a important part of my own ethics and morals and allowing people to have their private lives is incredibly important and who cares who you are in crypto i mean we're all here because some dude called satoshi nakamoto decided to drum out his thoughts on a piece of paper like we we are born of anonymous identities um so that was a big part of the burnout and then the politics and the best advice i can probably give to my you know my younger self is just keep focusing on the technology ignore what people say about you ignore what people say about your

character or the company you keep just ignore all of it because at the end of the day as much as we don't like it blockchain is a technocracy and we are all here for the tech and that's what we need to focus on if your opponents can't have a logical technical debate with you then you've already won that argument and you just need to keep that in your back pocket and keep uh telling yourself that eips are political not all of them are but ones that affect you know anything to do with fees or profits or the monetary policy will always be political just like it is in the real world and it can be

incredibly draining but just keep focusing on the technology and keep articulating and standing your your ground um and that's that's all that is is needed and finally have a good group of friends that you can vent to christy i want to say thank you so much for coming on the show and sharing this journey with myself and the listeners very nice to meet you and get to know you a little better absolute pleasure thank you so much for having me cool and i want to say thank you to the podcast producer andre thank you to the podcast editor henrik and to our listeners thanks for listening