Why Evmos HALTED an entire blockchain

Transcript

welcome to the interop this is a podcast about understanding the decentralized economic networks which make up the inner chain and my hope is that my listeners will get a better understanding of the topology of these networks the technologies that make it possible and the opportunities that it provides to investors and developers i'm sebastian i'm a podcaster and investor i'm here today with feliko kunsai kulma who's the co-founder of fmos and the ceo of tharsis thanks for joining me fede hey seven uh thanks uh for having me today really excited to uh be part of the intro and also uh this new venture that you're starting this is a new podcast it's called the

interop uh this episode is gonna go out on epicenter uh which is my my other podcast but also go out on the interop uh so it's a bit of a cross promotion here to get people in the epicenter universe uh to also you know start uh listening to the interop because there's so much happening in the interchange ecosystem right now and i think like there's a lot of interest for it and we've seen that like with all of this new content coming out and what i want to do with this podcast is um bring more technical discussions so the kinds of conversations that we have on epicenter you know the very deep technical discussions but that also go like kind of high level and looking at the ecosystem um you know i i like a 10 000 feet level i want to have those conversations about about the

interchange and the cosmos ecosystem but yeah for for listeners who aren't familiar with you i mean you've been in the tendermint uh and cosmos uh kind of ecosystem for a really long time you were like an early employee at tendermint um tell us a bit about like you know how your background i think also sunny is the person who um kind of got you hired at tendermint or introduced you to the tendermint organization back in the day like give us a bit of a background here like how you came into into this world yeah as you mentioned i i was one of the i would say like first employees uh a tendermint um but more importantly i was also like one of the first interns from

tendermint so i was part of um blockchain in berkeley as part of my semester abroad that i did on user berkeley and uh well being at blockchain berkeley uh sunny had just dropped out during that semester and he started working at cosmo's tendermint and sunny saw my work that i was doing with the blockchain berkeley organization and working with different uh fortune 500 companies developing smart contracts etc and he introduced me to the uh tendermint team that was working super hard back then uh was post ico so they were like working super

hard on developing all the functionalities of cosmos back in the day they started working on the cosmos sdk they had a pegi and they also had uh of course ethernet which is a precursor project of atmos and um they saw my work and um yeah i applied for an internship and started working with them and then once i graduated i moved to a full-time position and relocated to berlin yeah because you were initially working from the san francisco office i think and then you then moved to the berlin office where you're now based what was it like in the early days of working at a tendermint um what was the five there there was super rush there were so many

projects going on there was uh peggy which was a former name of gravity bridge as well ether ethernet uh tendermint core and cosmos sdk and gaia which initially was only part of the cosmos sdk but then like it was moved to another base so that you can have like separation of concerns and the cosmos have could evolve into its own particular code base with different functionalities etc um like the ones we now see with gravity dex etc um yeah so lots of projects uh teams working on

on all these different initiatives um interns working on and everything back in the day um when i joined full-time i was working on the uh boyjager wallet i'm not sure if you remember that uh with jordan and fabo they were the first ones to introduce ui for cosmos with the governance they implement we implemented also the ledger um the ledger ui for to connect directly with the ledger cosmos applications that had just launched and yeah it was uh exciting times back in the days

and when did you start working on putting an evm on on tendermint i think you were part of the first uh kind of team within uh the tenement organization to uh start working on that project so i didn't work exactly on etherement back in the days i was mostly working on either voyager peggy so gravity bridge and i also worked on the cosmos sdk but when so like initially etherman which was his first uh iteration of evm on top of the tendermint consensus

tendermint core uh consensus engine um it was developed first as an tendermint application which is the base layer that the cosmos sdk now implements and then it migrated fully to um like a cosmos application so a cosmos based chain and that took a while and was uh spearheaded by jackson polling who coincidentally was hired um to pm the etherman project and then once uh once more resources were required for the cosmos hub he switched teams and started working on on that project um

we were also working with alex bess um yeah bess was um the core contributor for a long time and also started working on interment and then migrated entirely to the cosmos sdk and the rest of the tendermint core and basically he's been working on the full like the full stock of cosmos applications for a while so yeah i i worked yeah but so back to your original question when did i start working it was uh in 2020 when i was contracting for chainsafe [Music] so after i left tenderman i started freelancing and working for different entities and one of them was

[Music] one of them was a chainsafe and i started working on etherman because they had received a grant from the icf to work on internment so i want to get straight into atmos here and uh you know i i heard you say on a community call i think it was a couple weeks ago that um what you're doing is difficult and you know i i think you know it's it's difficult enough to build uh to build applications on blockchains there's all sorts of things that need to be taken into account that perhaps um are abstracted away in

you know web development or like more mature development environments there's just the fact that it's decentralized you have to take you have all these stakeholders into account um there's the fact that you know like you have to keep the the chain secure with like validators or miners and but you guys are adding like an extra complexity onto this which is uh you know like marrying these two technologies that weren't uh made like weren't made to work together initially i think and yeah i'm coming at this with like uh not not a very technical uh eye but like it seems to me that these technologies are are are not um well suited to go together can you break down like why

building an evm on tenement consensus uh and and then like in integrating ibc in to that technology stack is so complex so the main problem that we see in integrating not only into the call to tender minutes specifically but to the rest of the cosmos ecosystem is the lack of support for ethereum tools ethereum keys so for example one very particular case that has been super hard for us to like go through and we had to take we we had to make like really

tough decisions in terms of like how can we improve user experience i was the the lack of support for ethereum keys on the cosmos ecosystem um because cosmos only supports one type of key that is the ones that it's shared across the cosmos hub etc so like when you create your address it's always in that key format um we see which is sakp256k1 um coincidentally ethereum also implements that uh key but has a different derivation for the private key to the address so the address generated with the same key is different and that results in just imagine you going to kepler wallet and opening your tab and then if you export that private

key and then you import it to say metamask you have a completely different key like a different address so to say that's one case and then the other thing just like misc things like for example the address format on ethereum is uh hex addresses like 0x blah blah blah unknown cosmos you have like bec32 so you need to create compatibility for the user to in order to okay i'm now using a cosmos address i should expect to use the back 32 the bec32 format so like the ones that start with cosmos or or fmos in our case or uh osmo osmosis um

so you have like all these formats that are different that's more like on the ux kind of thing um and then on top of that you have the problem that in the intenderman consensus um it's bfd so fast finality so there's no concept of a pending state so when you have to wait for the transactions to finalize sometimes on ethereum you have like transaction pending on the on on ether scan um like all this functionality that on tendermint is is now there because there's never i don't know there's never been a use case that

needs to support handling pending state because like the state is is committed to the chain yeah it's not probabilistic like in like an ethereum or proof of work blockchains so it goes through the mempool the transaction is verified internally and then broadcasted to the rest of the network and then the the transaction is finalized but on on ethereum you actually need to have a probabilistic finality so like a nakamoto consensus where uh that the there needs to be like confirmations uh for the transactions in order to be included to the to the heaviest um subtree

as well i'm not exactly sure how uh polygon works under the hood but my understanding is that polygon is uh is an evm chain on on tendermint right what's different about polygon and fmos aside from the fact that maybe there's ibc on atmos so the main difference is the architecture polygon was created in a way to entirely support the ethereum ecosystem and to create checkpoints to the ethereum blockchain so you can even though you have fast finality on the tendermint consensus you can you can eventually roll back to the previous uh

to the previous uh blocks if um if needed be so the approach here is different and then i i think they're working on a cosmos sake but they're still implementing checkpointing the staking this taking um approach is different so it's not very modular and i think they're working on like an sdk as well um a polygon sdk so that they can have more uh chains that are evm compatible in their ecosystem but the the modular approach they have is very different from the ones that we can find right nowadays on cosmos

because they don't need to run into this issue of like oh compatibility between cosmos and ethereum and and they took the approach of like creating the or drafting the entire architecture based on ethereum so they don't have to handle any of the cosmos cases okay yeah i understood and when when ethereum starts moving uh when the evm i mean like basically ethereum two is going to be fast finality proof of stake uh are are there aspects of like the e2 evm or whatever that should look like that make it easier for fmos to implement where you don't have to

take into account this this technical debt that has to do with proof of work um so most of the components from proof of work we actually uh are are not relevant to us so for example the on the json rpc then minor namespace or setting the coinbase address for the miner like all that stuff so like the the validator itself is the one that is um like using all these functionality itself and it's all handled through the cosmis component of atmos which is which are the cosmos sdk modules um whereas on ethereum it's all handed through like internal geth architecture

or like in this case uh um yeah the eth2 architecture so like one big thing is like even though ethereum i evolved to ether ether 2.0 um we still need to ensure that compatibility with some of the existing tooling and so there are two approaches here either you reduce compatibility with say metamasks or other evm explorers clients etc tools or by for example just implementing the evm and you define your own say json rpc endpoints and whatever

um which will cause all these clans to break or you start you need to all the time try to catch up with the latest developments or the latest versions uh supported of maine and on ethereum so there's a double effort in in terms of like supporting all the cosmos ecosystem tooling uh supporting all the ethereum ecosystem tooling and trying to catch up with whatever the latest state of ethereum is and implement new functionalities on top of that so our trim our team which is uh fairly small is able to quickly adapt and and catch up with ethereum and implement all this new functionality for the evm

for the atmospm and also implement implementing functionality in terms of like token economics um interoperability etc so we're looking to hire more core engineers and also full stack engineers that can help us like merge and marry these two uh ecosystem into a single user experience that is uh really smooth for the end user yeah so let's let's talk about this like so there's lots of things here i think that need to be unpacked so you know the the user experience the ecosystem of applications the types of features and you know you talked about tokenomics and um evmost had a

novel approach to launching the chain which was the wreck drop it's been covered extensively and i don't want to go like too much in the details here but you know just uh for you know i think most of our listeners will be familiar with the fact that there was an attempt to launch uh fmos a couple of weeks ago um that attempt um well for lack of a better term it failed and we'll get into what happened later in the interview but what is the current state of affairs and um how you know how does this um [Music] affect you know effectively either the the enthusiasm around the network

launching and have you seen any uh evidence that what was like like a highly i think hyped project with a lot of enthusiasm have you seen any evidence that that is going away or is the community you know is is the community like uh confident and and sort of supporting you in your efforts to relaunch the chain the upgrade as you mentioned didn't go as smooth as we wanted to um there were like different bugs especially because of the the lack of support from different like cosmos tools and there were also like some edge cases that we had to handle in

terms of ibc because now besides having like all the functionality on cosmos the functionality on the evm uh we also need to handle all these edge cases for um for ivc and yeah definitely and i think the hype has lowered a little bit i i'm confident that once we launch again um and we restart the chain the hype will continue and but mostly because users will be able to claim their tokens um through the dashboard super easy and um will be able to like stake their

tokens natively and uh there are also a lot of projects already trying to build and deploy navimos and also like a few of them are uh creating earth uh creating um a few air drops so that the community can also like benefit early on and and i think the earlier the users start to stake and on and use their tokens for lping it's etc um the quicker the hype will return i think i guess also been really humbling for us for the team so then we're not here for the hype but we're also like trying to build as much as possible and try to

i don't know improve our processes improve the functionality that we want to bring and ensure that we implement the changes and this is one key difference with other evm ecosystems i believe is that we are not only tackling this from a like core protocol perspective like the infrastructure that supports the smart contracts but we're also trying to push for products that support the protocol so like the dashboard we've been closely working with kepler in order to support ethereum signing we are adding for example metamask support for cosmos transactions so then users can

use um their metamask with cosmos on ethereum so like trying to create this user experience natively and we need to ensure that this is not only supported on the in the protocol side but it's also supported on the product side and i think the the protocol itself was ready i think what was missing was more qa in terms of like the tooling and um the the product itself and uh how are you doing through all this uh how are you holding up through this uh this ordeal yeah it was hard during the the first week i think but um our team is also

very supportive and the community is very supportive um i think a lot of people were like super hyped and also the price of the token and whatnot but um for the ones that know us and we've we've been working on building for so long and we were the only team that was able to finally launch this after more than more than six years already so [Music] yeah we went back and building to build and and be able to support these new functionalities that we wanted to add and and go back to the drawing board and test every functionality that we wanted so

i'm yeah i've been able to hold up with the support of friends family and also the community that is supporting us but i think once we restart the chain it's going to be like the best user experience you will see yeah i'm really uh we'll we'll talk about like the launch plan and everything towards the end here but yeah let's let's dive into uh you know some of the um some of the things that uh evmos uh brings in terms of innovation to the evm like currently or like when the chain launches but then also uh yeah you know into the future and the road map for the chain um yeah can you can you break all that down for

us yeah so we now have a bunch of projects that are looking to deploy to atmos in the first week or two after restarting including ave on other blue chip projects and we have a lot of new projects that are deploying into the fms ecosystem because they saw the potential of like the new token the innovative talking token model that we created so there are multiple branches in terms of like the feature roadmap that we want to implement one of them is of course called protocol like what is the team going to support in the future besides just avm and on on the default interoperability that cosmos allows you

and then the others the other side is like what are we doing in terms of products in order to support these all these use cases um and in terms of their protocol we've been also like working closely with celestia for seven months and this is something that we're really looking forward to implement and eventually deploy once celestia is ready uh which is um a settlement layer for celestia rollups so it's basically a rollup and like supporting evm rollups on top of atmos and that's one thing the other thing is working on interpretability and uh interpretability not only in the sense of transferring assets uh

we are already supporting erc20 tokens to be transferred in and out from the atmos chain but we're also looking forward to enable interchange composability so that chains like somalia or region network who have specific use cases targeting d5 can directly call smart contracts deploy in evmos and make sense of that data by unpacking the events and then realizing okay like here's here's the event data that was logged uh on the smart contract what am i gonna do with uh and then like they handle like specific use cases like for example in some earlier they can uh rebalance lp

positions or work directly with ave so this brings a lot of benefits in terms of reduces latency if you're using gravity bridge or any other bridge accessing any existing bridge currently deployed in the ecosystem because ivc will drastically decrease the latency and then the the transaction fees will be way lower so i expect a lot of interest in this functionality from other teams in order to support communication directly with smart contracts and that's one thing the other in particular is nfd nfd interpretability we're also working

closely with the stargaze team in order to transfer nfts from the from the atmos avm into the cosmo awesome environment that they use and so like making interoperable nfts um is going to be something huge for the rest of the ecosystem and for game fi in particular so that we can support application-specific blockchains or application-specific evms that are directly sharing security with atmos for example and be able to directly use this functionality to transfer nft assets and

yeah like that's that's more or less what we're doing and in the future we're also going to be supporting um cross chain nfd like delegate calls from one chain to another and then yeah once you once we implement that we'll be able to share composability between uh evm chains so that uh a chain that is on a smart contract that is deployed on fmos can communicate directly within smart contracts that is upload for example on chronos or other evm chain that will be deployed in the future so i'm i really want to dive into this in this interoperability aspect because i think there's different components of it that um maybe at least i i don't fully understand so

if you're used to using uh if you're used to using sort of like interchange or ibc assets uh you you can move assets from one chain to the other so if you have assets on uh you have like atoms you can move those atoms over to uh to osmosis for instance and then those atoms live there and then you can move them back to another so from the user perspective it's seamless and uh i think like people kind of understand that once they've used it what's that going to look like forevermost uh oh so i i might the way i'm thinking about it is like okay so you'll be able to move assets outside of evmos but these are not uh ibc assets or like

erc20 tokens or kind of evm tokens and then what's the interaction with other evms so uh what you know ethereum or uh polygon or what how does that uh how does atmos create interoperability also with those chains so the evidence interoperability uh components are yeah they're they're multiple and the main one of course is ibc in the in the cosmos ecosystem so ivc allows us as you mentioned to transfer native assets from the cosmos from osmosis directly into the atmos

but unfortunately the the native functionality for of ivc doesn't allow conversions of those tokens directly to um to the atmosphm so you need to create sort of like a a mapping between like your c20 on cosmos ivc tokens so for example like a cosmos hub atom or an osmosis token that has been that has been transferred over via ibc how do you represent now those tokens within the avm and how then those tokens are bridged over which is the second component i want to talk about how are

those tokens now bridge over uh through their erc20 representation to other chains and that's going to be through nomad and connects and other bridges that we are deploying on atmos so we're going to see an influx of all these erc20 tokens from all these different chains polygon avalanche and ethereum of course moon river and moonbeam to the atmos and cosmos ecosystem and then those tokens those erc20 tokens will be able to be converted to the cosmos ibc representation so that they can be

transferred over to cosmos valve or osmosis so it's going to create a lot of liquidity so we expect atmos to become the main point of entry from all the evm ecosystem for all these assets to come through these ridges to ethnos and then through this functionality that we're allowing these year c20 tokens to be converted those tokens will now be able to be transferred via ivc to the rest of the cosmos ecosystem okay so let's just take an example here let's say i have uh usdc on uh on on ethereum uh or or even polygon

or like any evm chain that supports usdc i'll have to move those through a bridge which will either so that right now you've talked about nomad and connects so those are two separate bridges that will be evm have most compatible move those assets into evmos and then from there you can take those usdc assets uh there's a some sort of conversion that happens that you allows you to move them into the native interchange uh assets that we're all used to moving around it's like changing the address format

more or less you're you're changing the format for how the token is represented on chain either like an erc20 token format or it's a cosmos token format okay and will those be fungible like say you move from from connects or you move them from no bad are those going to be fungible once they're in f in fmos yeah so there is a direct mapping so uh governance approved uh token pair so that the computer is just a mapping between the cosmos and the erc20 asset so that there's only one canonical representation for say we on ethereum

for uscc so that there's like uh only one token um so that this ensures fungibility and then this token will be able to be transferred via ibc to other chains okay and his fungibility also insured so they're they're insured via these two bridges but is it insured like is is a polygon usdc or a polygon wrapped eath equivalent and fungible to a ethereum wrapped eath once it arrives in atmos or those still kind of separate tokens

so the the the fungibility needs to the fungibility between like all these bridges is something that we're currently working with multiple teams frogs is already working it's also deploying on atmos and so yeah like as you mentioned there they're going to be like multiple bridges it's the same as with it's the same as the problem with ivc and fungibility when you relay the same token via be a different sort of like ivc routes um the tokens are not fundable because there are different security guarantees in all these different chains that the token has passed through so that someone needs to basically assume the risk and um

having well i wouldn't say like a canonical bridge because it's not maintained by the team but there's like a preferred way of routing all these different chains it has different options on security levels also helps in terms of fungibility i would expect different also like d5 protocols state like curve or or i think it's kinesis the name of the project that is also going to do like stable swat um state a stable coin amm [Music] um so yeah to handle all of these fungibility issue yeah i was talking to the kinesis guys uh the other day and it seems like they want to

address this fungibility issue which exists like it's independent of atmosph this exists already in the ethereum ecosystem and there was some debate you know around i think it was a couple months ago where like about which bridges should uh be should be allowed on osmosis and like sunny weighed in with like his opinion which was we need to limit um or like at least for now we can't create an experience where there's all these representations of the same assets which creates like a user experience that's degraded for friend users um

like we've seen on ethereum wherever you have like 20 different representations of usd assets for example like on the same on the same uh liquidity pool uh like on the same like on unison for instance so why are there two bridges why why nomad and connects and why not like a canonical bridge um so nominal clinics are able to provide it's it's sort of like the same breach under like underlying bridge it's uh connects is using like nomad infrastructure but there are different options for um in terms of speed of the transfer in terms of like

um because it's an optimistic bridge and there are different like the pros and cons of using like an optimistic bridge in terms of latency or like how like if the transaction is optimistic and then you you have this uh a period that you can uh that you can challenge basically the transfer um so they're they're more or less two sides of the same coin um in terms of like using the same infrastructure by offering different like options to the user okay and will what what about the sort of inter-chain accounts and the ability for

like uh an address on like on cosmos or on osmosis to control addresses and smart contracts on fmos what what is that going to be possible and what kinds of things does that allow yeah so interchange accounts is a it's a functionality that we're really interesting to work on um we we wanted to implement it for for mainnet but unfortunately the the ivc release wasn't ready um so one of the cool things that we want to support is for example like you have all these funds all these evaluators that have tons of items tons of osmosis and other chain and other

um tokens but how do you how do you manage the token safely uh in an organization right now on on on cosmos you need to create a multisig like a two out of three and there's no ui for it if you've used the costumes multisig you know the paints that i'm that that i'm explaining to you right now because it's it's really hard uh we even we've been core uh core contributors and core engineers of the costumes ecosystem for a while now and even us we have an internal guide of how to make a transfer with the cosmos multisig which only is supported on the terminal

cli yeah this this is a this is a huge pain point i think in in cosmo it's a huge family yeah and uh one of the big things that we want to support is for accounts on the customers have on osmosis to be able to manage and um assets on for example diagnosis multisig so you can have the same assets and and you can just use gnosis multisig for everything um and manage your cosmos uh coins your um your osmosis your atmosphere c20s and yeah like atmos tokens as well so it's it's going to be like

improving the user experience is something that we really care about and using all these interoperability functionalities to also benefit the rest of the cosmos ecosystem i think it's going to be like a key component here yeah i mean that being able to use a product like gnosis multisig on uh on cosmos would be just like amazing i think you know they've really nailed the multi-sig um ux and and feature set and um i mean i i'm also kind of surprised that no one's built this already on on cosmos maybe it'll come i mean like you got the guys working on dow dow and there's like some

[Music] you know multi-sig there but it's kind of clunky still and the ux isn't great and you can't add addresses and the potential is there but it's it's not it's not a production product yeah i think you can in the contracts but like the contracts support those calls but you still have to do like a you know command line um thing to to get that to work yeah so what what what will be the defy compo what kind of d5 applications will be launched on fmos you mentioned ave earlier uh but are there native projects that are uh anticipated to launch there and i think like broadly what what will be

the defy experience on fmos and how will it be different from what we're used to in ethereum and maybe how is it different from uh osmosis what does it bring to the table for the team it's more or less the same but now it enables them to so for all of these um chains to access the token directly via ivc and for the token to be transferred over the ibc which also provides more liquidity to the project additionally you have the token model that allows users and users for these all these protocols to spend 50 of the transaction

fee uh on on those 50 transaction fee goes to the monitors and 50 of the transaction fee goes to the develop the developer team uh so the contract owners and so like that is able to fund additional sources um like provide additional funding for the team that's more like uh like why is it better it's able it has lower transaction fees it has additional liquidity sources so from all these evm chains and also from the cosmos ecosystem and with the new interchange composability component as i mentioned before

other chains will be directly able to interact with these smart contracts via ivc and also the smart contracts on fmos will be able to call at the other smart contracts that are supported on the evm on the fms evm on other chains for example so you don't need you don't need to um fragment composability at all okay got it so the the that's one thing i i remember hearing about but i forgot is that um transaction fees can go in in a d5 application on fmo's transaction fees can go to the liquidity provider but they could also go to

or sorry not transaction fees but the swap fees can also go to the validator yeah so the the key thing here is normally on ethereum and this is this is also a key difference with the rest of the vm ecosystem is on ethereum and other evm ecosystems if you're paying the transaction fee that transaction fee is going directly to the blog proposer either the miner or the validator here instead the main difference is that 50 goes to the um developer and then 50 goes to the blog proposer to the developer right to the developer of the of the of the contract of the contract

yeah yes what this creates effectively is a decentralized uh marketplace a decentralized app store where the users that are interacting with the with the product with a defy application are paying percent of the transaction fee to the developer and fifty percent of the transaction fee to the block proposer and it's not that they are paying an extra fifty percent it's just that the fee that they're already paying uh same as ethereum or other evm chains is now being split between the two like the two organizations or the or the developer and the block proposer uh what this ensures like alignment of

with their protocol itself so that dividers still get the rewards that they are generated during the validation of the protocol and proof of stake plus they gave 50 of the transaction fees but now you're also incentivizing all these developers to com deploy their applications to fms and these create a larger it's like chicken and egg problem like how do you how do you bring in more users if there are no applications and um through this you're also incentivizing new developers to come in and deploy um through this uh what we call the app store model so this is a really novel uh i think uh functionality in

in blockchains generally i don't think that there's any other chain that does this where the smart contract developers are also remunerated for uh transaction fees um do you think this is something that other chains will adopt like is this something that you could see kind of spreading in the evm ecosystem as well yeah i think there are a lot of i mean so if you think about the way of funding a team right now the teams either um raise funding uh for the company like traditional sources of funding go to vc go to like early

angel investors and they provide like um yeah they they dilute themselves in order to attract new investors and then you also have like the token model where they launch a new token and they disrupt that and the team holds a large percentage of the token but if you think about it there hasn't been a like a sustainable long sustainable um funding mechanism that is directly correlated with the usage of the application so the more users interact with the application here on fmos the more transaction fees

the developer will get so it's a long sustainable and it's and it's in parallel to these two other funding models that i i mentioned before but it's a more like it's a healthier one in the long term because it's a it allows to generate uh continuous revenue to the developer team yeah it like totally changes the economics i think for the developer team and um when a developer launches a contract uh are these are these parameters that you can set say okay i want to take like 50 or i can take less or like if i don't want to have any other transactions like you know different different uh

different contracts might have um uh different ideas about how these fees should be uh repetition between the miners or the validators and the and the contracts uh or the teams um is this like by default or is it something that you can set up yeah so it's set by default um or or like this this is something that we need to we need to still analyze with with the core team um but for now the the current specification that we have it's it's set in by default but what you can do as a developer is if you don't want to actually earn these rewards you can set the

address that you want to instead of you the developer team receiving 50 of the transaction fees you could um redirect those tokens to the community pool if you want or to the validators itself so you can get the validator address and then reward divider or if you want to allocate those tokens to like the entire pool of validators you can also distribute them to entire pool validators um so there are different options here and you can even you can even like set another contract uh developer address or or fund another contract

uh through this mechanism yeah or you could fund some charity or something exactly and uh that's totally up to the developer and they can change the address whenever they want does does that do i mean do you think that so let's say we have developer because then you you i it seems like you get in this situation where developers are competing also for their transactions to get processed so like if a developer uh is awarding more of the transaction fees to the validators so awarding less transaction fees for themselves then there may be an incentive for those transactions to go through

um first is that is that something that is is taking into account here or does that not really come into play at the moment so at the moment the trans the uh tendermint mempool only supports uh um uh first in first out cue so you send a transaction with say 100 tokens in fees and then you send a transaction with a thousand tokens in fees the one that was process before it's actually executed before unless you run some like fork of tendermint where you modify the mempool internals completely um it's not

currently supported but i think the next major tendermint version is supporting like prioritize q and we'll definitely need to take that into account but the current the current model doesn't support it um straight ahead and but yeah this is something well depending when this functionality is released we'll need to take into account so if we if we deploy this functionality first we don't need to account for this particular case but once we want to upgrade to the latest uh tendermint version we'll have to we'll have to consider this for sure okay yeah

and what what other kinds of innovative things is evmos uh does it are integrated in atmos that don't exist in other evm chains or what other innovative things that are are are you implementing in the token model so the other innovative thing that we took uh from existing defile applications is uh the concept of usage rewards so if you read the uh rec drop token model um you can see that we are dropped tokens uh to users on ethereum based on the gas that they've spent um which is called the gas drop model and the more you spend on gauss on

different contracts the more tokens you get air drop and so we wanted to implement this on the protocol level for users that interact natively with different contracts so what we're doing is taking the alp incentive model that osmosis has so like incentivizing the liquidity pools we're implementing this but for smart contracts so the more gas you spend on smart contracts uh the more usage rewards you'll get so you're incentivizing certain contracts so for example bridging could be incentivized or amm swaps could be or like i don't know like an nft marketplace could be

incentivized and this is all voted through governance so the community votes for which are going to be the contracts that are going to be incentivized yeah you define a number of weeks that you want these incentives to run and then you yeah you're basically incentivizing on all the users that interact with this contract so it's going to be like a growth hack mechanism because now if you have this contract that is incentivized and also the developers are getting transaction fees the users are getting tokens just for using the application but they are paying with the transaction

fees right and then half of those transaction field will go directly to the developer team which will attract more developers more contracts and incentivize and more users interact with the protocol itself so it's creating a positive reinforcement cycle in which you are attracting more developers and more users over time so you know are are you confident that fmos um will also act as a growth hacking mechanism for the interchain ibc ecosystem because like you know as a as a developer it makes sense to go deploy your contract there

um there's going to be a number of blue chip a number of blue chip [Music] ethereum adapts that are going to deploy there like ave is one of them um the liquidity it'll be pretty seamless to move liquidity in and out of fmos yeah kepler is very supported natively with ethereum metamouse is supported natively with traditional ethereum signing plus custom assigning transactions through eip 712 which is a meta transaction standard so now with metamask you can sign ibc so you don't need to add any new functionality to the existing

project or to a new york system client you just need to support um erp 712 if you are using like a ui for example yeah i think it's going to be huge for the ecosystem because attracting new developers and new pro a new projects in general and new users that are native from the ethereum ecosystem it's also going to attract more tokens more use cases for the interchange ecosystem yeah i i was part of the core ivc team and were close with other teams in the ecosystem so all this functionality that is going to be deployed on atmos is going to be available to the entire cosmos ecosystem through ibc

and this is something that we're not gonna try to for example like capture all his value only for the fmos chain but we want all these all this liquidity to come into the cosmos ecosystem all these different use cases to come into the cosmos ecosystem all this new functionality from the ethereum tool used from this ethereum tooling that already works and it's already supported by multiple teams it's gonna be available for like the the one that i explained with the multisig is going to be available for the rest of the our costumes ecosystem um through interchain accounts for example

yeah and all of the dow tooling and all i mean all the yeah i mean this is just like solidity and the evm have such an advance i think uh like any other other blockchain in terms of tooling um yeah i think this makes so much sense and something i tweeted about um just before the call and then we're really excited to work on is creating like a product um for new products to make it super easy to deploy contracts uh to avimos but also but also airdropping new tokens to atmos takers to uh gas gossler contracts so the same

that we implemented for the cost drop on the reg drop on direct drop will be available for users to just like select those parameters filter the con fill like select some filters like um leave out some centralized exchanges select the few validators they want set up minimum amount set up a minimum amount of gas i want select ivc transactions governance participation and then you get a csv file or or like a next spreadsheet with all the information of all the addresses that is publicly verified because you will also get the amount of the the list of queries that you can also run manually in order to verify this and

then you will be able to urge up the tokens directly to those addresses um by by using it directly on the on the evm okay this is this is this this is huge so i mean you you're essentially you've got to build all of this data scraping uh infrastructure to do this right i mean this is a centralized tool that you're describing it's not a it's not an on-chain tool yeah it's essential it's it's uh it's basically it's basically the mechanism that we implemented already for the wrap drop but turning it into a product not only for ourselves because we already implemented the

direct drop mechanism and we also already selected the the contracts the the users from that staked on osmosis from customs hub etc but we are also supporting this mechanism for new projects i wanted to apply to atmos to be able to use this tool without any like it's without thing without just spending months a month into like selecting the contracts and stuff you just need to select the contract address you just need to select like oh what's the minimum gas that i want and then you can run these query bigquery of course and then you get all these addresses and

you get the amount of tokens that you want because you're also being putting the you also provide them as an input into the amount of tokens so you won and then you yeah you just get the addresses okay that's that's interesting i mean that's that's a that's that's a product that you could you could probably you know uh charge for like uh as a separate as a even like a separate product so you know i i i want to i want to zoom out a little bit and talk a little talk about the interchange smart contract uh landscape if you will you know there is like cosmosome has a decent decent sized foothold i think in the cosmos ecosystem with like juno

being you know one of the main chains where people can deploy his cause and modern contracts um other chains are also implementing cosmosome yeah yeah terra osmosis club i think is probably also going to do it at some point um or keychain um you know then we have agoric that a lot of people are you know very excited about and now evmos um i i don't know i mean my feeling is that this fragmentation it could go either way like i mean the fragmentation could be good for the ecosystem because it creates a lot of uh different types of applications and

use cases but i think that alignment also helps accelerate uh development because everyone is kind of focused in on one thing and my my thinking around this is as the ecosystem gets bigger and uh lots of new people come into the ecosystem is this bizarre approach that cosmos has always uh kind of stood by going to work in creating a a like a competitive ecosystem where people can actually like an ecosystem that can compete with you know the likes of solana and you know ethereum and some of the other uh big layer ones

i think a lot of these uh chainsaws specialize in different use cases um you have of course a base layer infrastructure like the like the atmos the junos etc but you also see like application specific chains that have bms that are specifically targeting a use case like osmosis um cosmos environment is probably not going to support say i don't know random governance uh smart contracts or or whatever like nfds necessarily is that's gonna be probably stargaze but it's uh on osmosis is gonna have custom

awesome environment for for only d5 so it's kind of like even though they have like the same application underlying application they're kind of like narrowing down on the use case according to what they're already implementing on their chain that being said i think there is there's also going to be alignment between all these different protocols through ivc um even the even the the ones that not necessarily have the same uh run time like or same vm like cosmos money on atmos that that implements the evm so once we implement the functionality for

smart contracts on atmos to call smart contracts and juno or any cost of awesome chain like the uh collaboration we're currently working with startups for nfts it's gonna be huge because then you'll have the same standard for calling smart contracts on on both chains and that's going to be creating a huge alignment uh where the chains now don't actually need to compete necessarily but they can also access the benefits from the other counterparty chains that are that they're connected to so so there will be so it won't matter what the underlying run time is accounts

on the interchange are going to be able to control contracts regardless of what the runtime is is what you're saying yeah they will be able like contracts on the avm will be able to call contracts on cosmos and vice versa i think assets like cw20 which is the cosmos representation of tokens are going to be compatible with erc20s on the evm and both of them are also going to be compatible with um the cosmos token forum um [Music] so the like creating standardization and creating like interoperability components between all these different runtimes and evms it's also gonna help

solve many use cases and it's going to be able to provide more uh functionalities like and and something that we haven't even talked about is like interchange farming so farming on the interchain is something that i'm particularly very particularly very very interested in and um it's it's a model that i call uh interchange workflows where you set up a workflow for one contract to be called in one chain which calls another contract on another chain or interacts with osmosis or whatever you create the you get the lp pools and then you're

creating the same uh hype that you had on ethereum and 2020 with d5 forming um but you will now expanding that to the interchange because of the interoperability and that is going to be huge that is going to be huge and and in order for us to support these use cases we need to provide interoperability between like these different evms uh different vms and uh this is something like our team is personally really excited to work with and and happy to work with any other team and that's why we don't see like for example juno or any other customer watson chain as like competitors or anything like if we are able to reach

assets and reach contracts through ibc it's going to create so much so many opportunities for all the end users yeah i mean i i i fully i fully am i'm fully on board with the idea that interoperability uh just creates more value everywhere and which is why i'm i'm so bullish on on on cosmos and the interchain and what ibc brings in terms of uh you know all these ecosystems being able to um to interoperate and we haven't talked about like polka dot also like the ability for you know polka dot assets to move in and out of the inner chain um yeah i think like interoperability is

is uh is the way that we're going yeah and on avermost will be a point of entry for all these polka dot from from us is in the um pocket of the ecosystem and also from kusama what what happened in the days leading up to march 4th and what happened afterwards and i just want to kind of set the context here and then maybe you know talk about with the vulnerabilities uh that were uncovered and and then what's the relaunch plan because i think that's what people are really interested in is like where are we going from here before

maine and launch so we have finished all the components we had finished all the claiming functionality the dashboard was pretty much ready uh in terms of um i think like only governance was missing but that's why we also gave like that we said on the on chain the claiming to start like one day after minute launch so we han and this is something that we unfortunately realized right after launching was the ledger addresses were being incorrectly displayed on um on kepler so like

if you generated a soft key like a software key you would get the right address but if you had your for example your address that was generated through ledger and then you change uh the network to avmos you will be displaying the incorrect or an address that is invalid more than incorrect so the as i mentioned before like we're not supporting the cosmos case in order to to have all these ethereum functionality tooling and and and like working properly et cetera like metamask because of this the there was a huge risk in terms of like users transferring their

claimable records theme to invalid addresses so that all those funds will be stuck um and they won't be able to be claimed and so so we decided to delay the launch added new functionality in order to check for these use cases and this vulnerabilities um there was another that's what's like the kepler one um on the other side we had the um metamask issues where on on the evm functionality not working properly because um so in order to for you to convert the cosmos tokens into erc20 tokens so like changing the format

the the cosmos component needs to listen to the events on on from a contract in particular and then convert this this token format and this functionality was disabled but because it was disabled it was also disabling by mistake the eoc 20 functionality which was erroring so like any transaction that was going through the evm then went to this pro post processing logic um made the evm transactions fail and this was initially tested locally on the on the test net and everything but

because the parameter was disabled it was actually the parameter that was preventing this we had planned an upgrade schedule upgrade for the following monday but during friday and during the weekend we got a notification from the core ivc team that there was a potential critical vulnerability uh that you can read more about and on the post morning that we created on on the security tab on github where we disclose the vulnerability so we started working on it during the weekend and we planned on emergency upgrading for the day before for sunday

and unfortunately what happened there is that the upgrade procedure was successful but the upgrade itself so like the the notes that had to hold their chain their their values i had to hold on their notes download the new up the new version and then restart the notes that process failed and that out that process was meant to be automatic on the automatic upgrade as the processor had failed and then soon afterwards which is something that has already happened in the ecosystem like the automatic procedure is not working as expected and then but it's a super quick fix in the sense

of you download the new version manually and then you restart the note because your your note is already halted so you just need to download the new version unfortunately some values were still running on the previous version so that cause issues with consensus and blocks not being produced and then after some time a few ballots were still offline so we weren't able to produce new blocks and then finally somebody's were um by mistake double signed while trying to fix this consensus round because it was like at 66.

so really close to producing a block and a few values double sign unfortunately during one of the boating rounds uh in the tendermint consensus and they were slashed and then took the 17 of the voting power out and yeah then it was pretty much impossible to um restart the chain uh without the required uh threshold so so there was two things so there was the issues with kepler and ledger integration where users were sending funds to keys that aren't valid first part that i mentioned in the

interview that the most challenging thing is actually getting all these tools which are not part of the core protocol to work on like to work with the evm that they were not completely designed for or with all these evm functionality because the ledger up that we're using for example it's not the cosmos led drop this is a theorem like your own or or like soon we'll be launching a new atmos ledger um that supports this but yeah i encountered this like i was trying to claim my airdrop and i yeah i was using a ledger and i was following these guys and it wasn't quite clear what was happening and then i i don't know i think i got the tokens but

then you know because the chain was restarted essentially like kind of um kind of like uh you know forgot about it but so there was that and then there was this other vulnerability a lot of attackers to i think kind of claim multiple air drops yeah through on uh on malicious ivc chain but it was a really really tricky vulnerability where you had to basically create a malicious chain basically spin up button with custom functionality change the synonym authorization mechanism um once that was changed you need to speed up the

the chain with with the values with then afterwards you need to connect that via ivc um by setting up a relayer on and creating the handshake and then sending an ivc transfer uh to mmos in order to claim the tokens yeah and here basically you were you were spoofing one would have to spoof the address from which the transaction came i think uh okay so i mean this is all described in the post-mortem and i'll link to that in the show notes the post-mortem is like really detailed and people should take a look at it if they want to learn more about what happened um and then there was this this double signing thing and i

think that was the most interesting aspect of the of the postmortem um you know because so the these these validators were were tombstone and i went and i looked at the cosmos documentation to look what that to look up what that meant and uh from from what i from what i read it seemed like this is a proposed um action it's not actually is is it implemented is this something that exists double signing causes your validator to be jailed permanently so you're kicked out from the valider said you're not able to produce new blocks

and you're you're basically yeah putting a tombstone on the on the validator because it's not gonna be able to get out from the jail's jailed state ever but what does that mean practically because i mean like couldn't i just spin up a new validator with new addresses and move the tokens over there like yeah you can do that but you won't have the voting power oh right okay so all the delegates uh you lose all the delegations okay yeah that makes sense it's a it's a byzantine action where you don't know if the voter is actually committing something mandalicious and that's why

it's designed to be tombstone where you like traditionally with like leaving the upgrade procedure out where you where you are where you're most likely to double sign because you're running two conflicting versions and signing one block and then another block and then you're double signing at the same height or at the same round um which is it's not malicious but during the normal operation of of the blockchain where you're producing block if you're double signing a a regular block that could be something that's potentially malicious

but in this case like you were having um violators they would this was a case where the violators wanted to um ensure that the consensus was reached by trying to so the main problem here is that because so a lot of the validators were offline there were not new blocks being produced and the the peers were basically reset so you were losing all these spears all the time so in order for you to become online you have to reset the peers manually update the peer list and be able to start uh sending messages with all the rest of the alarms in in

order to produce this new block and you had to do this manually and sometimes the batteries that were already offline um were kicked out from the consensus round because they didn't have healthy peers or some of these spiels were removed because they were not producing blocks and then what happened is uh with this like trying to become online again during this process that took so long it took like four or six hours um being in the exact same state um caused some validers to just double sign by mistake

while trying to become online again because for a long time it was like 66 of the voting power online so like just just right below the threshold in order to produce a block and actually a block was produced um the thing is like in the next block some of the values were offline again and then you had to start this new process again yeah so so these validators are tombstoned uh will they continue to be tombstone once the chain relaunches because it's i don't uh from from my vantage points it seems like if these validators um that this was a consequence of all the confusion

and of the upgrade yeah was there any evidence of malicious behavior or uh no there's no evidence of malicious behavior and um what we're doing to do is restart the chain from the previous upgrade so that there is no evidence of uh double signing on chain so these evaluators can resume operations as normal normally what's what's what's the next steps and uh what are the plans that we lost to chain and when when is that going to happen yeah so today we're launching an internal testament uh sorry the public testnet with with all the changes

um this is going to be the better the better version and we're going to run that with the community of validators and once that's done um we're going to ensure that the bugs are well the box will be fixed if if this tesla goes successful but then we also need to test that the restarting the chain is actually not breaking any anything else based on on what i just said the the upgrade was now successful and you basically need to restart with that with the previous state okay so are you is the plan to restart the chain from zero or from from the block before the upgrade which i think was like 58 700

like you're not sure yet where um it's it's going to be 58 000 okay so like a lot and this is what a lot of people have been complaining that some users were able to claim their tokens while others not and the ones that were able to claim their tokens were the ones that read the specification that we had publicly available and um we're able to you know like create a guide or what went on and um but as opposed to other chains in the ecosystem uh we have uh we're distributing the tokens uh epoch space so like every day at the same time

um so that effectively reduces the amount of allocations to like only three events so like the chain was running for four days uh but we we started the claiming process like a day afterwards so the only ones that were able to sort of like benefit were the ones that claimed during these three periods um so that's where the i mean instead of like 58 700 blocks where in every block you were distributing all these rewards you only distribute them on three events um so the users were not necessarily

i mean there were some users who were able to claim yes but um there weren't many events there weren't many tokens that were distributed because of this um was only three events instead of like more days or more events or like block and then a per block reward yeah i i heard concerns also about validators who had sold some of the tokens over the counter um so so those transactions by by starting at 58 700 those transactions are also preserved and aren't aren't reverted yeah so restarting the chain will

preserve uh restarting the chain from the latest height will preserve the entire state of the of the blockchain yeah what what what is the impact on the wreck drop here if any are are you just going to get more tokens because they've had to go through this or are you just keeping the wreck drop sort of like tokenomics as is so this is something that i mean we already increased the uh airdrop allocation initially like before this happened because of the previous delays but yeah we're gonna give more time um for users to claim their full amount before the decay period starts and then yeah we're probably gonna be um

yeah yeah we're gonna give more time but uh we we're not considering at the moment um giving anything on exchange for this but there's something we haven't put into consideration and then we um we are going to be the ones providing as i mentioned this product for you to get a next um spreadsheet and on airdrop tokens um to different users and probably one category would also be so like users who weren't able to claim during direct draw and uh what's the lesson here like

what have you learned from this or what's the team learned and i think there's also like a broader lesson for the blockchain ecosystem as a whole yeah yeah so the the main lesson that we learned is we had focused our testing on qa processes mostly on chain um not actually with all the uh products or all the um the tools that our users will interact with like kepler metamask etc so also focusing on like an end-to-end interaction with all these

um tools is something that it's we've learned that is really important as part of like deploying new functionality of course because something some tools might work and sometimes my non uh or some might be explaining correct information with like for example what happened with kepler um so ensuring that everything works end to end is something that we're also to spend more time on and we've also like creating we're also creating like a readiness checklist ensuring that uh all these functionalities tested well specified discovering like potential attack box vectors not only from

internally to our chain but also um without affect ibc uh etc and do you think there's a there's a a broader lesson here to be learned uh from this episode and i i can't i can't help but you know they're totally different episodes but there's been like two kind of crises in the cosmos ecosystem over the last month you know there's been the whole like juno whale governance uh proposal and then um you know what's happened with evmos um what what are the kind of takeaways here for the ecosystem

and uh you know how how can we uh you know hope to maybe uh do do better in the future like avoid these sorts of things in the future or um yeah yeah so big part of our approach and launching this functionality is ensuring that the cosmos ecosystem is ready to support ethereum and the ethereum ecosystem is ready to support cosmos and we're working a lot in um [Music] to ensure that is is going to happen uh through atmos on on supporting all the cosmos tool and the ethereum tooling is really important

the other thing is creating more guides and more documentation in order to ensure that testing functionality is as expected upgrade procedures work as expected so that we are not we're going to be the only ones um we're going to be the last ones to encounter these sort of issues so that other teams can also learn from our mistakes also other teams can learn from our practices in terms of like testing a qa i'm really happy to share without those guides and those testing practices that we're introducing to the rest of the community i've already been uh working with the ivc team to ensure that

they um like they enable more qa processes like behavior tests against a specification etc yeah i think like the the testing and auditing aspect uh remains the biggest challenge i think in in blockchains as we increase composability and interoperability and uh you know it it's going to get more and more complex and there's going to be more things to look at in terms of ensuring that all the eyes are dotted and the t's are crossed and um but yeah hopefully with you know it's it's like uh

you know it's it's like i look at it sort of similar to the airline industry or like the aviation industry and how you know we've we've it's it's unfortunate but like we've learned so much from uh from the from all the incidents and crashes that happened you know in the last like 70 years or whatever and and now like plane crashes are fairly rare when when they happen it's it's horrible but they're reasonably rare compared to you know like what it was like in the 60s and 70s where you know your risk of crashing on an airplane was like you know not incredibly high but i think was much higher than than what it is today

and i think we'll we'll go through the same sorts of things in crypto and this like you know that wrecked leaderboard is like you can kind of compare it to the to all the airline all the airplane crashes uh that happened you know in the last 50 years yeah the main thing is like cosmos is hard and then on top of that we have the evm and on top of that we also have like ibc which adds one layer and then another layer of potential vulnerabilities which is harder to odd in harder to test and harder to um but we're implementing all these processes and setting all these things

in place so that we ensure it doesn't happen again yeah and the good thing is i mean also this happened at the launch of the chain like no funds were lost essentially there's there's been no real economic loss from this and so i think you guys will recover you you're definitely doing the right thing about being transparent and um and i'm i'm still super excited and bullish uh about fmos so like really looking forward to you guys launching it awesome yeah so when when is the relaunch do we have a date uh we don't have a date because we're working with we're gonna be working with the violators and to ensure that the transition is smooth i'm also going to

ask for their availability um because we don't want any posts any days without them knowing that's a big doodle that's a big calendly to have like all the validators available yeah uh cool well thanks so much uh for your time and um looking forward to uh yeah looking forward to seeing where this goes like i think it's going to be such uh such a big um push uh in the blockchain space generally and in in bringing together like the evm ecosystem and the ethereum system in the cosmos ecosystem and i can't wait to see what that brings uh in terms of usage and like new

applications and just like growth and it's gonna be massive thanks thanks so much for coming on thanksgiving