A local first approach for Web3 development - Miguel Palhas

Transcript

[Music] Uh so it's actually a bit ironic for my talk to be right after the Google Cloud one because I'm probably going to advocate for the exact opposite thing although yeah probably different use cases in uh in a lot of things. Uh but yeah, so my talk is about local first development and actually the more extended title should be local first and open source first uh development because that's usually what I try to to uh to use to to take advantage of in my tools and how I build uh everything that I build and we add some visual uh and I want to start with probably a controversial claim that I

think the way we build things is in in many ways it's broken. Uh, and I'm I'm going to explain uh a little bit more, but so first of all, when I say our I'm not talking about everyone in the Ethereum ecosystem. So there are there's the Ethereum core development itself, the EVM and Solidity development. Uh I don't work uh directly with with in in those projects. So I I don't know much about their development practices other than looking from the outside. I work mostly on smart contract development, front-end development for those smart contracts. Uh so the products itself, protocols, uh things like um unis swap and so on. Uh lately I've been involved

with liquidity v2 which just relaunched and uh working with a couple forks. So those sorts of projects that use ethereum as their their back end and then they need their own front end for the the end users to actually use. So there's people involved that are writing the smart contracts. People are building the front end. These people, they may or may not know how the smart contracts work. Uh there's a lot of uh diversity in there. And then there's a lot of nontechnical people that are involved in the project that they need to test the product. Uh they need to test it with with end users which are another stakeholder. So all of these people are in some way involved in testing your product, your protocol. Uh but a lot of the tools that we use or I would say

perhaps the best tools that we use are only the ones that uh maybe the smart contracts developers and the front- end developers uh use and know how to use and they're not the more you go to to to the right or to the right if in this orientation uh the the easier the harder or the less useful those tools become. Um then I include mostly here because I don't don't want to be unfair. There's a lot of really good tools uh out there uh that are really really well made and have caused me no problems at all. But again, the more I go towards the right uh the more I start uh having problems with these tools and they're they're not just solving problems for me but

creating even more problems uh in my workflow. problems from just not working correctly to not working correctly locally, which is what I wanted in the first place, or just downright introducing problem additional problems that I have to solve. I have a particular pet peeve with indexers uh but oh well. Uh so yeah, I think the way we use a lot of those tools or and the way some of those tools are built uh or how they have to be used and how they force us to work uh in order to to use them uh that whole thing uh is to me a little bit broken just and to give a bit more context. So before uh coming to Ethereum, I used to

do traditional web development mostly in Ruby on Rails and Elixir. You may not know these these two logos if you're if you weren't part of those communities. These were traditional uh backend frameworks or full stack frameworks for web development even before React was the the big player that it is today back when we were still just doing non SPAS uh non front end heavy applications. Um and there was I mentioned these two communities because there was a big focus on both of these communities. They were high highly related. Elixir kind of came from the Rails community. Uh there was a big emphasis on developer tooling and that tooling being comfortable and easy to use not just for experienced

developers but also for anyone uh coming in. So if a new developer was coming into Ruby or Elixir for the first time, the not not only was the tutorial very well written, but the accessibility of the tools and the the intuition about how those tools should work was very easy to uh to get on. There was a lot of introductory courses. Uh the hello worlds kind of kind of apps and to to-do lists were very accessible if you have if you had just very basic programming knowledge. Of course, this really can't translate to web3 because the fact that we're using a blockchain by itself adds a lot more complications, but we can still do a lot better and maybe learn

from a lot of the things that these folks used to do. There were there were a lot of principles that yeah, people gave random names to random principles. This is just one of the principles I liked a lot even though the name isn't really that catchy. The principle of the principle of list astonishment was just the idea that the the tools or the the system that you're using as a developer or as an end user, they should work exactly as you would expect them to work. When you click a button, you should already have some intuition about what that button is going to do. If it does something different or if it fails to work for some reason, that's no longer intuitive.

it maybe causes you to not use the product at all or to spend a lot of time frustrated trying to figure out what was wrong. And I think for developer tools in the Ethereum space that's kind of my feeling of where we're at in in a lot of places. So when using uh Ruby on Rails and then Elixir I was used to very simple uh workflows. There was a one command setup. We used to have uh this kind of uh not a rule but we try to enforce that every repo we worked on. A new developer coming into the team could just get clone the repo run a single command that would set up whatever had to be set up. So just a single command no need to read through a a huge contributing guide installing various

tools and so on. One commanded he was live ready to develop. We had shared sta shared staging environment that was updated automatically. You could just go there and test whatever the current master version of the product was. Uh and then we had live previews for every pull request with some fake seed that database so that whenever I open a pull request, the project manager or the user tester or so could open a random URL and start testing my product with an actual back end with a whole stack deployed for them. Uh, and most importantly, and this this is kind of a given nowadays in web development, but back when Rails was maybe first starting, this was kind of a

new thing, the idea that you could not have to compile your code and then go to a web page and refresh and finally see your changes. You could actually edit the code and almost in real time see that change reflected in how the front end or maybe the back end uh would work. That was a bit it kind of ties back into the that principle of bliss astonishment because especially for newcomers to development or in this case web development editing something in the code and having to realize that you may need to compile something to run a data migration to refresh the page those kind of things could sometimes break um your understanding.

uh and now when I came to web3 I mean now I came to web3 in 2016 17 uh but still today I feel a lot of a lot of things are missing from that comfort that I had in uh in Ruby and in elixir so now I I it's very hard for me to have that one command setup because there's just so many different different tools it's hard to have those tools work together if I had if I had um a database or an indexer to my project then the setup for for that tool itself becomes a whole other project. Uh data management across all of those things across a blockchain and an indexer becomes a problem as well. Feed feedback loops are really

hard to solve because now if you're changing your contracts you need to comp compile deploy your contracts but if you deploy your contracts the address of the contract may change you need to update that on the front end. So going from editing a contract to going to your UI and seeing how that change in the contracts affects the UI behavior. There's a a whole lot of steps that you need to go you need to go through uh in order to test that. Um test nets or public test nets like Sapoleon so on they are they are meant to be the place where we create those staging environments and live previews. Uh but they're really hard to use. First of all, finding test net is sometimes a big

pain, especially if you're using using the test net to test with real users because they may not even have a wallet setup, let alone test net. uh interoperability is pretty much non-existent. If you if you're building a protocol that needs to interact with the let's say unis swap for a specific token on on unis swap and that token doesn't exist or doesn't have good support on then you need to deploy your own mock version but then if if you need to interact with a third protocol that doesn't know about your mock version that's it's a whole problem that's it would only be solved if every single protocol would have their own uh fully uh supported version on on that same

test Not to mention the UIs and so on. Uh once you get into subg graphs because usually when you have a front end for your protocol you may need a subgraph or some sort of indexer. Deploying a subgraph locally it turns out is a not an easy problem to solve or I mean it's it's easy in terms of complexity but it's a a big pain. And uh actually I find out found out last year that even the developers from subgraph they didn't know about a few of the tricks I was using to deploy a subgraph locally because it seems that that was that only worked uh almost by accident not as a a first class citizen. I may have a couple of tips to

share on that. Uh managing the state of the subgraph is a problem in itself. Again, if you're redeploying the contracts after the change, managing the fact that the subgraph already had its own states can uh can cause unintuitive problems as well. So, these are some of the tools that I use and probably people in similar roles as me, they they also know or use these tools as well. Uh other scan and other scan are where you go to interact with a contract when you don't have access to an actual UI. You can go to a contract that's verified and interact with the source code. Again, the contract needs to be verified, which is kind of a limitation. Uh, Tenderly is really great

for doing those same interactions, but seeing a full a full stack trace and being able to debug contracts more thoroughly or simulating transactions before you execute them. Uh, impersonators is maybe a less known tool. Uh, but it's a website that kind of it uses wallet connect to hack your browser into thinking that you have a different wallet than the one you actually have. So you can you can go to unis swap and connects using the address from vitalic for example. You obvious obviously don't have the private key but you can trick the UI into thinking that you have that that address. You won't be able to sign transactions of course or sign messages but you'll be able to see the UI in the state that fatal could see

which may be useful to let's say debug uh an issue that only happens for a specific user of your protocol where you need to see his own his actual state u and the graph is uh how we deploy subgraphs uh as I mentioned before so all of these tools except maybe other scan actually they have a very a very common problem, it's that they only work on live nets. If you're working locally, you're deploying maybe an anvil node or hard hats, you're deploying the contracts locally, you don't have them verified externally, maybe because you're not using a test net or just because you don't want to publish the source code ahead of time, then you're kind of locked out all of these things. Uh you can run a subgraph locally. Again, it's

a bit of a pain, but it can be done. Uh but you're locked out of all of these other other things. uh except that you're only locked out of them because they weren't built to be local first. It's not like they have some magical feature that is only capable of working on a remote server on the cloud. Everything that they're doing is fairly trivial and can be replicated locally, especially the impersonator thing which with Anvil is just a a trivial thing that you get for free. Uh so yeah, think the typical web three tool and I mean the tools from the previous slide and a few others. I didn't even mention wallets because there's a whole bunch of them and they

each have their own specific quirks. Uh each of these tools is I think too powerful for low-level users for uh um I think I meant end users in this slide I think not low-level users. So a lot of times I see nontechnical people having to go to Weather Scan and Tenderly and so on and they don't really know what they're doing there but they end up having to go there. Uh and it's they're not powerful enough for us developers because they kind of do the work but I wish they would they would have done it a lot better and then don't allow me to do this uh locally. Uh and yet those are the tools that both both sides end up having to use because it's the only ones that exist.

And so I built uh at the time or during this past couple of years ETI which somehow the lining of the slide was completely messed up. Uh but this started as um a wallet or a wallet replacement app uh that you you would run uh locally. This is all built uh open source and it's already working. So this was I think one of the very first demos I did a couple years ago uh where I have the app running against a front end that I built as a demo. Uh I'm clicking a button for for minting this is actually minting an NFT on this Anvil chain that's running locally. And one thing you can notice here is that as I'm clicking the button, I don't see a popup

uh asking for confirmation as regular wallets would do because this was on purpose uh in this case because I'm using Anvil in this case. I obviously don't care about security. I don't care. I'm actually using a hard um plain text private key because when I'm doing that kind of workflow where I just want to test the UI, I don't really care about security. that part just gets in the way. Having to click a popup and waiting for a simulation at every step would just slow me down. Uh and minting this same amount of of NFTts in that other workflow would uh would take an exponentially longer amount of time. So this was one of the first uh quick wins and again this can be disabled. This uh

this fast mode is what I called and you can still get the previous behavior if you want to inspect the transaction and confirm it before submitting. And obviously it doesn't work uh it only works when um on non-live uh chain IDs for security purposes. Uh and then I add added other other features on top uh such as this from the terminal. This may not be very clear, but what I'm doing here is rerunning the script that deploys the contracts and the wallet has a built-in kind of explorer that detects those contracts deployed in real time because this uh this is all running on Foundry and Anvil running on the same machine. I have access to the compilation outputs from

Forge and I can automatically match that this deployed contract matches this API and then as a result I can also expose the same thing as headers scan uh with a fancy preview of the solidity call that you're about to do. Uh and this is all running locally just by scanning the file system the compilation outputs from from F forge and essentially mimicking the same behavior that uh that you would that you would need on either scan you would need to deploy the contracts verify them uh externally uh not only does that take time it forces you to expose the source code and so on and again those iteration times become a lot

longer and I didn't mention I I don't don't have a demo for the impersonation part side of things because there's actually nothing to demo. Uh you just go on the wallet, you create a wallet of type impersonation instead of a private key, you paste an actual address and you just just use it uh as it would be because being an anvil node uh you can get impersonation pretty much for free. So there's no wallet connect hack that needs to be done, no special behavior. you just say I want to impersonate this address and as long as you're connected to an anvil node that just works and with the added benefit that you can actually submit transactions because on anvil you can do that as well without

actually having the private key. Uh and then this is the product so to speak uh that I uh finished a couple weeks ago or at least the the first version. Um, this is where I try to solve the subgraph and other external tools side of things. So, there's no UI for this. This is just something that works locally through a a REST API and then I'm going to integrate uh eventually into the the UI of the app that you just saw. This can be run locally on our machine. It can be hosted on a a server for a to share among your team. And the only thing this does is you send a post

request to create a stack. In this case, I'm creating a stack called FU. Uh this I'm pointing to local.ety.dev. This is just an alias. If you know LVH.mme, this is just an DNS record that points to local hosts. So it's just a simple way to to have the ability to use subdomains while still working in your local environment. It doesn't it doesn't depend on any external server. It doesn't need to depend on DNS if you if you are offline. Uh and by sending this post requests, you get a stack created for you. Uh this is running on docker behind the scenes if that's relevant. Uh and you get a fu.local which is actually just an anvil

node. You can use cast forge deploy and so on against it. But you also have a subgraph deployed pointing to that same uh anvil node. Uh you can deploy a subgraph to it if you ever had to run the subgraph locally. There's a weird docker compost setup that uh that is running out there. There's a couple of obscure environment environment variables that can be used to uh optimize the way that subgraph runs. Uh that's all encoded there without you having to go through the same hours of research that I did. There's an explorer uh that we're also building right now that's it's not actually pointing to this node specifically. you can just paste the URL of the RPC and it just works right away with any anvil

node. Uh, and right now that's it. Even though there's a couple a couple more components uh that we're trying to build as well like a faucet uh um and so on. The the fact that the graph node is there also means that there's a posgress database and IPFS nodes. Those are usually not not needed. uh we don't need to interact directly with them just through the subgraph but they're there if you need them as well. Uh and again all of this is open source. It's again if it's relevant it's mostly built with typescript rust and a bit of elixir for this stacks part. Uh it's all public on GitHub and we'll have we have will always be there.

Um again free and open source and uh importantly lo local first. So these are all tools that you can run locally in your system. You can modify them as you wish. You don't need to depend on any external cloud. You can host these stacks uh for your uh internal team. I've seen a lot of projects including liquidity v2 that's they create their own uh private test net which is just usually wiring together a few random scripts that then become hard to hard to maintain and hard to replicate by other teams because there's always very specific needs of how those things work.

I actually had to go through a lot of those scripts in liquidity v2 when I wanted to I was trying to replicate their setup and I had to go through the scripts manually to reverse engineer what what they were trying to do. Um so this is just a a way a quick way for you to to get your stack running either locally or privately for your team. And and that's it. Thank you. Thank you. And I think we have time for questions. Yeah, thank you Miguel for this amazing presentation. And we have few minutes for questions if there are any.

I guess there are not. So thank you again. Thank you.