Open Source & Open Data: Bringing Ethereum’s Values to Source Code Verification - Manuel Wedler

Transcript

Yeah, doing we got Okay so Hey everyone. Um yeah, I'm happy to be here again at East Prague. It's a really nice conference, not too big and focused on Ethereum's core values. We at Sourcify believe um transparency and verifiability are at Ethereum's core values and today I will talk about how Sourcify brings that to source code verification. But let me start with an forum post that um yeah, my colleague came across this recently and it's in the OpenZeppelin forum and the user discussed about some ESCs done that and they deployed an example contract um on on Görli testnet. So the ESC provide like give some standard about Solidity

func Solidity functionality. So I wanted to see the source code now of this contract. And when you click on this link you see uh this now. This is because um Görli has been deprecated. So was the uh explorer shut down and the question now is where the contracts? Where where where can I see the source code of these contracts? Um the problem is the data was just on one one database and it was shut down. So a part of Ethereum's history is gone actually.

Um yeah, you may say this is just just a testnet but actually the explorers of pro um productive chains have been shut down recently too like Arbitrum Nova or Scroll. So that's a that's a problem. But but actually Ethereum has very strong verification culture. Like the community demands um developers to open source the contracts, to publicize them. So, security researchers can verify that the contract code does what it should.

And yeah, the contract can't steal your money. That's why we do this. And it's good that we have this culture in Ethereum. But we're in a quite weird situation now. The contract is open sourced through closed source verifiers and closed data set. So, it's quite contradictory. Um But let's step back for a moment and revisit what is source code verification. So, when you deploy a contract on chain, it's um stored on chain in bytecode.

It's some concatenation of EVM opcodes. And when you see this, you you don't know what it does. Um it's meant to be read by a machine, not by humans. So, humans write in programming languages such as Solidity and Viper. But now, how do we know if I show you this contract in in Solidity, um that it belongs to the bytecode that I showed you before? Um for that, we need a mapping between the source code and the bytecode and store that somewhere.

And that's what source code verification provides. Let's have a look at the verification process. So, usually when a developer wants to verify a contract, we um it takes the source files, some Solidity files, as well as the compilation settings. Um yeah, the Solidity version, some other compiler settings. And sends these to the uh to a to a verification service. And then the verification service takes them and re-compiles using the compiler, gets some re-compiled bytecode out bytecode out.

And yeah, the verification service then does an on-chain call, text the RPC, and gets some on-chain bytecode out, and simply checks if these match. Um and then we store the result in a database and make it publicly available. So, it's a bit over oversimplified here, but at its core, that's how source code verification works. So, I I said like in the example before, you saw that there problems. Let's have a look in detail what what problems there are.

Um the largest verification service is closed source. So, you can't verify how the verification process itself works. Um there could be bugs. There could be intended misleading behavior. The verification process is just a black box. Um yeah, that's a problem. There There have been pretty pretty bad bugs in the past, and that that were only possible to to identify by white hat hackers. Uh another problem is that it's centralized. As you saw in the beginning, um it fails the walk away test. When the service disappears, the protocol fails.

These contracts on Görli and Rinkeby are gone, and yeah, we we lost part of Ethereum's history with that. Another problem is that the data set is closed. Um while on one hand you can um easily build an explorer because the data's on-chain, it live it should live forever on-chain, and you can just fetch it and build your own explorer with it. But the verified contract data is siloed and lives in a database, so you depend on the service's database. And yeah, the the service gatekeeps it.

You may have to pay money. You You can't really download it in bulk. Um so, the community doesn't have ownership of the data. Um another problem we see is the tooling stagnated for for a while. Like the there was an API that was developed years ago and it hasn't been really developed further since then. Um for example, you you need to provide ABI keys, constructor arguments, uh which are not even required for the verification process itself because you can just fetch them from chain.

Um and yeah, that that leads to a sub-optimal develop experience. And yeah, we we also think there a lot of interesting outputs that the compiler give, but yeah, they're simply not available by by the current APIs. Um so yeah, the current players are not really incentivized to open up because it's a business model for them to to close this data set. And Sourcify is here to change that. So, what is Sourcify? Sourcify started a few few years ago within the Ethereum Foundation as a side project of the Solidity compiler team.

And last year we spun out with the Solidity compiler team and a few other teams into a new organization called the Argo Collective. Um so the Argo Collective is non-profit. We don't generate any any revenue. And yeah, we were seeded by a by a Ethereum Foundation grant and yeah, live live with grants at the moment. So, Sourcify is a small team. We're just developers. It's Marco, Khan, and me. Um so, I talked about problems.

Let's see what what Sourcify offers, what kind of solutions there are. First of all, all of our components are open source, of course. Um our core verification logic lives in a TypeScript library. You can just fetch it from NPM and run it yourself. And you can see the source code, so you can review how it works. You can find bugs. You can um maybe shoot a PR if you see room for improvement. And yeah, that's pretty good.

Because that enables also self-verification. So, you can reproduce the verification result. You don't have to trust us that we just claim that the source code and the bytecode match. You can take the verification logic and rerun that. Um we have an example for that. We So, Otterscan is a local-first block explorer and it integrates Sourcify. So, when you visit a contract page of Otterscan, then you can click a little button and fetch the sources from Sourcify.

It then um locally runs inside your browser our verification logic, re- re-compiles and checks that the bytecodes match with the source code. So, you can reproduce the verification result. Um open source also enables self-hosted Sourcify instances. So, we don't want to be the only source of truth. Um so, we are happy when others also run Sourcify servers. There are a few chains and security companies which have have their own Sourcify instances. These six are the ones we know of currently. Maybe there are even more uh we are not aware of.

But yeah, it's good that they they they are able to run these. Um one paradigm that Sourcify um and fosters pretty heavily currently is verify everywhere. That means we want to reproduce the verification data, not only in our database, but everywhere like on other verifiers. So, it it the like we don't want that when we are gone the data is gone. So, for this we we founded one initiative called the Verifier Alliance.

It's initiative by BlockScout, Sourcify, and us. And yeah, we we aim to be an open standard for the verification data. And it works like that when when any when you verify a contract on any of these three verification services, um the result is not stored only on their own databases. We also write the result into a shared PostgreSQL database, which is then made publicly available. So, anyone can can access the data. Um next is we also want to replicate the data on the databases of the very like the other verification services

themselves. So, we have two mechanisms for that. When you verify on Sourcify, the Sourcify server also sends a verification request to Etherscan, BlockScout, and BlockScout. So, they can run their own verification pipeline and store the result in their uh databases. So, it goes also to their places. And we have another mechanism for that. And that is um via the tooling directly. So, one step ahead, we push the tooling into the direction that they not only uh send the verification request to to one place, but instead to like all verification services.

For that, we implemented a Remix plugin, which became the default now. Um so, when you when you verify on Remix, if you don't disable any of the verifiers, the verification request just goes to all of these places. We have the same for Hardhat. And yeah, Foundry we still have to push, but we are we are in contact with them. Um Another principle of Sourcify is open data. We have 28.5 million verified contracts across 190 chains and make this complete data set freely available to anyone for anyone to access it.

Um there are there are multiple ways to access it. We can just fetch the source code from a single contract via the API. We also And, yeah, but that just works for for uh single contracts and not so much for for bulk access. For bulk access, we provide a full database dump um in a modern format called Parquet. And, yeah, we we can like we export all columns of our database and into this Parquet format um daily and upload it for for people to download it.

So, this exists for Sourcify and for the Verifier Alliance. What we also do is we push our complete data set to BigQuery, so you can use the um the Google's platform for data analytics directly. And this is actually uh updated live while while contracts are verified in Sourcify. Um And, yeah, open data also enables us now to run other services. We run a four-byte service today. Um four-byte service is like uh a lookup service for the four-byte uh function um selector

uh via the function signatures and vice versa. So, there was openchain.xyz, which we have taken over last year. Um we imported the database, implemented the same API. And yeah, this is this this is currently used by tooling internally like Foundry uses for the transaction decoding. You So, you might not even notice that that you use this. And yeah, already use it. So, it's the actually actually our service is now the biggest four byte data four byte service uh in the ecosystem.

And the nice thing is it's also powered by the verified contract data now. So, when a verification goes into Sourcify, we also write the signatures and selectors into the four byte database. What uh Sourcify also aims to provide us better tooling. Just to name a few, so we try to make almost all compiler outputs accessible. We think it's useful to also um have storage layouts and source map, and hope people can use it for for useful um for for their their tooling.

Um we have metadata JSON support. I don't want to get into detail, but if you use the metadata hash which is appended to Solidity um byte codes, you basically get cryptographic guarantees about a verification. And we are actually the only verifier that makes use of this uh cryptographic guarantees in the verification. We also have broad compiler support. We recently uh got support for uh Solidity from version 013, and that enabled a project called Ethereum History to verify contracts from the very early days of Ethereum from 2015, I think. Um They were only able to verify these on

Sourcify because like other verifiers simply don't support such old uh Solidity versions. We also support all Viper versions, and we got recently a support for Fe Lang, which is a um a new contract language also by developed by the Argo Collective. So, yeah, that's that's what Sourcify has today, and now I want to give you a call to use this data set to move EVM tooling forward. Um we think it's very valuable to to have this data set open and publicly accessible and free for everyone.

So, we we are happy when people build stuff with it. So, I think we're thinking of um security pattern detection, um training AI models, building a better decompiler, or whatever you can think of. So, where if you if you have a use case and and need help in the process, we're always happy to help. Um yeah, so that's that's what Sourcify does today, and now I want to give a short outlook um what other initiatives we support for transparency and verifiability.

There is namely namely the clear signing. That's a EF um Ethereum Foundation led initiative um basically to get rid of the blind signing. Blind signing is is that. So, when you um sign a transaction within your wallet today, you usually just see some call data hex string and don't know really what it does. Uh and that's a problem. So, we want to go from this to this. You you should see the clear intent in your wallet when you sign a transaction.

And for that um we like the the working group developed an ERC. The ERC was initially written by Ledger and refined in the last half year within the working group. The working group by the way by the way consists of few wallet companies, security companies, includes Ledger, Trezor and yeah, more. And yeah, so we have contributed by building a library for that and there's a small playground and yeah, ESC is pretty ready now and we we are now in the phase to go actively to wallets to support this.

Yeah, if you want to try it out you can try it out the Safecypher playground for clear signing and if you build a wallet just reach out to us so we can keep you in the loop about this. That's it for from my side. Thank you for listening. Here you can follow Safecypher and Verify Lines on X and yeah, do we have time for questions? >> Yes, we do. >> Okay. >> Hey, so if anyone has questions feel free to ask the speaker now. >> Yeah.

Okay. >> Okay, if there's no questions and you will get some later you can reach out to the manual and he will be happy to answer them I believe, right? >> Yeah, I will. >> Okay, thank you so much for your speech. >> Thank you. >> Have a great day.