Blockchain Data Availability & Agility Layer

Transcript

hello hello everyone good afternoon here so um today I'm going to give a talk about train storage which is the open source broad trained data availability and Agility layer income base so here's my own introductions and I'm the head of data Performance Services in combay so my team actually runs all the convey stator business status that including business status and blockchain dators and blockchain data applications and blockchain research Etc so I am just a second

um I am a a father of three three wonderful kids and my crypto career including 2013 I was buying my first Bitcoin from comb base while helping my daughter in my arm and yeah I still regret I didn't buy that many and 2016 I wrote my first line to solidity call in ethereum um you can actually contact me using Telegram as well as that's a new one which is a comb-based wallet you can actually message then the exam PT protocol in case you don't know what combase is

combase is a the largest compliant Exchange in US probably in the world compliant that's number one and co-based missions is to increase economic freedom in the world coinbase is exchange but it also has a very rich lines of crypto products it has coinbase exchange which is a compass app combat's main app is an ex centralized exchange allow you to buy and sell cryptos coinbase wallet which is self-custodian wallet that enable you to store your cryptos in the self-custodial way we have coinbase um usdc okay that's called usdc which is the stable con the one of the most used

stable coin in the world coinbase earn is a program that enables crypto users to learn the enable users to learn in crypto and also earn crypto as well coinbase Pro now rebranded as an advanced Trader which is enable professional user to actually trade crypto and we have a bunch of the business line of the product as well okay here's the agenda um we have like 12 minutes ish so let's get faster so I will actually go through what data is income base yes and also we talk about very specific blockchain data income base and we'll present our solution trained storage and explain the why and how and also introduce the

ecosystems convex theater this is what a crypto company data needs to look like Compass has three kind of data the first part is the blockchain data second is a product data external data all these data have their own characteristics blockchain data is properly available and when we talk about blockchain data it really means the transaction blocks and all the derived datas as well as train specific datas and we use that to actually support variants of different blockchain applications product data is a proprietary some of the data's user generated and

some of them is user Behavior we use that to help with users to make them have better application experience external data is interesting is when the supplies because no one has all the data we have to Source some of the data outside which is from some of the vendors and for example we're actually getting the price status some of the price data from outside we also actually getting some of the extreme trading warning from outside for analytical purpose use case opportunity so because this talk is about blockchain data right so let's focus on the first vertical of that use case of the blockchain data we have three kind of use cases number one

it's a near real-time blockchain data applications that including when you're actually building a wallet and then you need a wallet indexer when you want to display nft index nft then you need the nft indexer right so these these kind of application including block transaction tokens balancer you name it the key as for this kind of data is the backfield performance because you won't get it right at the first shot you need to do multiple iteration and as the application evolve you net you need more or less data for your index and also the data fashion is matter right you want to see your token balance in seconds not hours not minutes and that's matters the

second category it is complex analytical workload um we produce a lot of the on-train metrics in coinbase for example we need to analyze where the money flows and for example the bridge analysis and we also have machine learnings on-chain machine learnings on-train data machine learnings and for example to produce a nft price predictions this kind of application require us to have ETL tables streaming data sets some of them in real time some of them in less real-time format and we require this set of application will require the data need to be flexible and ecosystem Integrations

and because we do not want to reinvent everything right we want to connect our system into the Big Data ecosystem things like spark and Ray in order for us to process it last category is very Enterprise specific it's called high trusted data as a crypto company even though everything on blockchain we still need to actually do things to connect to the real world for example we need to making sure the particular is com particular transaction is confirmed we then actually send out the token because otherwise you lost money and we also need to do account consolation to making sure that the accounting is The Ledger is actually balanced in real world so

that we can do quarterly report right company is a public company just a reminder and we also need to do fund flow to making sure the money flowing in flowing out uh coming from a quick guy and never Flow to a bad guy and for these cases data lineage methods and verifiability matters so let's get into um the requirement giving you see so many use cases is very very mind-blowing coinbase is a crypto company I stress that again broad train data is the core need of its business

as a company the first part of the requirement is data self-services why doesn't matter that's important right he's a public company you need to making sure your company uh you need to actually make sure the data that you have trustworthy high quality that's a that's that's a must requirement also privacy and compliance think about that would you like your bank to send your account numbers to a vendor in order to fetch the balance probably that's not a good idea right so we need to actually build all this capability in-house to protect user your piracies

cost efficiency matters we cannot afford to put 50 million dollars a year in order to use all the blockchain data in Mary's form and that that explaining itself we also need to get repeatable calculations so when something is wrong and then we can replay the calculations and we know why it is wrong and more importantly we want to support heterogeneous blockchains at our own pace we don't need to wait for a vendor to have that second requirements the data availability data availability is a very big topic the definition of the data availability here in some sense is similar to ethereum but in a more centralized form it has three dimensions

number one you have the data there and then second our user can access to the data and third is a trust the trust is really means in the sense that in the sense that the systems have sufficient assurance that the full transaction data was made available for verification without the need of our user to download the full transition for example things like macro trees is one of examples that we support there and the last concept the last requirement is called Data agility this is relatively new data agility really means three dimensions flexibility user can iterate and perform their own data experiment to ship the data into the

target format that they want so that they can support their own applications efficiency with the cause and the human time in that particular dimensions economics is easy to use and easy to integrate with the existing systems Solutions that's chain storage the tldr if you only remember one thing about train storage it is a file system of blocks and I give you a 10 000 feet wheel um 10 000 feet is about 30.48 kilometers so just in case you don't know

so um give you the overview for that it has a Comics okay this list of components you have the injection system that taking the block from the nodes in the robust form and then in the the story in the lossless format block storage you store the block into you store the block into a block so you just put that in the S3 in some practical sense metal storage maintaining the relationships between the blocks so that you form a train with all these files and we also have a output format transcoder that will convert the block or range of the block into a Target

format Json parquet or some example including Rosetta as well and each of these components expose manageability endpoints in grpc service so you can interact with each of the system and understand how it runs so you can manage that over there um chain storage can take the block in the truss in the robust and the trust trustless form what does it mean it it is vendor failure resistant so you can have a multiple node vendors you name it I can't actually name the actual vendor here right but you name it you can pick the vendor that you want you can pick multiple that you want they will actually get the plot from there if some

of the vendor went down it fails over and also we you do not need to actually trust the vendor data because when inside the injection it actually built in the verification form making sure the data you ingest into the system is correct mathematically proven to be correct and when user consume consume the data users can use the SDK to verify the data that they get is correct as well so this is the overview let's move down a little bit 10 000 feet ish to take a look at one aspect of train storage because we cannot actually go through the whole design because the whole design will take hours in order to

go through that then I just cherry pick three very special characteristic of twin storage number one lossless all the blocks content are stored in the raw form inside the file so this is important a you can recompute the macro tree to verify the data is correct b is all sort of the data application can build on top without the need with minimum need of the node again second we are aware and that's very important that really means our user doesn't need to worry about blockchain report for the reason fastball trained you probably will notice that the real the the train reord

event is happening much often than you thought and some of them is like more like five percent that give a huge problem for the downstream user to consume the blockchain data because when you have the block all the sudden the block disappear right so we provide the interface for people to consume the stream or consume the table always in the canonical train and the third is we adopt the blockchain problem this transform the blockchain problem into a big data problem so if you're good data engineer after you interacting with train storage you probably forget you are interacting with a blockchain data blockchain directly

um result here's a result um this is the performance so the throughput we try to Benchmark that in a fair way it is around 100x performance comparing to a vendor comparing the popular node vendors and much better latencies and 20x of the course saving the same amount of workload so without using Trend storage we probably paying 700ks they may actually give us some discount but you know that's a different magnitude problem but with train storage only 30k and it also resists to a particular rental failure as well okay now let's talk about the ecosystem I have two minutes

so um chain storage provide you give you the access to the block but that's not enough we have a last amount how about big dators we have a system called Transformers on the high level Transformer it is a data adapter that twin the raw data inside chain storage into unbounded well defined skimmers it's a stream so some big data concept when data emotion is a stream when data static is a table so unbounded stream which means unbounded table so for users that what they get this you can see the example here right they

can just read the data source and then load the particular data set and then they just write that and for users they will actually get is a table just keep populating itself you can build your application directly on the table and the job actually here the job actually here is actually written in Scala of course you calculate that in python as well so these enable our data engineer to use the familiar tool to perform the the work that they have okay the second key component of the ecosystem is called train node think about that a lot of the DF application depends on the RPC interfaces right so training it is the component that converts the trend storage data into the

IPC interface and exposure to user user can directly build on top of that and what's the result the result for that is very similar or better performance comparing to the no provider besides we have 10x to 100x throughput and a fraction of the course so open source um that's the last slide and we believe that we have been we have a very very solid foundation for Enterprise grade blockchain data stack when we talk about Enterprise I really means big company as well as if you are running a train you should actually adopt that as well the reason that we open source it to helping the industry not to worry about

blockchain data access in the coming bull run this is the last thing that you need to worry about we enable practically we enable the alternative to build blockchain data availability and the agility layer for Enterprise with self-solventies and more importantly I want we need your help it is still very early time for the open source and welcome to feedback feel free to reach out to me and yeah here's my content thank you very much [Applause] unfortunately we don't have time for any questions there is the vitalik talk in five minutes in building a main stage

thank you foreign