Reinventing the wheel: going away from TheGraph

Transcript

foreign [Music] thanks a lot for being here I'm like I work at Atlas labs and today in this talk I represent to you why and how we migrated away from the graph so the graph is a very well-known product in the serial ecosystem I will assume that most of you are familiar with it if you don't so in the next 15 minutes will be a bit long

um the graph is an amazing product we used it for quite some time at Atlantis we launched our production with it it worked very well it was just that at some time we realized it was not properly aligned with our intended objectives so in this talk I will first begin by presenting to you the reasons behind this migration then we'll see how it was done or what has replaced it and finally I will share with you the lessons that we learned in this journey a little bit of context first so what is that in this so it's a capital efficient credit protocol connecting D5 with real

world business use cases on one side it will allow borrowers to set up their own liquidity pool on chain and to borrow from it and on the other side it will allow lenders to select the power deposit their funds in the associated liquidity pool at the at the rate of the of their choice the weight being the weight at which their firms will be later borrowed uh what do I do in all that so I'm a software engineer for around five years I enter the space in a company name consensus and I joined Atlantis lab since April 2022. my job is quite simple I basically make sure that everything works mostly from the smart contract and

to the front end M so I will spend a bit of time on the front end helping on structural decisions I will help a bit on the smart contracts in order to take decisions and implement the smart contracts and most importantly I will spend most of my time on the back end which is used in order to connect the dots between smart contracts and the front end I'm the one developing it and maintaining maintaining it and it's the service for placing the graph in architecture today so that's why I'm here talking about it to you today um so first uh why did we migrate at the time our needs were first to serve the data that the front end needed so we

have a defy app so we have a lot of complex data to show we have amounts of rates everywhere and the goal was simply to be able to serve this data and simplify as much as we could uh the work that was needed needed and done front-end side then we wanted to have some insight about what was going on in our protocol so the ability to be able to send notification based on particular events or perform some data analytics in order to see what's going on Atlantis lab is not the kind of app where we spend 12 hours a day clicking buttons and sending transactions so we don't need a crazy charge in terms of user and we wanted to keep things simple

as simple as we actually needed it and finally what we wanted the most was flexibility so we are a young startup and things are moving fast and we wanted the flexibility that we needed reflected at the code base level then on the second side what were the issues that were faced with the graph at the time so as I said the front end was getting more and more complex so our subgraph the sub graph code was getting more and more complex we had a lot of entities to manage a lot of handlers uh to implement and to properly handle all each entities change so the subgraph code was going by quite a lot the debugging was quite hard we

definitely did not spend enough time on proper logging but even with that where we had an issue in a deployed subgraph actually replicating the issue locally and being able to fix it uh as fast as we can was quite painful a third point was we were using the hosted service so we were naturally limited by the by the way committing which is perfectly normal and also we were suffering from the from the done time that may happen at the hospitals at the time and finally but most importantly the hosted service was actually getting duplicated so in any case we needed to make a move um so what were we came up with two possible uh decisions the first one was

to actually uh use the graph node of the graph and run it on our side uh so we knew that we were starting from a strong base with actually a nice code that has already a lot of features uh the don't decide that we saw was actually that the code base was in Rust so it's very good for the performance but actually most of our team is heavily oriented towards JavaScript and secondly it doesn't serve very well so needs a front-end size that we had but in any case we needed to build another microservice on the side on that one so the other needs which means managing an infrastructure and really don't want to spend time on that other time the second solution is most important actually it's to redo everything

so we develop everything we maintain everything and the things that we saw in that was to gain the full control that I think we wanted and most importantly the flexibility that we wanted so that's why we decided actually to go for the for the second options and uh not gonna rely a bit of personal challenge I would say um so how is how was it done so in terms of Technologies we are talking about something not very complex we are talking about a simple node.js server we are running we are using less JS with the framework on top of Express and most of the technologies that you will find here are probably the no more technology that you will find in

a backend today you know not just back-end today one important choice is that we kept the graphql API uh in order to limit the migration work that was needed for the inside in terms of timeline so we started the development back in April 2022 we fully replaced the graph in production in September 2022 with the SIM backend and then everything went fine and back in February 2023 we started against development in order to bring the improvements over the API for the V2 of the protocol and some technical improvements a quick word about the architecture so

on your left you will find the networks on which we index so in our case polygon Matic and polygon Mumbai and then you will find all these squares which are actually modules in backend microservice the first one that you will find is the logs indexer which is the most crucial one I would say his job is fairly simple is to read from configuration objects the networks and the addresses that need to be indexed and then this job will be to be in sync with the remote blockchain State and pull the logs and save it inside the local database uh once this is done we will reach the second module which is the local index it is another index and this job is a bit different it will read pull the logs

from the actually the local database that we just saw uh because the logs thanks to the apis in the configuration obtain the events and actually send or dispatch the events to the modules that need it what are these modules we have two uh the first one is notifications module which is in charge of sending a notification based on particular event and the second one is the view so the views will be a set of database tables that are updated according to a set of rules in some handlers so basically what you will find today in in the sub graph an important fact is that there is the last module which is the graphical Gateway so it is not there automatically

derived from the ReUse table as it is done in as it is done in the graph but it's defined separately it will be a set of resolvers that Define their own graphql schema and resolve it by performing arbitrary queries in the views table and also this is the place if you want to aggregate some of chain data or other non-indexed blockchain data it will be the place to do so uh a quick Focus about the two indexes so the two indexes are actually two different implementations of the same underlying State machine and the state machine is a bit the the hard The Logical heart of the backend uh so it's a very simple it has only three states

uh the first one is the sync state so in this stage the state machine will assume to start from a null block and try to reach the latest block once the latest block has been reached the state machine will transit to a new state which is the polling State and in this stage that machine will just try to pull the networks if there is a new block deal with it and go back to the printing state if at some point a training organization happens during this process either in the sync or the pulling State the state machine will go to the third state which is a rewinding State and in this state you will have a Reconciliation between the remote blockchain Network and the local state generally by finding the latest common

block transistor between the two and once it has been done you go back to the sync State and you go again uh so uh finally what were the lessons that we learned during this journey so the first one is that the indexing is by far the most the most critical part of the app it's not only necessarily the biggest but it's just the part that you want to be very careful with um so uh if you have issues in this part of the app generally the issues will propagate in your logs table and if you have issues in your logs table at least in our microservice it will generally stop the process of any new events for example in these days and until you have

solved the issues manually you will be stuck with a fixed time views basically so we are talking about some done times until you have not solved the issue manually which may be very painful with a lot of logs to deal with in our case the success was to add proper monitoring so we make we made sure to include a lot of routines running over and over taking the checking the sanity of the database and checking center of the of the backend microservice and if there are some issues or warning or whatever some errors slack and mail messages are instantly sent in order to you know to alert us as fast as possible

and on our side be able to fix the issue as fast as possible um the second point is that in this kind of service the time is a very important factor so when you will start uh you will have only a few blocks and a few blocks to index everything is very fast very smooth and life is beautiful and as time increases we'll have more blocks more logs and everything we begin to be very slow and at some point you'll be extremely slow and you need to take that into account in your development life cycle so you need to take that into account first in terms of features because at some point we need to be able to take into account improvements performance improvements of your code base or just features that

improve the performance in order to deal with this kind of issue not issue but Improvement and the second one is to take that into account inside your in your development setup if you want to be able to reproduce an issue that is going on on staging or production you need to be able to have a way to reproduce locally uh this this state without the need to re-indexing everything uh so as a kind of conclusion uh rebuilding everything was definitely not was difficult to light decision and it is a long-term investment uh I think what did what we did well was to not rush things and we plan each step quite carefully and as a result now we have a

backend which have a very limited number of issues actually so we are not here uh with a maintenance burden on our hands and uh of course we well it fits very well our needs and we perfectly know how to scale it if we need to scale it so definitely happy with it thanks a lot for your attention and Bon Appetit [Applause] to question first is other services were

going to be shut down did you consider to ask your graph node yourself uh yes so it was one of uh possible uh takeaway uh so we could have done it uh I think what was going to be a issue in our case was uh for the fact that it was a rose code base so in our case we are mostly JavaScript engineers and first it was kind of a problem to rely on a very heavy rascot base and the second point was at the time we did not enough resources and time to properly handle this infrastructure because we needed in any case to have

another microservice and uh yeah we're a bit shy to proceed with uh with an infrastructure so that's why it was a it was summed onside on this kind of decision thank you and second question I was curious about um as the God does not offer the possibility to send notification natively did you consider using the websocket API of the graph and build a notification system that's triggered by stage change uh yes thanks a lot very good uh very good question I think it definitely something that we could have

used and it is definitely in favor of the first solution and for sure it's very interesting to consider uh so for sure now if our needs were simply on the front end and this notification part I think for sure we will it will something that will strongly consider uh just that on our side uh we had a bit of uh a loan on both the data analytics part and the flexibility that we wanted because we well we were moving quite fast and for example uh as of now we need some way to perform some kyc so that's why we ended up with a who is going with the full flexibility

option because uh yeah we in case we needed to add some options but definitely if you only these uh if your needs are well defined and your job is to do a front-end plus notifications then for sure it's it will be actually today very interesting to rely on an existing code base and an existing quality product thank you thanks a lot thanks um having gone down both roads now like building your own and and using the graph it sounds like a classical buy versus build decision how would you if you had to generalize the key decision drivers for like newer projects uh to go

either pick the graph or roll your own like when would you pick either out and I know it depends but if you had to generalize especially uh thanks a lot so I would say in 90k 90 of the cases I will go with the graph definitely like this because it gave you so much in such a little time frame so in our case we at the beginning we just developed the smart contract the front end and we we knew we could rely on the graph we launched production with it it was amazing so uh so definitely at the beginning I think it will be hard for me to say if you are not 100 sure of what you're doing uh go with the graph

because it will save you so much time uh in any case and then you will have to say okay what do I really need actually do I need all these other features or do I have time you know uh if I need other features do I have the skills in order to manage my info and uh then you can take good decisions but uh for me at the beginning if you want to go fast if your goal is to go fast and have something of quality in production real quick it will be hard to not advise the graph on my side then once you have your needs really well targeted you know what you want you know what skills you have in your team then you can take more inform decision I know some people I know a lot

of people that say oh yeah I have the info skill so I don't mind having my own graph node at home and doing my other micro service said not sure it will be perfectly I do worst every day et cetera Etc so very good it was not the case in okay it was not the case for us but that's definitely I think uh what your needs first are that you need to Define and what the team is actually what the skills that you may find in your team um but in general at the beginning you don't want to you don't really know what you want and you don't really have well it depends on the project but you don't necessarily have time you don't really know what other skills of your team maybe I don't know and uh junior so I

would say it's a very quick win and uh to go with the graph thanks thank you thank you thanks a lot foreign [Music]