Ben Fielding & Harry Grieve: Gensyn – The Deep Learning Compute Protocol #471

Transcript

foreign the show which talks about the Technologies projects and people driving decentralization in the blockchain revolution I'm Frederica Ernst and today I'm speaking with Harry grief and Ben Fielding the founders of Jensen Jensen is an AI blockchain project that is looking to enable you to buy AI compute in a decentralized manner and we will get to that in just a second before that I will tell you about our sponsor this week though our sponsor

Teddy ho an open source wallet redefining the wallet as a public good with Tally Ho you can safely connect to D5 and web3 with everything you need from metamask plus a lot more you can view your nfts in wallet across the ethereum polygon optimism and arbitrum there's also no need to manually add these networks they already come plugged in teleho has the best edger support around built by a community of developers that listen to users swap between Assets in wallet at a fraction of the price and conveniently view all of your account balances across multiple networks with our new and improved portfolio tab currently they're running a campaign called

dog ssei.gc a layer 2 Adventure that rewards users for exploring the arbitrary ecosystem with telehope from now until December 2nd so hurry up um Bridges Bridge funds to arbitrim with one of their participating Bridges and claim your trusty space dog nft plus we entered into a giveaway for a rare blueberry Club nft head to their blog at blog.telli.cash or their Twitter at telecash for more info tally ho isn't just building a wallet that works Teddy ho is building a wallet web3 can't believe it visit tallyho.org today and to download the wallet and join over 150 000 people in signing

their Community pledge okay fantastic guys hang on let me that let me order you a bit put myself in the middle here fantastic Harry and Ben thank you so much for joining me hey Frederica thanks so much for having us um I think we said this on Twitter but we've both been long time listeners of the epicenter podcast so really pleased to be here I think that's so good to hear Harry and Ben um tell me about uh yourselves what are your backgrounds and what did you do before Jensen sure um so yeah I guess my backgrounds uh in

machine learning research mainly um so I did a PhD in deep learning focused on neural architecture search as a problem which is essentially searching the space of uh deep neural network structures to find one that's kind of most performant for a specific task um so did a PhD in that finished that in 2019 and then moved up and moved into the startup world and co-founded a data privacy startup so I've got quite a kind of strong interest in uh individual data privacy kind of data sovereignty and things like that um did that for a couple of years and then joined an accelerator program in London called the entrepreneur entrepreneur first which is where I met Harry and we kind of went down the rabbit hole of what we're building with

Jensen yeah and on uh on my side my background's in applied econometrics so kind of a fusion of economics and statistics um I was sort of introduced to machine learning during my post grad doing my Master's Degree whilst um while studying econometrics and fell in love with it uh from there uh I just thought it was so cool to be able to essentially quantify everything uh the um the kind of next step for me was leading uh a data research team on an AI startup in London so whereas Ben comes from over kind of technical kind of academic background mine's been worn on the applied side commercially uh got to the point where I really wanted to build

something in the space I saw a lot of issues uh with respect to scaling and um yeah joined the entrepreneur first accelerator met Ben for anyone who doesn't know what the EF or entrepreneur first accelerator is it's been described as a love Island meets Shark Tank so you join as a individual and then you find a co-founder and then they kind of invest in you so it's sort of pre-idea met Ben shared a similar vision for the future of AI uh shared a similar sense of humor uh so yeah the rest is history so it seems like you both come from a fairly extensive Aid planning background um

what what moved you to kind of marry this entire thing with blockchain good question um it wasn't a sort of instant thing uh it happened over over a relatively long period of time to be honest and essentially it was technology driven um so we we knew we wanted to build massive scale uh AI infrastructure um and essentially as we were doing the research to figure out how we could make this the absolute maximum scale we realized that in order to do that you need to have a trustless layer essentially you need to be able to unite uh compute without having to do centralized onboarding of new providers because at that point you end up with an administrative kind of like scaling

limit and we don't want any limits um so we went down the kind of the road of verifiable computation research until we hit that kind of block of that always has to be a trusted third party there has to be this judge or Arbiter when you're checking a computation who makes a kind of con a decision on on whether something's been done correctly um blockchain represents a way to kind of break that and do it by by consensus essentially so a large group of people can do it without having to nominate a single person to make the decision and that was the light bulb moment for us where we said like this has to kind of be the next step for AI to get the scale that we want like planetary AI scale there has to be this kind of consensus layer introduced and blockchain's the

way to do it um before that interestingly we were kind of blockchain Skeptics to an extent uh we hadn't kind of dived into the space before we'd sort of taken the typical technical path of kind of saying a read-only database can do the same thing therefore I won't kind of dive into it but I know for me personally realizing that kind of trust layer was was an absolute liable moment it was when I realized the kind of actual power behind it and got very into the space yeah interestingly Ben and I shared a lot of the kind of ideals that you see Champions kind of in the The Wider decentralization scene so we both were very like large free speech maximalists and we kind of a lot of the a lot of the kind of censorship stuff that we saw

with Snowden and things like that we we bonded over prior to even talking about blockchain so it kind of felt almost like obvious that we should have started in the blockchain space but we didn't interestingly right before making the switch we were trying to do Federated learning which is an area of deep learning where you train lots of models across uh distributed data and then combine them to create a kind of meta model that can learn from all the data sources um and we were doing that with banks so the the the kind of realization for us or for me at least was that there's a much bigger problem with accessing computes um or essentially just the processors on which the models could be trained and to do that you need a decentralized um kind of method of trust and that's

basically a blockchain okay so basically it's kind of the platform and decentralized incentive layer that kind of did it for you uh in terms of that form yeah and in terms of moving this to a blockchain um maybe let's do maybe let's talk about AI first before we kind of go into what Jensen exactly does as a as a blockchain protocol because um most listeners of this podcast will be familiar with blockchain to a certain extent um but AI is not so much our usual um cup of tea so um let's talk about the state of AI today as an outsider it kind of seems like it's totally on fire uh I mean with

gpt3 and GPT for us going to come out soon I think um and then things like Dolly and I mean it just it's completely mind-blowing um can you guys talk about the advances in AI in the last couple of years so absolutely yeah um I think it's interesting being in the kind of AI space and watching this explosion happen because the sort of AI and machine learning space over the past seven years I guess has basically been a series of mini explosions so this one is just kind of the next one in the uh in the sequence but I think to The Wider world it's one of the first times they've seen it actually create real impact and create applications that

people see the value in essentially um but yeah I think deep learning fundamentally has been the big change that's kind of enabled all of this it was when I first started my PhD the Deep learning kind of explosion was just happening uh it just started it just kind of Taken computer vision as an area uh by storm they'd shown that essentially using a deep neural network you could blow away all of the benchmarks set by sort of manual uh computer vision methods in the past so very very I'll try very very briefly computer vision before that used to be kind of manually defining sort of filters over images and then figuring out how to detect lines and things um and then you would have to Define this filter to detect the kind of line

that you're looking for and textures that you're looking for and it's a very manual process deep learning essentially just came on the scene and said we can do all of this straight from the data and that was such a huge change it took away all of that kind of expert knowledge that was required and just allowed somebody with enough compute to design a kind of relatively simple model apply apply it to a very large amount of data and then just have the outcome that they want what we're seeing now is essentially the kind of building on top of that building models that can do even more and then crucially getting them to the consumer or to the developer who doesn't necessarily know the specific problem um that's been going on for for years but Imogen Darley gpt3 Etc of really kind of fast tracked that

um I don't know if you want to speak to uh some of the deep learning stuff as well hurry yeah I think um whenever we kind of talk to Crypts or crowds about it at conferences we we always do a kind of sharpener on the distinction between free terms so the AI machine learning and deep learning because they're used essentially interchangeably but they're quite different and the best way to think about it are a series of kind of like circles which are like a matrushka doll almost where on the outs of the big on the outside you've got Ai and Ai and by the loosest definition possible many people will disagree with this but the loosest definition is it's just programming machine to do something so you know kind of washing machine is in a sense a narrow version of artificial intelligence you tell it to do something

and it kind of programmatically does it um or it works out how to do it machine learning kind of came into the scene much more prominently in the kind of 90s and the kind of early 2000s wherein you instead of having as band set expert systems where you say you know if this then this you use data um to essentially work out the kind of probability with which a certain decision will be made deep learning takes that concept but allows different kind of Concepts to be modeled much more um with much more kind of fidelity so it kind of has hierarchical feature representation which means that the way that the model Works different parts learn different things um if for example if you're the classic

examples if you want to recognize handwritten letters um a neural network typically pushes the the image through lots of different layers each layer will kind of pick up something like a kind of oh this this kind of number has a has a closed loop in it or as a stem and then over time and over lots of kind of computational cycles and lots of tweaking the model will be able to generalize any new image it sees to one of these kind of categories you know a number between zero and nine so um that's basically the distinction between AIML and deep learning deep learning is where you see all the kind of big breakthroughs coming in so all the things you mentioned like GPT free the Ali Etc stuff like stable diffusion all that's deep learning and the story

for deep learning over the past kind of I guess you know I guess since like 2016 2015 has been Transformer models which are a specific type of deep learning model that have been very useful for things like large language modeling um I think what's crucial as well as a kind of more social point is if you told people you know at the beginning of the 2010s that they'd be able to essentially generate a comic book which is in really kind of convincing with really convincing art just from a series of text prompts I honestly don't think most people would believe that's possible particularly at the kind of consumer grade for like a normal person just to be able to type text prompts and create a comic book in the next few years the kind of same

order of magnitude jump is going to happen so in the 2020s the ability to sit down in front of say you know Netflix and instead of picking a movie which has been you know pre-made you simply enter a text prompt and you're like you know I want to see free technologists talking a podcast for an hour you know about about AI or something and with other kind of prompts and maybe like a kind of set of initializations they'll be able to generate an entire movie which you can then kind of steer maybe at different points if you want or as a final point maybe you have the same story but you can change the genre of the story so you could turn something like you know I don't know uh Halloween into like a

sci-fi movie or you could change Jurassic Park it's a love story or something all by changing the using the same script but changing the kind of rendering um lots of exciting things coming in my opinion can can I talk about kind of the Paradigm Shift behind this so basically I mean if you look at like old school programming it's a lot of deterministic of this than that and so on um and in my understanding and admittedly the this is a very lay understanding you kind of you use like some sort of neural network with you know like a complex connectivity and where exactly is I mean so do do people exactly understand how decisions in a neural

network are actually reached is this something that could you kind of um transport this but I mean obviously you don't use real neural networks right so basically everything's in a regular computer you don't have to go to like a bio lab um although that would be not yet I'm not sure whether that would be terrifying or fun but basically everything's anyways in a computer so basically kind of your modeling like in a a different system um that's kind of more interconnected and more flexible maybe you can maybe you can kind of qualify how the system

you're modeling with your regular computers different from just um you know just giving the computers prompts I mean yeah sure I think I think the black box sort of nature of deep learning models is just down to the absolute size of them um at the end of the day you're still tracing a path through a series of kind of decision points in the in the network it's just that that path is absolutely enormous um and it's hard to kind of Link the the weights or the parameters within that model down to uh exactly why they're that sort of value because they've come to that value after being fed millions of samples and you can deterministically you could do that you could track every

single update but the size of data that you would end up generating would be absolutely enormous um I think it's there's sort of two things that I see happening um as we kind of go through this one is the Black Box nature is sort of falling away a little bit as we start to understand more and more about the models that we're building deep learning is a kind of research area has sort of gone through a an interesting fast period where there's been a lot of experimentation that wasn't driven by the sort of fundamentals of the research it was more driven by seeing what we could get out of it so we throw more data at it we try out new architectures and we just see what happens rather than starting from first principles and designing this thing and knowing exactly

how it works so there's been that kind of exciting period where everything has been very black box I think a lot of the gains that happened there are starting to sort of starting to slow down a little bit and we're seeing people revisit those architectures and sort of check and say why does this work so well let's dig into it and let's kind of prove it out so in some ways that kind of curtain is lifting the other thing that's happening which is a bit more controversial I guess is the shift in people's perspectives as to whether a kind of computational system needs to be fully deterministic or whether we can live in a probabilistic world we live in a probabilistic world as people uh that's kind of self-driving cars example is probably the clearest where when we're driving around we accept that

there are kind of stochastic events that happen and that there can be small accidents and there can be issues that happen with a self-driving car system we don't accept that at all and we say that this has to be a fully completely deterministic process I think one of the challenges that the uh the self-driving car industry has had has been an assumption that people would just accept that probabilistic mechanism applied to self-driving cars and they haven't but I think that will change and that's the probably the controversial bit as we as a society go towards actually allowing kind of probabilistic computational systems to exist alongside us um not sure if it will be an easy road but I think it will happen yeah thank you um before we dive into the current

landscape there's one term I have come across often kind of in preparing for this episode also maybe that's a question for Harry because you already um talked about the different kind of machine learning deep learning artificial intelligence so basically there's this time of artificial general intelligence is that different from the the three terms you already talked about yes uh so it's a term which was popularized I Believe by Ben gertzel um who's on AI uh researcher and entrepreneur uh the idea of AGI is similar to also the singularity so it's the idea that you get human level intelligence from a machine uh so you have right now what you might describe

is a kind of like artificial narrow intelligence whereby machines are good at doing certain tasks so for example machines are very very good at detecting certain types of um cancer from from medical scans right so pattern recognition yes yeah uh but kind of scaling that up to general intelligence whereby a machine can be good at doing a task which is kind of maybe simple to humans but actually quite difficult to reflect in a kind of computational like can you give an example space yeah a good example would be a machine uh being able to walk through a crowded area in a smooth way um whilst being able to essentially make kind of discrete assumptions about all

the all the inputs around it it's one of the reasons that I can't believe I can't remember the the level of driverless cars I think it's like maybe level 10 or something it's one of the reasons that driverless cars do really well on the motorway because it's a very kind of it's a kind of it's a problem which the humans might feel quite complex but it's quite like a simple sort of mathematical problem uh because there's not much variation but when you take that same car and you put it kind of in a city street in Rome you know going over cobbles to tourists walking out in front of everything it becomes extremely difficult um so it's kind of yeah some of the stuff which we think is really kind of difficult like being really good at chess is actually quite easy for a machine but some of the stuff that we

think is really easy like being able to kind of walk down the street or you know being able to like I guess certain certain kind of things in conversation like you know understanding uh looking at someone's entire body language and looking at everything that someone's saying and being able to kind of withdraw an emotion from that um there might be she might be good at various things like pose estimation you know how someone's sitting but combining that all together and making a kind of decision is it's quite difficult so um yeah artificial general intelligence basically means uh a model or a set of models or a system which is able to essentially be as good as humans at everyday tasks um critically the kind of Advent of egi leads to artificial super intelligence

because it follows that once a machine is kind of mastered everything a human would reasonably do their rate of kind of marginal Mastery of other tasks moves a lot faster than humans because of as a kind of function of both the kind of complexity of their model and the amount of compute available to them so if we throw all the computers in the world that model which is already at human level it's got much more energy than the kind of normal human does and it's also got an infinite lifespan and it's also got a perfect memory so or your perfect memory so it kind of that's where you kind of get into the realm what kind of Science Fiction horror movies but this is what Elon is afraid of yes you hear it a lot and these kind of examples and you hear also kind of

but one of the kind of Pathways that people or at least I estimate will kind of take us there is the kind of fusion of of you know humans and machines so for example if you have a kind of brain computer interface or a brain machine interface BMI and you're able to essentially augment your lived experience with you know machine kind of inputs that machine learns from all your kind of the way your brain's working and firing it learns patterns you're helping it train it's kind of helping you train your own brain um and that's going to help speed up that process as well it raises a kind of you know Treasure Trove of ethical you know kind of issues um but the yeah that's basically

definition of AGI and then subsequently a um SI artificial super intelligence cool super interesting um so let's let's look at what the landscape currently looks like right so basically say I want to run an AI model um where do I buy AI compute so I mean I could just get an instance on AWS so I could run it on my local machine so kind of walk me through um through the options yeah so it really depends on the scalable model you're training if you're a kind of student learning about AI maybe you're an undergrad you'd typically just use AWS or for small enough models your local machine as you kind of say um the next level up uh you might be a

kind of startup uh you've just burned through your kind of 100K of Amazon credits um and you're kind of looking at the kind of marginal cost of training models um you might go for an on-demand AWS instance you might go for something more kind of fixed and more kind of permanent which is typically cheaper when you have you book them out in advance but the reaches a certain point when you're training models that use a kind of are experiencing enormous cost in AWS or B you can't actually achieve the scale required in terms of gpus so you just get kind of limited by AWS in terms of scale at that point you see companies go in-house so in our in our kind of research prior to raising our last

funding around Ben and I spoke to about 150 machine learning researchers and Engineers a variety of places from kind of foreign companies to startups to Academia and plus a lot of academics at top universities have access to kind of clusters on large kind of high performance compute and people that say Facebook have access to the fair research lab supercluster there which is the biggest AI cluster in the world um most people in our experience didn't manage to get the scale that they that they wanted and one of the ways that some of them kind of dealt with that would be they'd buy gpus themselves and they'd bring them in-house and then they'd manage them and we heard all these horror stories about people like in south of England having a spare

bedroom with a fan in it and loads of gpus it's like a kind of like almost like a Bitcoin mining upstairs and like also people who would have them in their offices and it's a bit of a kind of fragmented Market uh however basically the the bottom line is if you buy the gpus outright typically it costs less marginally over the long term to run them and that's a function of basically not having to pay the sort of 65 ish premium for uh or I should say margin for accessing Amazon ec2 instances um so those that's kind of cloud local or kind of getting your own cluster there's also high performance compute if you're in Academia and you have access to that type of compute but then again there can be bottlenecks there there's

other kind of um options so for example if you're a kind of I guess benevolent organization and you're wanting to solve a highly paralyzable computer science problem a good example that would be like folding at home uh you can you can access it volunteer compute networks using things like blank um from from Berkeley um originally maybe a lot of listeners will remember things like seti at home while it's not you know it's not uh it's not machine learning it's kind of just processing signals um it's a really good example of uh growth Computing reaching very large scale uh I think right now that I'm folding at home which is it's kind of successor has the largest kind

of compute volume anywhere in the world even greater than super computers like fukaku um so yeah to summarize uh you have you kind of go from your local machine onto the cloud maybe via a high performance cluster at University you're at and then ultimately back off the cloud taking it back on-prem the goal of Jensen as a Segway is to give everyone access to the same kind of compute scale that the people who currently have on-prem clusters can achieve and crucially to do so in a way which allows Fair access so kind of it's not it can be turned off by uh by a centralized entity

there have been projects like this in the blockchain space before one of the very very old by blockchain standards projects is called and I believe they actually did their um Ico in 2016 which is basically like 50 years ago and blockchain yes so um how does uh Jensen compare to Gollum yes great question um so we think of it in kind of two axes the first one is the kind of thinness of the protocol so to speak so Golem's a general compute protocol you can do lots of things on it um and we're a thin protocol more similar to like render protocol if you want to kind of um analog there um where we do one thing and that's

training machine learning models um the the second kind of point is on the kind of scalability of the verification so what we've seen a lot the kind of earlier projects is a tendency to use things like reputation or to use kind of less Byzantine tolerant or fault tolerant I should say methods of like replication um and when we looked at those um those kind of those architectures for verification systems we just didn't work for us as people who trained machine learning models we just we wouldn't have enough faith in the results doesn't mean that they don't work it just forever is comfortable purely for kind of machine learning they just weren't suitable and when we had conversations with kind of

web to machine learning people they they kind of agreed so uh for us the goal was to basically take a lot of those initial learnings around how do you kind of position a compute protocol in in this kind of world of crypto world but do it crucially in a way which is only for machine learning so you can make super you know optimizations around the the kind of the speed and the cost of the protocol number one but number two how do you can reach a satisfactory level of verification right now that verification and consensus piece is really like the vast majority of our time and energy you know it's it's the question and uh we had a good initial stab at it with our kind of uh inaugural light paper we've

expanded on it since since then I don't know Ben if you've had anything to that yeah probably just to to emphasize the the kind of general purpose approach that most people before have taken um it's quite an attractive one you want to get the biggest Market you can possibly kind of get to so saying we do general purpose computation any scale any kind of computational problem is attractive at first but you fall so quickly into the two traps Harry mentioned uh the first trap is the verification problem it's very very very difficult um our thesis is that you have to narrow and that we will have a big sort of set of thin protocols at the bottom of the kind of the decentralized infrastructure stack if you if you think about AWS but in web 3 we think all of the kind of

functionality that exists there will be ported over and it'll exist as this sort of hierarchical stack of things getting closer and closer to the user as you go up and on the bottom is protocols like Jensen protocols like render token where you do one specific type of computation really efficiently with really strong verification and then on top of that you can have the kind of general purpose compute networks that fall back onto that so that's our kind of vision for the the decentralized infrastructure I think as part of that when you launch is one of those thin protocols you have a much easier job in initially targeting your market so our Market isn't doing kind of like chess simulations and things like that they're just building machine learning models that's it uh it can be really sort of attractive to say

oh we could just do this extra thing we could do this extra thing maybe we could attach ourselves to an existing sort of thing that's quite popular right now maybe we could generate nfts things like that but I think when you do that you split the Mind share massively in terms of product and people don't know what you are um for Jensen we will always be very clear that we're machine learning compute uh if that's what you want then you come here if you want something else you go to a different protocol maybe it falls back onto Jensen at some point but fundamentally that's all kind of we are and I think the very long term of it is we're behind the scenes we're just like HTTP but for machine learning compute to an end user and a developer you won't even know that Jensen exists all you

know is that the world has changed and now when you train a machine learning model it goes out somewhere and it gets performed by someone in the world through a series of kind of apps and daps and things until it eventually sits on the Jensen protocol and we think that's the kind of best way to provide this compute to the world is via that kind of hierarchical infrastructure where we gradually go more and more behind the scenes essentially I'd had one final point to that which is there's um when we think about the kind of properties that the network has to have it needs to be targeted towards machine learning engineers and researchers it needs to have the verification piece but crucially on the kind of permissionless side it needs to have that that level of sort of censorship resistance but also

kind of an agnostic relationship with Hardware um so in the kind of I guess deep learning Hardware space you know dominated by companies like Nvidia there's companies which are doing their own proprietary Asics like um Google through tpus tensor processing units or graph core another good one with their ipus intelligent townships processing units uh what a kind of trap is I feel which some protocols not even in the kind of deep learning space I've gone down before is shipping like proprietary Hardware so the idea that you you know I think maybe a good example for General compute I've listened to actually the epicenter session with um ICP and Affinity where they have their own boxes

basically and they're sold by them that's actually very attractive to us um the idea that you can essentially ship your own Hardware because then you all the kind of issues you have with sort of you know re-running proofs in a way which is deterministic for hashing Etc a lot of that gets solved but crucially it creates a choke point of centralization um so one of the kind of rabbit holes we've seen some other kind of computer protocols go down is they rely on certain kind of um how would you say kind of uh secure enclaves uh so certain like secure enclaves like Intel sgx where they're like you know we run we can run you know computations for you in a way which are

kind of private but you know they have to use the specific chip which is manufactured by this specific company and you know it's only rentable on these specific services and it just it doesn't hold true to the decentralized ethos in our opinion um it also doesn't scale well currently at least yeah I mean if if you look at what appears to me about Jensen's offering most aesthetic kind of it can use resources that are currently lying fellow and I mean this would not be the case of you actually had to buy a dedicated piece of Hardware to kind of partake in the network nope exactly

um I think like Harry said it's really attractive to go that route from a technical perspective because it's so easy but I think it intersects with one of the biggest things that we think about when designing our verification system which is how what assumptions are we making and how are we constraining the system because essentially we have to make some assumptions and we have to put some constraints in but a constraint like that to us is massive it's huge we don't want to do that unless we absolutely absolutely have to there's other things that we can do we can sort of narrow the space of devices in in a temporary sense or in a permanent sense we can look at certain manufacturers we can look at certain libraries that are that provide determinism and things like that but every time that we make any

decision like that we make it very deliberately and I think it's quite easy to to jump over those in the rush to ship something but if you're going to build the network that we want to build that kind of takes the entire world and turns it into an AI supercomputer you have to be very deliberate about that and maybe it takes slightly longer but you've made it generalized visible and that's the kind of Step change essentially it's it's almost zero or one if you make those assumptions you won't reach that that kind of end State um it's I think it sort of fits on three axes there's product assumptions there's research assumptions and there's technical assumptions and essentially you have to balance all of those things which makes it I think uniquely tricky um you have to have kind of voices of each of them equally kind of valid in

the company and that's something that we've we've focused on quite strongly with with hiring and things like that just making sure that we don't accidentally overweight a certain kind of area I think there's some protocols we've looked at before who've fallen into traps uh there's some traps with research where you can go down a let's make the most formally verifiable system we possibly can and then you never ship anything and then you can go the other route where you make the kind of flashiest thing that an end user will like you ship something really quickly and in previous startup terms that would be fantastic ship it it breaks build it again in the web 3 world not quite as good it breaks isn't just a little thing anymore it's a big problem um so I think it's it's sort of a unique

area web 3 where you have to walk this I think of it like a ridge where there's really attractive looking paths that go down either side but they're not attractive they uh quite quickly drop off the cliff and we're being very careful to stay on that Ridge um before I even to the ins and outs of the protocol itself so um Jensen is its own layer one blockchain um in principle it could have also been built as a Dap on another chain why did you go with the layer one route yeah it was uh it was a big question for us at the start I think like we said the the sort of blockchain world for us was

all about tech um so when we when we entered it we were quite sort of deliberate about it we looked at all of the potential ways we could build it we made a massive list of pros and cons and we kind of navigated through figuring out what the again I guess the constraints and assumptions were for each each one we quite quickly moved from Layer Two to layer one because we wanted the freedom to kind of change things on the the layer one side essentially the consensus mechanism um we didn't want to be constrained by a certain smart contract system we wanted to be able to do as much as we possibly could because we knew this was going to be a big sort of open-ended problem um essentially being a layer one allows us to do a lot more work on the Node side than we would otherwise be able to

do I think if we'd built in the evm which you could absolutely do you could build what we're talking about there you'd be very very constrained by what you can do in solidity essentially whereas building in Rust for us we can do certain things we can fall out and do some machine learning processes maybe we can do some tensor processing things like that that just wouldn't be available to us within the evm it was a I guess in a nutshell it was a future-proofing thing for us we don't want to constrain ourselves early when we don't understand fully why we're making those constraints so we kept it as open as possible and fundamentally we also believe in a multi-chain future we think that the future is true multi-chain it's not sort of ecosystems full of Chains It's individual chains interacting with each other with a kind

of generally agreed messaging protocol I think we've seen some movements through the ecosystem having their own messaging message passing and now we're moving back into kind of General message passing and I think realistically we're seeing the multi-chain future sort of play out so we're quite pleased with that kind of bet so far um so you're um looking at building this as a parachain um why put this on substrate in the polka dot ecosystem so we're not we're not fully certain whether we'll be a power chain or not yet um the substrate decision was essentially the technology um so when we looked at everything we looked at the sort of Frameworks that we

could use in the libraries that existed from a tech perspective just what was nice what had sort of the the best technology built in and substrate came out on top for us um we weren't blockchain people we were machine learning people we came in knowing that we wanted to like stand on the shoulders of giants if you will like we don't want to rebuild consensus from scratch we want to use whatever the best one is and then carry on with building the machine learning stuff that we're focused on and substrate provided that to us as a way to very quickly iterate uh build up the chain and then get on with the off-chain stuff uh with enough flexibility to change it when we need to so the kind of frame subsystem allows us to quickly get something running but then if we need to step in and

completely change it which is really attractive um it's written in Rust where fans of rust as a language um it just kind of made sense from that perspective it's interesting this was a year and a half ago and the kind of two that came out on top were Cosmos and uh substrate and essentially substrate one because of the attack and the kind of nice libraries and the developer tooling and things like that um but yeah the power chain decision is one that we essentially will make later as a bit of a cheat answer uh we can be at parachain we could not be a parachain we don't need to decide right now so essentially we don't um if the ecosystem starts to fill up with things that we can interact with so if there's like storage layers in there if there's sort of sovereign data layers

and things like that that we would want close ties with then maybe it makes sense if they Exist Elsewhere then maybe it makes sense to kind of Bolt IBC on and exist in the wider World um but yeah we'll see you guys should look at Solutions like katisi also so things that kind of allow you to kind of have a um Legacy operating system that kind of hooks into a blockchain for a provable compute um it's uh super interesting I was just gonna say it sounds interesting I've not come across it before but uh yeah we'll definitely check it out I I share the link later

so let's dive records so there's a couple of um but in the Jensen economy there are submitted cybers verifiers and whistleblowers the submitters are the people who actually once worked on um so let's say let's start in the beginning uh let's say I'm a submitter um what kind of AI problems can I submit am I constrained in any way yes currently you constrained by your AI problem has to use gradient-based optimization at some point in the uh in the computational process basically we use portions of the gradient calculations as part of our proof system

that's not necessarily set in stone I think as Harry mentioned earlier we've got our light paper which is public we're iterating on that internally and there are lots of kind of things in play essentially but right now it's gradient-based optimization we use the signals from that as part of the verification mechanism um other than that sorry sorry what does gradient based optimization mean so to me as an AI Noob so how would I know whether a problem falls into that category or not sure um yeah so I guess fundamentally if we think about a neural network um it is a big set of layers that have parameters in them and those parameters are essentially just real numbers

um there could be millions billions now trillions of those numbers in there but fundamentally they are the kind of deciding factor in the output of the network and the training of the network is setting those to realistic values that allow data to go through and Trigger the kind of outputs that you like at the end of the network so you go through lots of sort of matrices layers of these real numbers it sort of it changes the the current input as it's going through and then you get the output that you want by all of those changes that have happened you need to update those numbers to reflect uh the output that you want for a certain input and previously way way back in the day that would be done manually so you maybe not with a neural network but with kind

of certain systems you would set those using expert knowledge and then you would know that when an input goes through you would get the the right output there's also different ways of setting them programmatically so you could imagine a super sort of naive way of just randomly setting all of the parameters running a sample through checking how far away from the realistic sample it is and then just doing random ones again and then doing a random search essentially until you make a smaller error value at the end and then you just keep decreasing that error value you can do other strategies where you do sort of more targeted updates and there's lots of ways that you can do that gradient based optimization talks about essentially what was the big change for neural networks and deep learning which was showing that you

could essentially use the gradient or differentiating the parameters of the layer with respect to the error as you go through the network and you can use the chain rule to apply that all the way back through the hierarchical network that Harry described essentially in that way you get the position on the hill of loss if that makes sense so if you modeled the loss as a in like euclidean space you would see it as this kind of really bumpy area where somewhere there's a big dip and at the bottom is where the loss is really really small and you're trying to find that dip getting the gradients for each layer essentially shows you for that layer where you exist on that surface and what direction you should go in so

you use the the gradient to say hey we've got a massive like Drop here let's go down it so the direction that we want to update the parameters in is this way and we want to update them with this sort of size of Step because this is really steep or it's not steep so we want to make a big jump or a smaller jump and essentially that's it you're just navigating this huge bumpy surface looking for a big dip and the gradients give you sort of a position on that surface so that you know which direction to go in and it was a huge Elite because that signal that direction is is kind of really clearly useful rather than just taking random leaps all over the space and figuring out hey I'm on the top of a hill now or hey I'm at the bottom of a

trench you know where you are you know that you're on the side of a trench or that you're on this weird flat bit and you need to make a big jump to try and get out of the flat bit or something like that how do you know this how do you know there's only one trencher how do you make sure you're in the right trench right because basically if there's lots of trenches you kind of you want to end up in the deepest one you don't want to get stuck on a Motorhead right yeah I mean you want to go to Mount Everest so basically how do you make sure that basically how how do you know how how how you how low you can go all right how how you can go with your model very good question that's one of the big big problems in uh in deep learning

itself essentially there's lots of techniques for for doing that uh the very very simple answer is assume it's convex and then you don't have to think about the being any other yeah but obviously in the real world it doesn't work like that um essentially there's lots of sort of regularization techniques that happen in deep learning training that make it a really complex thing it makes it more of an art than a science because a lot of people have their sort of little tricks that they do um there's things like within learning rate schedules so you'll use a learning rate to set the magnitude of the jump that you'll take in that gradient space but you'll you can use certain schedules to sort of Decay The Learning rate make it smaller over time which means make

smaller and smaller jumps so you don't accidentally jump over a kind of trench but in the same case you can suddenly randomly introduce a huge jump which just allows you to know that maybe if I am in a global minimum maybe I'm if I'm in a tiny trench here in this is a massive one over here I'll just do a huge jump I don't know where I'll end up but it should it could be better if not I'll probably roll back to where I was before so there's lots of techniques like that that are sort of more trial and error than they are sort of deliberate um but like I said before they're becoming more deliberate over time so now that people have introduced these regularization techniques drop out norms and things like that now people are looking back at them and saying hey did

this work for the right reason or was it just some weird random Quirk of the model architecture that made it work here and can we kind of figure out exactly why it works um but yeah it is it comes down to an art more than a science to be honest it can be it can be very frustrating um so now I understand that gradient optimization problems are what I should submit um in terms of I mean can you can you uh can you talk about like real world problems problems and say which ones are gradient optimizations and which one which ones aren't just so I can get like a uh a feeling for what kind of problems I should be able to submit

yeah I mean the simplest way is thinking pretty much every neural network is uses gradient-based optimization um there are other problems that use it as well um but within neural networks all of the kind of big steps that we've seen all the big changes have been neural networks recently so it's a it's a logical place to focus for us whilst also allowing this big space of other places so any optimization problem you could theoretically as long as it's differentiable and can use the chain rule to to flow back it could use gradient-based optimization um and some use other optimization techniques with the grading as a signal in there as long as you're calculating a gradient we can use it it's it's useful um but yeah fundamentally it's all neural networks every maybe two three

years somebody comes out with a paper that says hey we're training neural networks with evolutionary optimization that doesn't use gradients and it's better it's never better it's better in a really constrained system and it never takes hold uh not to say it never will do but so far gradients have managed to stay pretty kind of solidly at the top okay then I turned the question around and what what problems can't I submit so what what what uh problems are not solvable or not well solvable with neural networks good question expanding it the question neural networks in general neural networks are generally quite data hungry algorithms so if you have a have a

problem with very low data volume um a good example of that would be like I I I might be wrong a lot of this but some of the kind of toy examples which are used to teach people to do machine learning like the IRS data set and stuff where you have like a very like you you could literally have a spreadsheet like 100 rows and maybe I think like seven or eight features I don't think that intuitively they're like well suited to to neural networks are typically better handled by like statistical machine learning techniques um so I think data volumes um one of them there's also just um certain certain types of neural networks which is very large so like fitting them on edge devices can be a challenge but in terms of like the actual I guess when

you say problem if you think about the kind of what's the type of thing you're trying to predict in in the world there isn't something which immediately comes to mind that neural networks aren't are like explicitly currently and always will be bad at I don't know if you have any intuition there been I guess you can think of a neural network as a universal function approximator so theoretically it can do all of the things that you would do with other methods I think like Harry said the reason you would not use a neural network would typically be down to data volumes where with a statistical machine learning mechanism a method you could get a better result essentially and then you would you wouldn't train that using

um gradient-based optimization but fundamentally you could do it with a neural network if you wanted to it just might be a bit worse Okay cool so um I understand that I can submit almost any question so basically you say um do I do I ask for an entire program or I mean do I ask for like a dolly kind of output can I say like um I want a picture of accountants in hot air balloons over waterfall and there should also be a rainbow with scorpions on it and it will do that for me or can I ask um I'm building I'm building this car and I need an AI to drive that car can you deliver that AI is that kind of the both within the scope or do one of those

fall out of scope so I guess it's more like you you'd use Jensen to train the model itself so what you would do would be you'd think I want those things I want to be able to create my scorpion rainbow kind of image generator from the text prompt scorpion rainbow and which I love and uh you'd you'd build a model which you know receives a text prompt and then converts that text prompt into into images and then you would have the training data which facilitated the the kind of learning of that model um and then to magenta Network you would submit the model the data and then some hyper parameters which determine you know like Ben mentioned like the learning rate schedule things like that uh I'll do how long you wanted to train

for and then you're kind of um the artifact you received from that training process the kind of the product you get is the train model and then that model can then you can then host that and then you can submit you know rainbow yeah how do I decide which untrained model to use that's a brilliant question I think there's kind of two ways of thinking about it so there's a kind of emerging um and highly kind of popular concept around foundation models currently which is you know you get like a big um company like open AI um or some like mid-journey or something and they build uh um they build them up the base model and then you take the base model with your training data which

might have lots of rainbows and scorpions in it and then you train the model on that um and then the output of that and then that model is very good at you know approximating those outputs that's kind of option one and that's the most common for people who are quite compute restricted which is the kind of theme in the industry just now second option is you build the model but they were they would otherwise build from scratch um and I don't know if you want to talk about that then from our yeah I suppose a lot of our thinking comes down to the foundation models approach because we think it's the kind of the future of the space um my research um back in my PhD was specifically on essentially Auto ml techniques so the idea of allowing somebody to optimize that that model structure and find the

best model structure without necessarily being an expert that's another way of doing it and you've we've seen that sort of happen within like AWS sagemaker for example where they and gcp's compute Cloud as well where they build in some automl techniques to say to a developer you don't need to know the specific machine learning architecture because essentially we can just see that as something that's trainable as well and we apply an optimization technique on top of that Jensen is a protocol can have that if you wanted to we would see that as something that you would build as a Dap that would use Jensen and that dap might Implement a evolutionary optimization technique or something like that it would submit the sort of individual architectures it wants to train and test to the Jensen protocol it

would have them trained and then it would iterate on the structure and it could build up the kind of the model that you want and that's a bit of a theme in the way that we think about Jensen as being purely machine learning compute all of these interesting things that exist around it we would love to see build out as an ecosystem essentially so all of the nice things that you see on sagemaker and gcp we see as being additional things on top I think it could be very attractive to build them yourselves but ultimately it's it's a it's a trap um but yeah on the foundation models we've seen that because of the compute problem like Harry described we've seen people take Foundation models from very large research papers that have they've spent maybe 10 million dollars in funding in order to kind of trial out

all of these different architectures and then they publish a new architecture and say hey this is the best in the world at doing these three computer vision tasks and then you take that you use the vast majority of that pre-trained Network that cost Millions to train you would add some layers on the end you chop some layers off and then you train those layers on a smaller set of data and you have a kind of usable model for that and it generalized loads of information from the first kind of training it did so you call that pre pre-training and then you've got fine tuning and that's very kind of classic in the in the Deep learning space one of the things that we find particularly difficult with that is the bias that gets introduced in that pre-training so one organization doing that on a proprietary data set or on a

data set that they haven't disclosed features about means that when somebody else comes to use it they don't know what's gone on because of those black box issues that you mentioned earlier it it can't kind of you you can't go back in and say why did it make this decision the solution to that in our minds isn't to go fully deterministic and kind of get rid of the black box it's to open it up to everyone and say hey everyone train this Foundation model so design it together we train it together on an infrastructure that nobody owns and at the end of the day we have a model that we can all use that's kind of global and how hasn't necessarily been biased by a specific company's cache of data that they've kept back and they don't want to tell you what's in it and things like

that so once we've got those kind of global Foundation models then anyone can come along and say I'll take I'll find the hash of that model on the Chain I know it's been trained I'll pick that spot and I'll continue training from there on my data set for my problem or task and then I'll have a model that I know at least is as biased as the entire global population rather than being as biased as a company in California okay so basically um until we have the global Foundation model maybe we'll talk we can talk about um how we plan on kind of delivering that later but before we have that I kind of have to decide on one of the commercially available ones and um I've now submitted my my problem

um who gets to work on it uh do servers need to kind of profess some prerequisites and basically is it one server per problem or can you parallelize this yeah so oh do you want to go ahead Harry oh sure I would say um it's at the task level it's one solar per task but a model can break out into lots of different tasks so typically when large language models are trained it's interesting you've kind of been built in a way which maxes out the current Hardware at their time of creation so you know they're designed to fit chunks on you know certain video processors Etc but you imagine a similar thing happening um across the network

It's Complicated by the fact that there's heterogeneous devices on the network but essentially for any given task uh you know A supplier of compute so a verifier or a worker they have their um they have their ability to basically say I'll I'll take that from the mempool and then they are randomly chosen from the pool of people who say that they'd like to take that task uh so everyone can do it if the model and the data can't fit on your device and you've said that you know it can then it follows that there will likely be a penalty there um because it's kind of clogging up the system but um essentially if you're if a task can fit

on your machine then yes your ability to run it is essentially just determined by a verifiably random function which selects you from a sub um a subset of the available miners or workers I should say how do you verify um the what kind of capacities the miners have so busy if I say I have like a 16 core GPU and uh 400 gigabytes of RAM um how do you verify that yeah so it's essentially in the verification of the computation uh they won't be able to do the computation if they don't have that um compute device essentially or that capacity and when they come to submit

their proof when that gets checked it'll be found that they couldn't do the computation okay um there's a little bit of a sort of question there in how big you make a task because you if you made a task an enormous piece of compute then that would kind of be an issue because you could quite easily dos the system by grabbing lots of tasks and saying I can do these never doing them wasting everybody's time and kind of money and things like that so it feeds into that decision exactly um it feeds into that decision on the size of a task and that's there's lots of other things that feed into that there's a parallelization that you mentioned as well and how you split the tasks up into the most optimal structure at the end of the day we're doing a lot of research on figuring out what that

should be based on the constraints when we launch our test net we'll do it based on the kind of practical aspects as well when we see how this actually works in the real world we're very conscious that it's easy to kind of Define this in the kind of perfect system and say yes this is the best size of task and then you go out launch a test net someone does something really weird and you realize that you have to completely change it again so it's part research part let's just see how it functions when we get it out there essentially so basically if a specified model and the training data um how can you make sure that I've actually done the job right because it's very much not deterministic so it's not like

you know you can make me do a hash and then the hash will tell you whether I've done it or not how do you build in checkpoints into this uh into this process because otherwise I could I could just you know pretend to do the work and then kind of you know this was this was a lazy model it kind of didn't do the work it kind of it's maybe stupid I don't know I've done it but it just couldn't be taught yeah um essentially that's the big challenge that verification system um it's a huge Challenge and I think the simplest most secure solution to say is a zero knowledge proof of the entire computation essentially that's sort of what you think about in X years time we

should be able to do any computation as part of a zero knowledge proof and then we can you can check that proof uh to say definitively whether someone's done that computation or not um but right now don't you need a new circuit for each given computation yeah so so right now that's that's the case and it's it's horrible to try and do for machine learning work the computations are massive you need a DSL for defining a circuit with respect to a machine learning computation um it's it's horrible essentially our approach is to have a hybrid between that and a probabilistic mechanism um we've sort of follow some principles uh in a work called proof of learning by

Nicholas paper nuts group um which is it was a paper within the machine learning world that essentially showed that using the path through gradient space that we described before you can sort of create this certificate proof using checkpoints in that space that theoretically it's just as hard to generate a realistic looking path as it is to just do the work and then using a kind of financially rational assumption on all of the participants you can say they would just do the work essentially there were issues with that paper and kind of flaws with the with the ways it things but fundamentally what it showed was using a essentially random auditing scheme on top of a path through gradient space you can build up a relatively

robust check and essentially we take that one step further by introducing zero knowledge proofs at certain steps and on top of the kind of global loss of the model just to add another definitive kind of proof on top we package all of that up within a game theoretic mechanism that looks quite a bit like true bit from from way back in the day with staking and slashing and solving the verifiers dilemma with um like random jackpots essentially with whistleblowers and that's the full system uh although I'm aware that is a just a big word vomit of things so happy to to dig into bits of it yeah yeah maybe they're still they're still unhappy yeah so I think lots of

lots of prove it it's kind of of this uh basically um uh it lets you do um large computations of chain and then basically you can prove it where the binary search on chain of any things is that is that a fair summary you're a muted Harriet this I can't hear you sure I think Harry's got some headset issues um yeah I think essentially that's that's exactly it so true bit proved that you could take a very large computation that wouldn't fit in the evm or would be absolutely massive and really expensive do it off chain and using that challenge mechanism and that search that you described eventually

prove it on chain with the chain doing a tiny operation um we take that same principle but we apply it on top of some of the sort of certificate proof stuff that we mentioned before so if you applied that to a full machine learning training job you'd be searching forever like it's enormous so we distill that down into a kind of a smaller proof that is still representative of the larger I.E rather than doing the full thing you do one in a hundred checkpoints or something like that you've already reduced the size by 100 then you go into that challenge mechanism there's also some work in the machine learning space again which is applied the true bit mechanism to neural networks but rather than using virtual machine instructions for that search use a graph and you Traverse a kind of

Merkle tree graph of a neural network graph essentially of operations and you can do that at different granularities so you can do it at Native operations like within pytorch or tensorflow on a convolution and then you can step into that convolution and do the Matrix multiplications that are involved and then you can step into that matrix multiplication and do the individual kind of floating Point operations that are involved it's quite large overhead so it requires you to do that big reduction before you get to that stage but once you do it provides that crucial link that goes from random off-chain participant to full consensus of the chain with the chain running something and that links back to what we said earlier about being a layer one versus a layer 2 as a layer one we can also

increase the size of computation that the chain can do so if we make the chain do a matrix multiplication and that's okay then we get to kind of skip that step in to a matrix multiplication doing floating points and things which is is quite nice at the end of the day it's it's constraints and assumptions again you increase the hardware the validators and things so you've got to be careful but we like the flexibility in being able to kind of change all those levers and things I I totally get that so I think basically uh kind of uh fixing the the block gas limit and kind of maybe repricing some up codes and I mean obviously it gives you it goes a long way right um maybe let's talk about the the how the

blockchain itself Works in just a bit um but there are two more parties in the process so there's the verifier who actually makes sure that the checkpoints have been checked um and then The Whistleblower who makes who makes sure that the verifier actually operates correctly can you go through what what their respective role itself yeah so the verifier in The Whistleblower have a relationship um similar to the verifier in um on the kind of worker in the tribute paper so essentially uh the whistle board solves the The verifiers Dilemma problem uh which is the idea that you won't necessarily you know verify work unless

you know you can reasonably expect there to be work worth kind of you know catching as being wrong and being rewarded for so uh The Whistleblower essentially checks that the The verifiers Works being correct but it's also incentivized to do so by forced errors from the verifier so the verifier also from the true bit paper right yeah yeah so kind of like the the dog that the baggage carousel where they see if they don't find you know any drugs their handlers put you know like a suitcase with drugs so you know they don't end up depressed and you know still working the dogs need dog treats occasionally so um the that's basically the the kind of the kind of thinking there

um so it follows that basically the solver does the work um if the work um is incorrect the verifier um shows that it's being correct and the list of block and then confirm it's incorrect that then goes back onto the the kind of chain which we can talk to in a moment in terms of you know being verified on chain um but essentially periodically and also kind of the the rate at which is kind of linked to the security of the system I guess the um the verifier will show up an error on purpose to twistleblower which keeps The Whistleblower wanting to be kind of engaged with time as well um if the Whistleblower does find a

problem they play a game a pinpoint protocol where they narrow down he puts it down the computation to a single kind of Point um in the in the kind of I guess you could view like the Merkel tree of computations for that area of the neural network and then that goes to the the chain for um for arbitration that's the kind of the the version of it in kind of a plain way that we originally had as Ben mentioned earlier we've Advanced on it in a couple of areas um after basically closing our RC drowned and doing more research work um but yeah that's the verifier and whistleblower so tell me how that that 15 as a whole so obviously someone

has to build blocks there has to be I assume this is some kind of staking Network so it has to be a staking token and how does all of this fit in with um the Jensen photograph yeah um essentially it's a vanilla to an extent substrate uh blockchain uh we use the proof of stake Grandpa babe consensus mechanism uh validators just kind of doing things in the normal way that they will um and all of the parts that Harry described and I described earlier happen off chain they're all kind of off-chain participants doing portions of work and kind of being incentivized by the fact that they've staked through a kind of a

normal staking palette within substrate but in a smart contract it could just be submitting a certain amount of tokens and then they will be rewarded when that work ultimately gets checked um the kind of game theoretic difficulty here is making sure that all the staking potential slashing amounts and the reward amounts all add up so that there isn't an incentive somewhere for somebody to either be lazy or to do something that is malicious essentially so it gets complicated when you add more participants in the the kind of having the Whistleblower there is a additional participant is annoying because it's over complicated but it's crucial for us given the size of the computations to have it there to assure the honesty of the verifier essentially it's not

certain that we'll always have to have that we do keep thinking about ways that we can potentially remove the Whistleblower there's certain zero knowledge proof techniques that mean that we potentially could but we don't want to get ahead of ourselves essentially so right now it kind of looks like what's described in the light paper but we're chipping away at each bit of it to try and simplify it um we think if you look at the way that other protocols have gone in the past there's a tendency to launch with a complicated system and then once you get it out there realize that you can simplify it and we're expecting to go through that essentially we kind of saw the same thing with polkadot on the fisherman mechanism that sort of got removed after the thing had launched and it was out in live I'd add one other

point there just on the kind of um our augmentation of the vanilla kind of substrate chain uh there's an issue in the verification system as we originally proposed it and also as it currently looks uh for our state-of-the-art whereby if the the data which is being used to perform the initial um kind of support from the solver is removed or made inaccessible halfway through the verification process you reach a kind of standoff because if the verifier can't access the data then there's it follows that they can't verify you have some kind of data availability solution that kind of plugs into it precisely yes so we are we built that in

um to on top of the kind of substrate so we have a proof of availability the POA is kind of what we've kind of dubbed it internally um layer which is Erasure encoded Etc and basically provides what we couldn't find in The Wider kind of storage layer um market and if anyone's listening to this who's building in that space and this does exist I don't honestly be fascinated to see it but essentially a layer wherein you can lock data for a period of time in a way which is pinned unpinnable for about period of time and verified on chain that exists there um it's too expensive on our weave so our

weave is the answer but the the cost for if you think about you know a terabyte of training data being stored Forever on our weave it just doesn't work when the kind of alternative is like you know storing it on S3 so yeah I should I should also caveat it has to be inexpensive um okay um but yes just on that with with our weave um the reason we need it is is for the the data the training data but it's also for some of the intermediate like proof data and that doesn't need to be around for very long uh it could just be 20 seconds while we go through a certain number of block uh block like block releases or something um with our weave we don't need like 200 years of storage of that thing but

that's what we're paying for essentially so if somebody has really short term but with the guarantees of our weave so bringing the price down because it doesn't need to be 200 plus years uh that's what we want basically we just haven't seen it anywhere um you guys should talk to storage rent this would be uh you know this might be a thing yeah it's like a sort of Parma web uh a temporary web a temporary a temporary Perma web um so um I assume that there's gonna be a Jensen token somewhere in this eventually tell me about that yes

um sure so the the Jensen token fundamentally is required for the attack essentially everything we've just described assumes that you have uh this token built in that can be used to stake slash provide rewards Etc and also maintain the consensus of the system itself essentially so use kind of a small inflation amount so payout validators and then be used in that game theoretic mechanism to allow us to kind of guarantee that financially rational kind of assumption over the entire system I think crucially for us that's what it's for and that's like the only thing it's for um we're very very deliberate to say it's a technical thing that we need that we will bring in when we technically

need it and not before um we've seen what's happened with with kind of utility tokens in the past where people have kind of launched them too early and then it's it's a distraction for the team it's a distraction for everyone um people aren't buying it to use it in the system they're buying it for other reasons we we don't want any of that ideally I mean it's easy to say that it's hard to kind of see what will happen in practice but our approach is to essentially delay it as much as possible and then quietly bring it in when it's required to maintain that consensus and pay out the participants with the game theoretic mechanism essentially um yeah it's critical to know as well that along with some other kind of early movers in the Deep learning crypto space we're very much a minority with respect

to the rest of a deep Learning Community um there's at least in our experience quite a lot of skepticism about crypto more broadly um I mean Ben and I is the history is kind of Testament to that you know we were skeptical and we've obviously from a technological and kind of ideological point we we like it I think it's it's the right way however when we initially provide when the network initially launches we anticipate that the majority of deep learning users will pay in in Fiat and it will just simply be swapped into tokens the solvers and the people participating on the supply side of the network will facilitate with tokens and we have huge interest from a lot of the kind of old ethereum one miners who have lots of gpus who want to attach them to

something um but yeah it's crucial that there's a kind of the kind of crypto the scary crypto words like Tolkien are kind of removed from the the end the end deep learning and machine learning users um which is exciting because to us this is one of those use cases which really Bridges the Two Worlds of web 2 and web free you know there's a economic rationale for this existing there's the technology now to enable it its existence um it's now almost like a kind of execution question to a large extent you know how do you just how do you get people comfortable with the idea that they're not using Amazon and you know

there's a kind of variable price kind of concept happening here with the token and how do you obfuscate that as much as possible and crucially how do you obfuscate it in a way which is decentralized because it'd be very easy just to stand out some centralized API front end which you know just yeah it automatically converts the tokens on a centralized exchange somewhere but that becomes that brings its own problems um so yeah I just add that to the part so what does the roadmap look like for you guys there's gonna be a test net early next year here yup test net early next year um it won't be um incentivized uh kind of to our chat earlier and it's more to pick up

um to kind of do two things firstly to kind of battle test some of the some of the tech that we've been building internally and two to kind of get feedback on the usability of it overall um that will be that precedes a kind of incentivized testnet which will be um which yeah Will essentially you'll be able to train models kind of you know in anger on it um the the rate of which we move is is something that we talk about a lot you know we could ship something very very soon which doesn't really give us any meaningful feedback but it kind of looks good because it's like oh you've kind of something and we don't want to fall into that trap because yeah there's

there's been lots of things which have kind of come and gone for us where there have been like incentives to kind of ship something super early so even kind of you know earlier this year there was lots of kind of hype around the idea of doing generative nft arts and we could have kind of provided like an inference solution for that you know really quickly but we decided that it kind of It kind of was out step of our principles you know it doesn't solve the big problem it's not really on the way to solving the big problem it had lots of kind of other things we'd have to build so yeah I guess I'm trying to say is we're we're not like in a kind of immediate rush to we'll release something tomorrow we'd rather release something which is Meaningful which takes time given how fundamental some of

this stuff is particularly the zero knowledge stuff which is you know which is which is pretty involved in terms of time cooler thank you guys where can people go to learn more about Jensen yeah so jensen.ai is your kind of primary source uh we have a Discord uh we don't have a telegram group uh the discords where lots of chat happens we're also hiring just now so if anyone's listening to this and is interested in you know building a permissionless deep learning compute protocol uh then we're we're it uh and I guess moreover next year we should be hosting a zero knowledge machine learning Summit so if anyone's kind of particularly interested in in that kind of crossover uh that there's that and maybe as a final point and

maybe more for traditional kind of deep learning or machine learning people that anyone's listening or sponsoring the the Europe's conference in New Orleans uh next week so we'll bet and I will both be in Louisiana at the conference attending the talks flying the flag for uh crypto um and yeah if anyone's there we'd be more than happy to chat super cool thank you both for coming on this was uh super interesting um we look forward to kind of seeing how this plays out with them the test net and the mainland fantastic appreciate it thanks for having us on it's been a pleasure yeah thanks so much for having us on really enjoyed it great questions as well really interesting perfect thank you