Episode 246: Adversarial Machine Learning Research with Florian Tramèr

Transcript

welcome to zero knowledge I'm your host Anna Rose in this podcast we will be exploring the latest in zero knowledge research and the decentralized web as well as new paradigms that promise to change the way we interact and transact online this week Turin and I chat with Florian Tamer assistant professor at eth suric we talk about his earlier work focused on side Channel attacks on privacy blockchains and then quickly shift the conversation over to machine learning and adversarial research in that field we Define some key machine learning terms tease out some of the nuances of

ml training and models chat zkml and other privacy environments where ml could be trained and look at why security around machine learning will be incredibly important as these models become increasingly used in our everyday lives this was one of the First episodes I've done looking at machine learning and it was a really great way to kick it off if you do want to hear more of these do let us know now before we start in I want to remind you to check out the ZK whiteboard sessions produced by ZK hack and powered by polygon this is a new series of educational videos that will help you get onboarded into the concepts and terms that we talk about on the ZK front it's a great place to start and you may also want to join the ZK hack Discord to keep up with other members of

the community going through these and other learning resources I also wanted to share there's a new batch of job ads on the ZK jobs board check out open roles at CK and cryptography focused projects like a Leo anoma and the web3 foundation the ZK jobs board is a great place to learn about relevant projects and types of roles that teams are looking for this could be your next opportunity to work in the space I've added the link to both of these in the show notes now I'll ask Tanya to share a little bit about this week's sponsor today's episode is sponsored by Mina protocol with Mina's zero knowledge smart contracts or ZK apps developers can create apps that offer privacy security and verifiability for their users leverage mean on zero knowledge Tech to bridge the gap between

the real world and crypto bring online data easily on chain and access ZK properties from other chains and devices since CK apps are written in typescript developers can easily get started without learning a custom programming language like other ZK protocols head to minaprotocol.com forward slash ZK podcast to learn more about their developer boot camps and open grants that's me enaprotocol.com forward slash ZK podcast so thanks again Mina protocol now here is Anna and tarun's interview with Florian tremor [Music] so today tarun and I are here with Florian Tamer assistant professor at eth Zurich formerly PhD student with Dan Bonnet who also spent one year at Google

research welcome to the show Florian yeah thanks for having me and welcome back to roon excited to be back as usual cool so this episode was inspired by an earlier episode that we did at uh Dev connect in Amsterdam back in April so in that conversation we were talking about I mean we were talking about a lot of things I'll link to that in the show notes but one of the ideas we came up with was like using adversarial modeling in things like peer review or trying to get a more adversarial and dynamic environment in things like Science Biology places where maybe there's not as much adversaries in cryptography and when you think about it like in cryptography that adversaries actually

pushed systems to be better and so yeah we were talking about that so we put that out there as an episode and then pratuche picked it up and tweeted at us basically that we should invite you on Florian to talk about adversarial modeling adversarial research and what that looks like and I know your focus has been much more on machine learning and I think that's what most of this episode is going to be about but why don't we start in a little bit about you sure yeah during my computer science studies I I got very very interested in in cryptography and security they were sort of the the first areas that I was really interested in doing research in and and was ultimately then also what I started off with my PhD with with Dan at Stanford and kind of a bit at random

actually uh I had met Dan at the at a security conference a few months before uh starting my PhD where I was actually presenting a paper that had to do with machine learning security which was sort of a a kind of a one-off project I thought at the time that I had worked on before starting my PhD and talking with Dan he was kind of like oh do you want to continue doing work on on this field for your PhD and I was kind of like okay sure and then we kind of got started on this and there was a it was a pretty exciting time to start working in this area because very very few people were already doing this in the security Community actually I think they were maybe two people who who got a PhD or

maybe three people who got a PhD on machine learning security before me so I was kind of uh very nice to to be kind of at the Forefront of of this field for a while and then actually in in times during my PhD where I got a bit bored with with machine learning and wanted to do something else uh cryptocurrency security was was often my sort of guilty pleasure building off on earlier collaborations I I had from from before my PhD and in particular with Ari Jules at Cornell and so we would often just every couple of months reconnect and and figure out something we could do in the in the cryptocurrency space and so we had some projects there on bug Bounty systems for ethereum

um gas token is something we worked on at some point it's kind of a fun little project with Phil Diane and and Lawrence and then yeah they sort of went on and off over the years and uh maybe the last big project I did in this space couple of years ago was looking at security vulnerabilities in Anonymous cryptocurrency so in Z cash Monero in particular and this is actually how I first started collaborating also with uh with people at um at TTA Zurich where I that I've now joined as a as a professor a few months ago one month ago actually to be precise it's all very new congrats uh yeah thanks and so for the moment I've

kind of I've left cryptocurrency research a little bit behind for for the time being and focusing primarily again on on machine learning security and privacy in the space and yeah that's kind of what my what my research group does at the moment very cool I want to actually explore a little bit of that work you had done on the side Channel attacks and privacy tokens and then I want to dive more into the machine learning side of things so can you walk us through that work yeah so the the background there was at some point I was just thinking you know there's there's all this all this new cryptography that the people are excited about and trying to implement and then there it's very complicated stuff like some people are calling it like Moon

math and I was just kind of thinking like we can't get very simple cryptography right like people are breaking you know like TLS implementations uh every couple of years there must be there must be like things lurking lurking here and so I spent kind of a summer just learning a bit how how zcash primarily uh works and sort of trying to look a bit that whether there could be flaws in the cryptography itself and kind of coming short on that and that I think they actually did a tremendously good job at uh sort of Designing and implementing the the cryptography parts of the protocol and then at some point we kind of got interested in in looking at sort of broader security issues that that one could find in a in a system of of the

scale of of the cryptocurrency and so this got us interested in looking for for side Channel attacks and um the point here is basically to to not try to to break the cryptography directly but to to sort of look for signals that are somewhat outside the threat model that one usually considers from a cryptographic perspective so things like how long do messages take to propagate through the network how long does it take for a node to decrypt certain messages process certain messages and so on and here suddenly this was kind of like a treasure Trove where as soon as you started looking at the system from this perspective there were a bunch of of flaws that became apparent very quickly what was interesting was that

the the reason a lot of those flaws existed in particular in the case of zcash was kind of due to the fact that they had just built on the original Bitcoin design which was not meant to be an anonymous cryptocurrency or like a private setting exactly and so they yeah pseudonymous is is definitely something that Bitcoin was trying to do but they definitely had a much weaker sort of privacy threat model than zcash yeah and so there were a lot of things that Bitcoin nodes do that are perfectly fine within the threat model of Bitcoin but suddenly not fine at all uh once you're moving to to zcash and so concretely one one attack we we discovered was that um so in zcash when

you when I want to send you a transaction I'm essentially gonna encrypt this transaction in such a way that only you can decrypt it and sort of find the coin that get that that is spent now what that means is that I'm just going to send this transaction to everyone in the peer-to-peer Network and so everyone has to kind of decrypt this transaction or sort of try to decrypt this transaction to see if the coin was actually meant for you and then if it is meant for you you're sort of going to do a whole bunch of of extra work just to make sure that the coin is actually spendable and so on um and so what that means is that if you are the recipient of a transaction you're going to be doing more work than if you're not the recipient of a transaction that's kind of Point number

one point number two is that the the Bitcoin client processes transactions one after the other mm-hmm in a single thread of execution and this is essentially all you need for the attack so what that means is that I can send if I find some transaction in the network and I want to know if it's if it's for Anna um I send this this transaction to to Anna's node and I also send her like some I don't know some ping message and then I just wait to see how long it takes for Anna's node to respond to this ping message and if Anna was not the recipient of this transaction she's going to process the transaction extremely quickly and respond to the Ping right after and if she was the recipient of this transaction then it's

going to take her like about a half a second longer to process this transaction and and only after that is she finally going to respond to my message and so this is something that's quite easily detectable even across uh across a network and so these were the type of attacks that we discovered there and um we then extended this to to Monero which has a quite a bit of a different system design but we found similar vulnerabilities uh there where you could also just sort of Time network packets and sort of figure out from this uh kind of what what a node was doing with incoming transactions this this model like the attack here is is it only working because in zcash at the time perhaps

there was like the people who were actually running like to to have an account you had to at least have I think a light client like you can't you're not borrowing services from anyone else so like the node Runner and the address holder could be one in the same because that's what that sounds like you were basically running your your node your peer-to-peer node and your wallet were essentially part of the same of the same process yeah and and so there's very clear sort of concurrency issues between those two very distinct parts of the protocol and that's that's essentially the vulnerability here and this is what zcash then then fix this is part of our of our vulnerability disclosure and I think since then they've also been been sort of striving to to completely

redesign this architecture to to actually really isolate the peer-to-peer node as much as possible from sort of the the much more sensitive wallet component of the of the client had any sort of side Channel attacks been explored in crypto before or were they sort of ignored because everything is transparent and it's fine if if things leak except for in these kind of like private systems yeah that's a very good question from from what I recall I mean we had we had sort of looked through through the literature to see if things like this have been have been talked about before I remember there was a sort of an early Bitcoin discussion at some point about

about some some similar attacks to this and in the case of Bitcoin this is maybe less I mean it's still an attack in that you might know someone's say Bitcoin address uh but you don't know physically who this address belongs to and so you could use the same attack in Bitcoin to essentially connect a Bitcoin address to an IP address this is something that people have talked about at some point in Bitcoin and because of this also kind of decided that hey we should really separate the peer-to-peer and wallet component and so I think by the time that we looked at this this wasn't a vulnerability in Bitcoin anymore it just turned out that

zcash was using the earlier version of the of the Bitcoin client that that was sort of before the discovery of this of this issue and yeah it's kind of the the problem when you're yeah when you then have fixes on on sort of a client but kind of for legacy purposes this this hadn't been integrated into zcash so so actually I think this is actually more of an issue in proof of stake right now um because the validators are selected in advance of them actually being the person validating a block right so I can eat neat too I think it's like you have two to four it's a six minute window uh where you know basically the next

validators until for the next six minutes you know things like tezos you know for like forever the whole epoch so there's basically this thing where in proof of work you you have this like uh adaptivity property which basically means you don't actually know the identity of the you don't know the probability distribution you're sampling from like people can drop out and join in proof of stake you always know the probability of distribution for a certain amount of time so basically you always sort of know the next like some number of validators and basically if you could figure out from their coinbase transactions and the addresses they're sending them to what their IP is you can DDOS them cause them to be griefed I don't think anyone has you know most

of the ways this has been taken advantage of is not actually for ddosing it's usually for bribing the validator so like an avalanche and Solana people definitely pay at validators directly for for front running and most of the reason is because they know who which validator is at which block in the future so it hasn't been done I wouldn't be surprised if basically the same type of thing ends up happening there because people haven't separated these two things yeah this is I think sort of generally a problem as soon as the the entities that are kind of in charge for the consensus mechanism are are sort of easy to know or or you can somewhat guess guess who is going to be a validator yeah you you run the risk that if if those entities are kind of

de-anonymized then yeah you you have these attack vectors I know that actually Dan and and one one of his former PhD students one of my lab mates Saba they had I think this was because of this issue that worked on this on this problem that they called uh secret leader election which was essentially about yeah selecting validators in a way that couldn't reveal who they were yeah I if it didn't need threshold fully hopefully homework at encryption I think it would have been implemented like people have made like proof of concepts of it but yeah I I think none of the production networks unfortunately are very close to implementing it not because they don't know about this they just don't really

have any other options this is something we can get back to when we talk about machine learning where it's unfortunately this is something that's kind of a recurrent theme at the moment that many things would be cool to have but are essentially just unimplementable at them at the moment because the cryptography is is still too slow just interesting talking about Security in this context because like whenever I feel like whenever we talk about Security on the show we are almost always talking about the cryptographic security or the engineering security were almost never talking about the side channel security or these like externalities it kind of made me think like you mentioned sort of the IP

address I know I know you've switched Focus so I don't know if you're if you're following this but like have there been other I don't know inquiries into whether or not IP can be revealed like I do vaguely remember last year I think there was something around like nfts the image pulling to like if you were whole if you had there's definitely malware that took advantage of people signing transactions without paying attention to who or what they were signing which then would eventually transfer your shitty board ape to someone else no no but this one was more about like there they were pulling the image from revealed your IP address and you could be doing that from multiple accounts and it would like link your accounts to each

other I think that was what I and it was like not all the wallets it was sort of the I think it was just linking it was like basically you you have two different accounts with two different images or whatever two different jpegs two different but because you're using it on the same device it would just link those two together I I mean you know people like chain houses and trm 50 of their modeling is not purely on chain stuff it's it's it's external metadata they attach to an address a lot of it IPS that basically like you know if you're a user who goes to coinbase or to binance when they use chain analysis the sort of agreement they have with chain analysis is that

they give them the IP that you logged in from so like let's say I go to binance I say withdraw to X then binance in order to do kyc using chain analysis is like hey we'll sell you the IP data so this already exists it's just not you don't even have to be very sophisticated to get it so so from that from the perspective of like how are we catching these hackers you know like the the Ronin Bridge hack right we recover 30 million yesterday where do you think that came from they came from like someone basically going to an exchange and doxing themselves crazy yeah and I guess you could imagine that people would want to use anonymity tools for things like this so something like connecting to to the cryptocurrency network say over Tor which I remember

that when when we worked on this project with zcash this was actually something we were interested in looking at is like do most zcash users just connect over Tor because this would mean that a lot of these attacks would not be as as useful but I remember that at the time this was actually not easy to do there was some technical reason I don't remember what it was that actually made it hard to combine zcash and and the Tor Network I don't know if this has been fixed since but this is this is sort of where yeah where you have sort of very very different security models that that kind of end up clashing with each other where you have extremely strong cryptographic protections on one hand that give you sort of near perfect sickness sort of privacy within the

transaction pool but then the sort of system side uh privacy is uh is a different matter interesting back then were very few people using the tour set up together with ccash it sounds like yeah I remember that there was it was essentially not not sort of recommended that people do this yeah crazy I wanna now like I think we should move the conversation a little bit over to machine learning and the work that you're doing now you mentioned to me before we started that you had spent a year at Google research when you did that were you focused on like adversarial machine learning research or or something else yeah it was really sort of a continuation of the of the research I had been doing during my my PhD at which

point I had actually started collaborating with with a bunch of people uh at Google already I was actually in a in a research team there that within Google is maybe a little unusual in that it's it's a very it really feels like a like a sort of an academic environment and that's the the sort of the main thing we were doing was really just sort of basic research without necessarily A Google product or or Google data in coming into play so it was it was a very very close to what I was doing before and uh yeah quite a quite an exciting year just because well you get access to a bit more resources than you would as a in in your average academic lab for sure cool yeah I remember I actually first

heard about you doing this stuff right at the beginning of the pen like after SBC uh 2020 where where there was this like Aurora paper and like you and Thomas stanky were like posting something on Twitter about it oh yes in insta hide yeah let's not go there that was a pretty funny online uh kind of thing were you at Google then or were you still no that was before that was my last my last year of PhD yeah but was was one of the one of the projects I did then in collaboration with with people like Google sort of yeah looking at the of the privacy of this of this scheme and sort of breaking some of the Privacy guarantees of it so so the high the high

level thing I guess sorry for for listeners who who haven't heard about this stuff is that there's kind of this idea that people wanted to use very core screened quote cryptography I wouldn't call it it's definitely not cryptography it doesn't really give you any of those guarantees um but people would be like oh like I did a bun I I ran this sort of like hash of like the the machine learning weights you can't really in figure out a reverse engineer you know something about the model or something about the input data I was trained on and that turns out to not be true uh and so I think maybe that's the thing a lot of the theme of some of the things you worked on so yeah and that it's it's kind of there's

there's a lot of hope I guess in in the space that that one would be able to somehow bring ideas from cryptography or sort of the the rigor of of some some of cryptography uh to problems in in machine learning security or privacy and this has so far just proved to be extremely hard to do and I think there's sort of a fundamental reason for this is that in in cryptography we kind of deal with very abstract Concepts we're just sort of trying to you know encrypt a bunch of of numbers a bunch of zeros and ones and um and that's about it in in machine learning is sort of there's there's

actually sort of an underlying natural phenomenon that we're trying to capture and this just doesn't play very well with with cryptography in in general I wanted to actually explore a little bit more about how machine learning training works and then talk about this adversarial research I think this is one of the first times that we actually are talking machine learning on the show we obviously we've mentioned it here and there but tell us a little bit about like how these systems are trained yeah I guess one one way to say it would be to say that no one knows um and that's the point right you're not supposed to know how they're trained because if you did that would be bad I guess no I mean more that we don't we don't necessarily know that well

why they work or but the the sort of the gist is you you take a huge amount of data and nowadays this literally means sort of going to every corner of the internet you can find and just downloading troves and troves of of data and then just having a huge model which is essentially just a function essentially that you're trying to fit to this data and so for example the the big rage at the moment is to train language models so they're what people literally do is just download yeah terabytes or petabytes of data and train a function that's sort of given the beginning of a sentence can sort of predict what's the what's the next word that's going to

come and that's kind of a statistically this is a well-defined problem somewhat and it turns out that yeah if you if you have a function that's kind of big enough that you represent with with enough kind of learnable parameters and you have enough data and you have enough gpus to to sort of optimize this function for like a couple of months you end up with a model that's incredibly good that's sort of predicting what is the next word that comes in a in a sentence and if you then just run this model over and over again it starts generating text that looks somewhat reasonable to humans um I think what's what's most incredible about these machine learning models we

use today about deep learning models is that you you train these models to do this sort of one specific task that's kind of maybe a bit boring but then somehow by doing this the model needs to learn some representation of of or internal representation of what text means what different words mean and so on and this isn't this is really not something that we optimize the model to do literally all we tell the model to do is predict the next word yeah but by doing this the model somehow learns that the word dog and the word cat are similar to each other or man and woman are similar Concepts in in some contexts and the Opposites and in others and so on so forth and this then means that you

can just take one of these models and and then very easily repurpose it for for a whole bunch of of more specific things you might want to do with it so teach it to become a a customer service dialogue agent or to to write poetry or to generate dialogue for for an online game and all these things and so this is kind of where the field is at the moment this is kind of the the Paradigm that's been very very successful in the past few years is that some people train these gigantic models usually on some tasks that isn't particularly interesting on its own and these models are in always trained

by some very usually some pretty big company that just happens to have the resources to do this and then people sort of build on top of this uh and find all kinds of sort of cool applications that they can that they can build on top of of some of these models and um yeah now we we have models that can generate sort of very realistic art uh where again the the sort of way these models are trained is is uh has in a sense nothing to do with with how people use these models in the end all these models are really trained to do is just to take an image to which we've added a bunch of noise a bunch of gaussian noise and remove the noise and it turns out if if you do this long enough the model

kind of just also learns that if you give it only noise it's gonna generate an image and um and so kind of we end up sometimes with these somewhat fascinating and even surprising behaviors that emerge from from something we really didn't optimize these models to do uh all just because they were optimized on billions and billions of of data points to sort of just learn to try and understand some of the internal statistics of of these data sets yeah I think like everything you just described though sounds real great so what is the what's the problem with that like what's the breaking point of this yeah so the the issue is essentially that these models end up being very very

good somewhat on average and then every once in a while they'll they'll just make incredibly stupid mistakes or sort of things that that are completely wrong according to sort of the way we see the world and then very often we we don't really have a good idea or a good sense for why these mistakes occur and how to fix them and so as a as an example this is sort of something that's been observed now about six seven years ago and kind of gave rise to this field that we now call adversarial examples in machine learning was this realization that we could train these models in in this case this was for classifying images so we were training these big machine learning

models where we would just give them pairs of images with with a label so we'd say this is an image of a dog this is an image of a cat this is an image of a train and after doing this a billion times you actually end up with a with a model a function that you can give some new image and it's going to be very very likely to to give you sort of a correct or a reasonable labeling of of what this image represents what what some researchers at Google at the time then discovered is you can take some image let's say the model uh will say well this is an image of a cat and the model is correct and then you change sort of the high order bits of of every pixel in this image in a way that is

like physically impossible for us to even notice uh like that that the image has changed um and you give this this new image to the model and the model will now tell you this is absolutely a picture of guacamole or of an airplane or of a fridge or of anything you want essentially and it's kind of this super surprising phenomenon that that people have have sort of worked very hard to to understand over the years and I think we're we're not quite there yet there's a bunch of sort of competing theories as to why this problem exists and even some some theories that suggest that this

problem might just be inherent of sort of learning from very high dimensional data what's your view from uh theoretical and like cryptography right it's focused on taking arbitrary input stream generate something that's Epsilon close to the uniform distribution right like the entire world is just this little Epsilon ball around the centroid of a probability simplex but machine learning is the opposite right you want to learn this structure that's like extremely non-uniform and like certain parts of the space are sampled significantly more certain parts are zero and obviously as Dimension goes up you're it's even sparser right so do you view it as yeah this sort of the thing you're just saying like is there a

theoretical limit to how well you could ever possibly encrypt such processes for learning not just the output and input but like the actual like mechanics or not because you know I think in the zero knowledge space right now there's certainly a ton of people trying to be like oh we're going to do zkml I'm not convinced that a lot of these things actually don't also leak a metric ton of side Channel information and so so I'm just curious like what's your what's your gut feeling on on like sort of is this like an impossibility theorem lurking around or is there sort of some like hey maybe maybe we can get away with it I yeah I think it depends on what you want to achieve is if if what you're aiming for is some notion of confidentiality kind of the thing that

that crypto cryptography is is best used for then we do have theoretical barely practical is maybe the better word things like like fully homomorphic encryption multi-party computation zero knowledge proofs and so on that you can of course apply to to any algorithm and so you can apply them to a machine learning algorithm and so in in particular what this means is you could very well say I want to train my machine learning model on this data set that is sensitive and so I want to do this in a way where I first encrypt all my data and then I do my entire training with fully homomorphic encryption and end up with uh with an encrypted model and in a way that I've never actually seen the

the data in the clear that is sort of in principle something you could do um in practice very very far from feasible I would say at the moment especially for for sort of very large models that that people might be interested in using today I I guess not even just that though right like in theory even if I did do that but then I could take the I could imagine I could run inference sort of privately so I like have some model I can run the steps of model in a private manner you give me a ckp at the end or fhe the problem is I have the input output pairs if I have any information about the input output pairs that is correlated to some something with execution trace of the model so the question to me is like will we

ever actually get to the point where there's a set of models that it's known that somehow knowing enough input up the amount number of input output pairs you need is just large enough just like the model's dimensionality or not because like somehow that seems to be crucial to like ever ensuring privacy in these things so you mean here kind of privacy of the model itself yes of the model itself yeah so so this was actually the this uh this project I mentioned the first machine learning project I ever worked on before starting my PhD was on on this very topic which we call model stealing attacks where the the point there is to say yeah that if you if you have a model that's you know sitting somewhere behind the cloud API and you want this model to to stay

private maybe because it uses proprietary information or you want to charge people for using it then the fact that you let people query this model is inherently leaking information and at some point with enough queries you can probably reconstruct a local model that's kind of similar to the to the one that is supposed to be hidden and this is inherent because the model is a learnable and so so because the model was learned in the first place well it has to kind of be learnable when interacting with it as well or at least we sort of don't know how to make a machine learning model into into something that cannot be then learned also by someone who interacts with it

whether this is feasible in practice depends a lot on the size of the model so for for somewhat simple model sizes this is this is feasible and usually not particularly expensive for sort of the very large models that people use nowadays so like I know open AIS gpt3 model which they actually this is actually behind an API and people pay for it most likely trying to steal this model and sort of reconstruct the similar model locally by querying the API is going to be way more expensive than just reading open ai's paper that describes how they train this model and sort of spending you know a couple tens

of millions of dollars to to do that instead just to clarify because it sounds like what you've just talked about is when you talk about ckml you're basically talking about either using ZK to protect the data set or using ZK to protect the actual model but can you define when you say the model like is the model predefined is the model something that the researchers are doing or is the model the output of all the data ah yeah sorry that's kind of an issue of terminal analogy which which I I must say probably also caused some confusion in the original Twitter thread you mentioned uh that that kind of led to this where I would say in most of science when people say model or modeling yeah they kind of mean a a

process of kind of I don't know like learning information about the world or whatever or process of doing science or something whereas in machine learning when we say model we mean like uh a function like or a program that that you can run that that was uh yeah so it's kind of the output of the training process people will call a model I see and this is kind of an unfortunate naming scheme that I know that people have complained about quite a bit because it it actually causes confusion when then going to people in in other scientific domains and talking about models but yeah for lack for lack of a better word that's that's what we're using yeah this goes back to this is more a philosophical question about the

notion of epistemology of like do you actually have to put in the features of the thing you want before you try to infer whether reality a base it or do you just say here's what reality does can I go backwards and that forwards and backwards thing is sort of the main problem in philosophy of science so I would say that it's not exactly solved most likely when when we hear about sort of adversarial models or something like that we think of machine learning but it could actually mean something a bit broader which I guess is something we'll we'll come back to later it's funny because I guess for me when I was hearing about sort of the adversarial approach I mean at least the context of this podcast it was actually that crypto itself is so adversarial all the time partly also the economic

incentives yes there's the security break but like if you find a hacker you find a bug there's like true like treasure at the end of the road so there's a lot of motivation for it in the case of ml is there motivation in the same way to break them like what you just described even with the photos it's like who cares if the photo thought it was a cat and then it thinks it's a fridge he he just gave you the greatest example which is the open AI API right like I mean people have made sort of versions of that model because it's you know that's the Crown Jewel foreign this is yeah this is an applicational sort of stealing a model but for for sort of evading a model or sort of making it give bad decisions yeah the

sort of cats to guacamole examples are are cute but ultimately not not very security relevant and this is actually this has been a bit of a of a pet peeve of mine with with this field for for a few years that people sort of always kind of say that oh this is a security concern but they never actually go and look at a real security uh systems where where machine learning is used and the reason is that I mean I'm sure there are many of them it's just that a lot of them are kind of hidden so like as an example online content moderation is something where machine learning is going to be used a lot like every time you upload say a picture to Facebook or Twitter or whatever I'm sure there's a bunch of

machine learning models in the in the background that are gonna take a look at this and and try to flag it if it's inappropriate and it's already happening I mean there's that big example with apple right I think it was they they exactly they had actually seen I think a photo but it was the context was missing and right flagged it yeah and then of course everything that's uh on the on the side of malware detection spam detection intrusion detection all these things have been using and they're using more and more machine learning in them every day and so here of course there's a very clear adversarial setting and very clear incentives for for attackers to try and evade these systems actually the the online content moderation

setting is the one that I think best best illustrates the sort of an attack Vector for something like an adversal example because here the end goal is always to show content to to a human right when you post something online you essentially you want it to be seen by many many people um and so here what you really want is to make sure that you can post say something offensive and that will look offensive to to humans but that the machine learning model is going to say oh this is perfectly fine it's a picture of a fluffy dog or something yeah and so this this is kind of the on the image sides maybe the an easy an easy attack scenario to describe the sort of adversarial setting in in

machine learning today that I would say I'm I'm most concerned about and I think has the most potential for harm is more on the side of attacks on the training data uh this is what we call a poisoning attack again with sort of nice nice illustrative terms I guess in in this field people like like naming things and the concern here would be something like you know uh someone training a model like github's co-pilot that helps to write code and by tampering with the training data you sort of make sure that the model is just gonna write insecure code or something this is actually there was a research project from some people at Cornell a few years ago that uh that showed that you could

do something like this and it's pretty cool or even even just as a sort of a trolling attempt right like these models cost tens of millions of dollars to train and you can imagine the pr nightmare it would be if like I don't know Google spent tens of millions of dollars to train one of these say language models and then once the model is trained it turns out that if you ask it which is the most evil company on Earth it tells you Google because somehow someone managed to put all of these training examples somewhere in the training set and so this is I think a type of attack that I think we're gonna start seeing more of in the future just because there there are sort of incentives to do these kind of things

for for Bad actors and it's probably not that hard to pull off because these models are just trained by literally pulling sort of terabytes of data from everywhere across the web I'm now so curious what your research is because this like what do you do like are you trying to break them or what are you doing a lot of yeah a lot of my a lot of my research has been kind of attack focused with sort of the the end goal in mind of really trying to figure out what what real Attack vectors uh look like and and whether whether some of these kind of generic attacks that that people have have been thinking about for machine learning models for

like the past 10 years or so whether whether they are actually practical whether this is something that someone could could reasonably pull off the end goal obviously being that we could we could also come up with with some reasonable defense ideas but uh it seems like right now you can sort of either look at machine learning models and really in the in the absolute worst case and sort of say like yeah if I give you my machine learning model I tell you exactly how it works it's like trivial to attack can you can you find a defense and this is just a it seems as of now a hopelessly unsolvable problem um unfortunately wow and so you could think of well but then why why don't we see Twitter and Facebook and so on like

constantly under attack and I think the the reason is that while these these systems are not open they're they're hidden from you they're not that easy to interact with there's probably a lot of security for obscurity at play here but it actually it actually works and so I think there's a bit of a something we don't fully understand yet at least from from kind of the academic research side is like how hard is it actually to attack some of these models that that people really use in production and what are kind of you know like pragmatic maybe security things we can do to kind of limit the attack surface of these models that's that's kind of one thing I'm on the side of security of machine learning that I'm I'm really

interested in the other thing which we talked about a bit previously is is all all that has to do with kind of the privacy of training data where there the the big issue actually with with these models that people train today is that they're they're very large and so the models tend to have a very annoying tendency to sometimes just memorize some of the training data that they saw this is again somewhat annoying because this is not something even that that cryptography can solve for you like even if all your data was encrypted and you did fully homomorphic encryption or multi-party computation or whatever if like the output of your computation which is like a machine learning model itself leaks the data that it was trained on well cryptography sort of

says well that's that's fine that's part of the game but with machine learning this this actually happened so we had a project two years ago where we looked at opening eyes gpt2 model this was the the precursor to dpt3 which is actually a fairly small language model by modern standards and we just found that if you if you sort of let this model generate text in some rare instances it's just going to generate you know someone's phone number email address uh fax number uh we actually we had we had one example we found where it really like there was was a specific person who's entire contact information appeared like two three times on the internet in like some random patent documents or

something and the model just knows this person by heart and it's very strange because it's really some some random person uh not someone famous at all and I guess what that gives away is that that data was used in the in the creation of this model yes okay and you could argue well this is this is data that's from the internet it was already sort of publicly available but I think this is a can of worms that's that's about to explode at some point that um we've been all putting like a bunch of data on the internet and without without any consent that this is ever going to be used by a machine learning model and there's now just a bunch of companies that just sort of go and scrape all this information and just say oh it's public information well also

they sell it I mean and then they sell the model and so then there's there's questions of copyright infringement that are interesting as well so yeah these are these I think are are kind of interesting and somewhat worrying also legal questions that are going to pop up in the in the coming years yeah actually so so I guess maybe a more sort of theoretical cool question is you know I guess in the last maybe few years more on the theoretical side people have really cared about what I guess people call ml fairness I feel like it has five million definitions that are in equivalent and I get that people are winning prizes and like best thesis of the year or whatever for it but like I I generally read the papers and I'm not

particularly uh convinced that there's something there but there is this trade-off between whatever notion of ml fairness you have and sort of some notion of privacy do you ever see those getting United because like there is sort of clearly some connection between privacy in and and resistance to adversarial models as well and fairness and there's sort of this triangle between the three of them we don't know the space of that do you ever see those sort of merging and I I guess we could start by trying to Define fairness but like half the definitions I've seen that actually are in equivalent so I like I'm not it's not like differential privacy where we cohere it on one definition that's the main issue right yeah as you said sort of in privacy we have a

definition differential privacy which is a mathematically very very nice I think there's there's settings where it doesn't really capture everything you want under under sort of a privacy label but it's it's sort of a very specific notion of privacy it essentially says whether you yourself contribute your data or not it doesn't really make a difference which is which is very nice to have and then there's other things we care about in in machine learning like yeah fairness as you mentioned and actually robustness which is the the thing we talked about earlier so perturbations to to inputs and so on these are things that are extremely hard to Define what we really want what what sort of our ideal looks like like in the fairness case it's sort of well we want

the model that's fair but probably if you go and ask uh 20 different people and 20 different judges what what that means you're gonna get 40 different definitions and that's kind of the issue is that if you fix a definition so mathematical definition of what you might mean by fairness then you can maybe optimize your model to to try and satisfy this definition this is something that people have done but then someone else might come around and say well my definition of fairness is actually different and by sort of optimizing for your definition of fairness you've made my definition of fairness worse and this actually happens and robustness is uh of these models has the same issue is that when when we sort

of say we want say uh an image model that is robust to sort of small changes of its inputs we also don't know what that means to say small changes like ideally what we would maybe want to say is something like anything that a human would consider as small but this isn't something that we can mathematically Define and so instead people say well I'm going to define a metric that says I can only change every pixel by so and so such a small amount and then you can optimize your model to sort of try to be robust for for this specific definition of robustness to be fair differential privacy still has a metric too for like adjacency yes but which I would say is

more yeah it's great it's more discreet usually so I I there's these are some like minimum distance you that's natural in your problem versus fairness yeah and this is maybe just more in line with something that we really want from a privacy definition and uh yeah but I agree I mean I've actually written at least two papers recently that that sort of call into question whether differential privacy is really sort of the right definition of privacy when it comes to training some of these language machine learning models yeah right right I guess my question is like somehow I feel like there's some sort of maybe this is just because I am bidding crypto too long but like some type of sort of like impossibility theorem where like

you you have fairness robustness privacy you get two out of three or like there's like some way you can't get all of them you can never get three or like you know like if you get all three it says something about your model's predictive power like your predict model's predictive power goes down or something right I think between privacy and fairness there's there's certainly there's certainly trade-offs there this is something that people have already explored even empirically also theoretically where it's kind of inherent yeah that if you if you want the model that's private in the sense that it it can't uh depend sort of too much on on the data of any particular individual and then on top of that you might want the notion of fairness that

sort of very tailored to to individuals well those are clearly at odds and it's kind of impossible to satisfy both of these at the same time more kind of empirically what what people have found is that when you enforce privacy of these models which you're essentially forcing the model to do yes that any time it sort of finds a very small subgroup in the data so some some kind of outlier data you have to ignore or or the model has to sort of ignore this data or it can't possibly be private and then yeah when you then use this model after the fact you find that it its performance on some very very small subsets of the population uh of the data population can be much worse than if you

hadn't tried to make the model private and so those those things are are clearly at odds yeah and so this is something that at some point is a matter of policy of of sort of public perception of which which values which public or societal values we we want to try and enforce in these models and this is a very very hard question because on the one hand first of all we need to agree on sort of what those societal values are and that's kind of a very hard problem on its own and then once we agree on what the values are we sort of have to find a way to formalize this into into mathematics or into code so that we can actually train a machine learning model or write software that sort of abides by

by these rules yeah another way of saying is I don't want the British Parliament voting on whether what definition of fairness I'm legally obligated to optimize in my objective function in some ways like you just don't really want that in some ways but then at the same time there's this like intrinsic sort of weirdness about the fact that some of these models probably should have some notion of being like less biased but like I I'm not sure exactly where where and how they'll go I think a lot of this also just comes from the fact that these these data sets that people currently collect to train these models are for a lack of of better word just complete in that they they

just really go and just collect what whatever data they can find online because it's really sort of the more data the better and I mean if you've spent any time on the internet well you know that whatever you can find on the internet GPT 3 gpt3 trained on 4chan sounds like a way to yes to like kill all humans on average it's not it's not particularly good data I would say and and there's definitely yeah entire portions of the population that are going to be absent or underrepresented in this data and so this is this is a concern and and one that I think requires a lot a lot more work on a lot more research and so finding good ways to curate some of these data sets that

people collect and sort of figure out how to work with yeah with issues of consent of of whose data you you include of trustworthiness like from from which sources are you actually going to pull data so that you don't you your model doesn't just end up being poisoned very easily and then yeah I'll sort of whether the data is representative of kind of the the world you're trying to learn about yeah I think like these are the questions that people in crypto who are like oh we're just gonna like bolt on fhe or ZK to ml don't ask themselves because I think it's more important to solve these first before you even like just throw otherwise you're just like I have a bazooka let me go kill a bunch of squirrels and you're like well why like maybe that's not maybe you should ask

yourself why you built the Bazooka and actually doing it in a private setting it means like it's even less understandable in a weird way like by starting in the transparent one at least maybe you can explore it a little bit more clearly right although technically none of these are really like none of the models used in production like GitHub or open AI or particularly transparent I think that's why stable diffusion is like such a sorry stable diffusion is this open source version of sort of Dolly you could think of as like open source version and like the reason that's just like caught fire is because just like all these people who are like ml's too inaccessible we're suddenly like oh actually I can use it but I always hear like a lot of vitriol from

people in crypto who are like oh like if we just did like all this machine learning in in fhe like we just solve all these problems like the big companies and it's like sure but at the same time I don't really see how you've eliminated any of the inherent biases in these things right like and the biases are probably even worse than the strictly like whether I can infer the model weights like yeah sure like inferring the model weights is probably like a version of like hey I can figure out this person was in this data set but there's sort of some like missing missing thing there and like cryptographers are just like they they love to be vitriolic about big companies but like I feel like they're missing the fact that like just throwing cryptography at the problem doesn't like

help you it's all sort of one one specific part of the problem which if you consider it the problem which which can be a sort of centralization of data uh and and the fact that people people don't have control of their own data and how they want to release it to the world and so on which yeah you you can argue is a one of the big problems with with the way that we do machine learning today but yeah it's it is ultimately a sort of a small part of the privacy and security issues that you're you're gonna have with uh with machine learning models today and so we we'd love to use to be able to Leverage cryptography more so kind of a really fascinating open problem in the space

that I thought about for quite a while I'm sure others have as well and so far there just hasn't been any progress would be to say can we build a model that is more robust to attacks but only only against the tax that are you know like computationally bounded sort of the way we think about attacks in cryptography and in cryptography that's that's always what we do we know that sort of information theoretically there's extremely little we can do so we have to think about computationally bounded attackers and we have a lot of cool techniques that allow us to do that in in machine learning for sort of all these attacks we're talking about here like things like data poisoning or perturbing inputs to evade the model and so on we're always just thinking of

these things in information theoretic terms of like the model is kind of either it resists all attacks or it's broken and we we don't really know how to how to find sort of something in between where we can say the model is not inherently absolutely robust but sort of finding a way to attack it is is hard and expensive and this this would be very very cool if we could do something like this but we we have no idea how so I sort of want to bring this back to what it kind of originally prompted this interview which was this idea of using more adversarial testing or adversarial like modeling for things in science and the one example that we had been

exploring was like could peer review be fixed by creating an adversarial motivated kind of system of and this is not so much the research this is this is like creating adversary like basically Market testing it to get the better you know results so peer review currently being done in universities by people who you know may have their own like research that they sort of want to push through becoming reviewers is what we talked about in that last episode so yeah I don't know is there any sort of research spaces or ways that you're thinking about like changing other parts of science or that you think these adversarial like setups could actually

help yeah if anything I would say are our current peer review system in particular in computer security is already way too adversarial um oh really it's too adversarial yeah it's one of those areas where I think because people just have a bit this this adversarial mindset of like oh everything is broken or can be broken that people's prior on any paper that they're going to read is like this is bad and this is broken and so overall it's actually it's it's sometimes refreshing when you when you go and publish in a in a field other than computer security because suddenly people are a lot a lot nicer and more encouraging and yeah it's uh but does it make for more rigorous like results like does it doesn't it isn't it sort of like

good to be challenged in every way yeah so I think this is this is certainly something that in in cryptography and computer security is is kind of nice that attacks and defenses are both kind of first-class citizens when it comes to research and so you you have this kind of back and forth anyhow and it also means that if you if you want to do research the kind of challenges other research it's it's publishable you just you call it an attack and uh and you publish it whereas my my guess is in if you go into other fields like if you go into biology medicine physics and so on writing a sort of a negative result is is much much harder where you sort of go

and say oh these these other things that people have published in the past don't actually work and so this is certainly something that that is I think very good to have and it's very healthy that that we have this in our in our community it does yeah it sometimes creates a bit adversarial settings also where you have to be a little bit careful not to not to make people angry when or or to be angry yourself when someone comes along and says oh I I broke your work from from like a few years ago so it's uh but then then on the other hand you have the problem in biology where you have this kind of like the Alzheimer's stuff where it took you know it took like 20 years to disprove something but 20 years and you know 100 billion dollars of investment into trying to make drugs if

for the wrong thesis right no and so definitely I think this this should be encouraged that that people sort of do this type of research Merchant in any field that sort of goes and and questions kind of other research that's already been done and sort of reevaluates uh things um this has been actually particularly hard in in machine learning so far because everything is very empirical and so usually what what happens is someone someone will write a paper saying oh we have a new method that is uh more robust more secure or whatever and then a few months later someone comes along and writes a paper saying no actually here's a new attack or here's a better way to evaluate things and

everything is completely broken and so it's a bit like sometimes people say it's a bit like symmetric cryptography in the in the 90s that like everything gets broken left and right then we have sort of no idea what we're doing but it's still it's kind of because of this sort of healthy back and forth uh we do now also have a much much better understanding of which things do not work and clearly do not work which things seem promising and sort of how we should evaluate uh things more rigorously and so yeah I think that in that sense having having this adversarial mindset and this adversal back and forth is definitely something that's that's useful in in science yeah I think the uh the other question that we talked about in the last podcast was

really about like how do you incentivize bad results yeah um and in computer security there's the natural incentive it's like right it is it is viewed as a form of advancement that you've broke something right yeah but in like machine learning or biology there's always this thing of like oh there's like too many people who've like staked their career on like this thing working and we'll do everything possible to make sure that that the gravy train keeps rolling and so I I guess the question was like how do you kind of bridge that Gap where you can like find the nice Middle Ground between those and maybe it doesn't exist but it I think that was sort of the pipe dream yeah it's a good question I think

it connects a lot with just debates people have been having for for years and decades about how we should be measuring impact in in science as a whole right like people have these have these debates all the time about sort of saying oh we really shouldn't be measuring citation counts or number of papers or H index or whatever her because each of them is fundamentally flawed and in the end we just we I agree uh each of these as a metric is is just not particularly good but if we had sort of a gold metric of of what it means to do impactful science and we could just focus on maximizing that and then sort of uh this would make things easier because then whenever a new paper comes out whether it's a positive or A

negative result we could sort of try to to answer the question like does it does it contribute to to the advancement of science to to impactful science and this is just a very very difficult question to answer even when it comes to positive results I mean we have so many examples in cartography and in machine learning and in probably every other field of extremely impactful papers the original zero knowledge paper being a prime example that have just been rejected uh like three four times before being accepted I mean I mean the the greatest example in distributed systems I guess is uh you know lamports part-time Parliament which was the thing that described Byzantine fault tolerance

fully but he wrote it as this paper that was sort of a huge troll he wrote it as if he was an archaeologist who found this Greek civil of ancient Greek civil whose Parliament used uh sort of like a bft mechanism for voting and so like basically every cestron rejected it for 10 years until it had like a thousand citations as like unpublished work and then eventually kind of got accepted oh cool example which at that point you could say well that that's ultimately the the metric people care about so it doesn't really matter whether the probably the paper could have been never published I mean I don't think satoshi's Bitcoin paper was ever published and I think no academic venue would publish it because it's sort of it's not written in

the way that we expect papers to be written the the technical things it it describes are are somewhat vague or and but in the end you you can't deny the huge impact it's had and so yeah it clearly shows that there's there's many different ways to measure measure impact and peer review I think just in general is not is not particularly good at finding this yeah cool Florin thank you so much for coming on the show sharing with us your kind of story some of your work you were doing in cryptography and with privacy tokens all the way to the machine learning stuff so thanks a lot yeah thanks a lot for the very fun uh discussion this was a great time yeah and yeah thanks for introducing my audience and me to kind

of some of the work that is happening over there this is the I think the first time we dig into it so very cool I want to say thank you to the ZK podcast team Henrik Tanya Rachel and Ari who helped on Research this time around and tour listeners thanks for listening foreign