Episode 371 - How ZK inspired AI Watermarking with Miranda Christ

Transcript

[Music] Welcome to Zero Knowledge. I'm your host, Anna Rose. In this podcast, we will be exploring the latest in zero knowledge research and the decentralized web, as well as new paradigms that promise to change the way we interact and transact online. [Music] This week, Trun and I chat with Miranda Christ, a computer science PhD student at Columbia University. We discuss the techniques being developed to watermark AI content. To watermark AI is to have some way of proving that a piece of content was created by an AI model, but

in a way that's imperceptible to the person consuming the content. Miranda shares her research and the research of her peers working on this problem, covering the history of AI watermarking, the architecture of an AI watermarking system, the techniques being explored, the challenges in working with everchanging models, how watermarks can be circumvented, and the cryptographic tools that make stronger watermarks possible. There is a ton of overlap and lots of parallels in these AI watermarking techniques to the ones used in ZK. So, it was fun to see new ways that these techniques that we often cover on the show are making their way into this adjacent field. Now, before we kick off, I just want to point you towards the ZK jobs board. There you will find job postings from top teams

working in ZK. And if you're a team looking to hire, you can also post your jobs there today. We've heard great things from teams who found their perfect hire through this platform, and we hope it can help you as well. Find out more at jobs.zeronowledge.fm. You can find this on our website and I've added a link in the show notes. Now, here is our episode all about the techniques used to watermark AI generated content. Today, Trun and I are here with Miranda Christ, a computer science PhD student at Columbia University. She is part of the theory group and the crypto lab at Columbia as well. Welcome to the show.

>> Thanks for having me. Yeah, I'm excited. >> And welcome, Trun. >> Hey, great to be back. It's been a while. >> Cool. Turn, you have been speaking with Miranda. It was this was a suggestion to have Miranda on. It's the first time I'm meeting you, Miranda, so I'm really excited to learn about the work you've been doing. Could you share a little bit about yourself and the research that you've been working on? Yeah. So, I'm especially happy to be on the show because I feel like I have some background in like cryptocurrency. I guess I just finished the fifth year of my PhD at Colombia and so I've done kind of a lot there. I started off knowing I wanted to do cryptography. So, I've been doing that since the beginning. And early on in my PhD, I focused more on

blockchain cryptography. So, I did an internship at A16Z that was really fun and I would say influential in my research. >> What year is that? >> That was summer 2022. >> Okay, cool. >> And that's kind of when I started to learn about the whole ZK world. Uh, which was very exciting. And then I've always been doing a bit of theoretical cryptography and a little bit more applied theoretical cryptography. I would say after my focus on blockchain, I still do some of it but a little less now. I moved more towards uh watermarking and other more like theory topics. And watermarking is probably the main focus of my research now and I

think it's going to be a bit of the focus of this episode as well. That's very cool. you've written a lot of papers with another guest on the show, Joe Bau, who I guess you worked with, Andre. >> What is sort of like the themes in that research? Because that line of research has been going on for like 3 years plus. So like I feel like you you have like a there's like a chain of research that's built upon itself. So like before talking about watermarking, maybe I'm I'm kind of curious just to give give especially for our listeners who will be more familiar with that type of work, just like a brief description of what you worked on.

>> Yeah. But working with Joe is super fun. Let's see. So our first paper was an impossibility result for stateless blockchains. Basically showing that you can't get better than a certain efficiency. And since then we've been doing I guess lots of things lately. We've been working on randomness beacons. So there, you know, you have a bunch of different parties who want to jointly generate even just one random bit. This is very hard if the majority of these parties are untrusted and super important in blockchain settings for things like leader election or like verifiable lotteryies. And so with Joe, I've been working on doing randomness beacons

using delay functions >> in settings where they're known to be impossible without delay functions. So in this like dishonest majority setting where you trust fewer than half the participants, it's actually like classically without delay functions impossible to have a secure distributed randomness begin. >> And so in one of our papers we showed that delay functions don't just allow you to get around this impossibility, but they're in fact necessary. And so yeah, Joe works a lot on this like timebased cryptography and it's been nice to show some connections between that and randomness beacons. That's cool. Also on the list of research topics, there's differential privacy.

When and like at what point in that thread of research was that kind of coming in? >> Oh yeah, that was kind of separate. Um, okay. So I think the first year of my PhD I took a really interesting course with Steve Bell who's a professor at Colombia and he works on law and security. >> Oh. >> So yeah he does like very cool actually impactful work. I think he's worked with the FTC. And in this class, we had to do a class project. And I think the class was very focused on like security that impacts law and society in a very real way. And so for my project, I chose to work on differential privacy in the

census, which is how I ended up with that paper. So yeah, this came out of like a group project for this class. I think the topic of the class which was like law and anonymity pushed me to move a little more outside my comfort zone which is how I ended up working on differential privacy in the census which was actually really cool. My co-author Sarah and I ended up presenting at nationalmy's meeting with a lot of people from the census bureau. >> So it was cool to see like our work getting to the people who actually you know really run the census. Funny we we have talked about differential privacy on the show a long time ago and Terrun I think it was in the context of the work

you were doing using differential privacy in like DeFi. Do you remember this era? >> Yeah. Yeah. Yeah. That was 2021. It's a long time ago. >> Should we quickly define what what that is again? I I'm a little rusty on on it actually. I think maybe this is a good way to get into a segue to watermarking because I view watermarking as sort of in the middle of proofs in the sense of succinct proofs and differential privacy in some weird way. It's not it's not really it it has some flavor of both of them.

>> Nice. >> But differential privacy basically is just trying to say the following. If I remove some subset of the data, can you tell which subset I removed? And so it tries to make the set nonidentifiable. And usually you do that by adding noise. So you can't tell the difference between removing some subset of the data versus noise. And in some sense what that does is it gives you a sort of like you have a notion of privacy that is preserved under composition but it's not statistically guaranteed like someone with enough compute power and samples can even in polinomial time in a lot of cases brute force >> what set was missing. So it it it doesn't have like the computational hardness guarantees but it's easy to use in practice.

>> Ah I see. Interesting. >> Did I miss anything, Miranda? I'm not sure if >> I would say you get even stronger than computational guarantees. Like you get a statistical guarantee that no matter how much time you spend, you can't tell more than a bounded amount of information about the set that was removed. So like yeah, there's no computational hardness, but I would say that's strengthening the property that you get, which is like even if you're willing to throw more resources than would break cryptography at reconstructing the missing data set, you can't do it with higher than some bounded level of confidence.

>> Yeah. The only reason I I kind of mentioned the computational thing is more in practice when it has been broken like the census data stuff where people were able to reverse engineer where it came from. It's come from like differential hierarchy was like invented because people were like hey you can identify which users contributed data to some particular model based on like the model's predictions. Like this was like the Netflix prize 2010 or 2011. And so you need something where it's compositionally safe where like oh if I use the same data set in some other regime I'm not leaking something. But the problem is the compositional guarantees of differential privacy worsen in some ways like when I multiply or add and from that people have been

able to attack in practice which is I think why it's had a little bit of a harder time >> that's getting in adoption. >> Yeah. The attacks against it have been worse. with like I think I think people have this is kind of why people I think seem less enthusiastic about it lately. But anyway, >> what family of cryptography is differential privacy in? Like what kinds of like things under the hood are actually building differential privacy? >> So I would say it's pretty separate from most of cryptography because there's no computational hardness.

>> Um so it's maybe closer to like statistics. I'm not super familiar anymore, but people study things like what distribution you want to choose the noise from in order to optimize accuracy for your specific task. So like Trun said, they're hiding things about the data by adding noise. And you can think of many ways to choose this noise. And so people will study what distribution is best to choose the noise from depending on what you want to be accurate. For example, if you're computing the median, you might choose noise from a different distribution than if you're computing the mean of the data.

>> Interesting. I'm now really curious how the watermarking work lives somewhere between this and the proof stuff. >> Again, that was my statement. So, maybe maybe Miranda will disagree with with me on that. >> But I guess like good good segue to talk about watermarking. So, you know, the naive version of watermarking just for as like a thought experiment that people kind of start with often is, you know, say you're a teacher, you're teaching a class at a university and of course you have to deal with the fact that 90% of your students are going to like use an LLM of some form to cheat on their homework. Is there a way to detect just from the text whether they cheated or

not in some some like that they use the LLM and asked it generated output and so the idea is like there's some statistical signature you've somehow embedded that can be detected. So that's sort of the setup. So Miranda I guess like maybe place to start is like you know what is a watermark and like what are the properties of a watermark that really make it like a unique object? How should someone who maybe has used public key cryptography think about you know what a watermark is as like sort of a not necessarily a mathematical object but like how they would interface with it. I think a good way to think of a watermark is like a secret key encryption scheme where cipher texts are LLM responses or more generally if you

think about watermarking some other kind of model really the AI generated content is the cipher text. Um but we can maybe focus on LLMs for now just to make it easier to think about concretely. And so the idea is that if you know the secret key and you have a piece of content, you can kind of decrypt to see a secret pattern which tells you whether it's watermarked or not. And that would be in the actual output, right? this isn't like in an attached file or you know like thinking about kind of older watermarking techniques and like music or whatever where it was like actually like code at the start of a MP3 or something you know like this is in the text right because a lot of people are

just copying this text into another place it's not like they have a file that you could check yeah that's a really good point so for text it's especially important that the watermark is embedded in the actual words >> because people are copying and pasting. In other mediums, maybe you can get away with embedding information in the metadata, like if it's really hard to strip the metadata from a video or something. I don't know. >> Okay. >> For text, it really needs to be in the words themselves. Interesting. Maybe also in some of the images because also those are being copied, but I I'm curious. Yeah. What that would even look like. That's cool.

>> Yeah. Continuing in on this, what kind of key or I mean you almost have you what you're describing is like within the text there's a key there's some sort of secret that's been added. It's added maybe in like the number of vowels used with I mean I'm just going to use something random here but like some number of vowels used in the first 10 words or something like that and it's always the same and they'll rearrange text to look like that I'm guessing. What do you use to find that? like what is the thing that you're actually like using to decipher whether or not there is something?

>> Yeah. So, usually there's some kind of rule that everyone agrees on. The vowel one is actually a good example. Yeah. Maybe you would agree that you're going to modify the LLM to always output the same number of vowels in the first three words or something like that. And everyone who knows that rule can now go and take a response and check it. And if it holds, then it's probably watermarked. And one good example that's closer to actual schemes that are being used, I feel like the vow one is not how it works. But it was just that was the one that came in my mind is like maybe it's something like that. Like just for us to to think about it like it doesn't actually affect the words being used. It doesn't actually affect what's being said. It's just like something in the

language part of things. >> Yeah. Yeah. Exactly. You want something that's not going to hurt the quality too much, but that's still like distinctive enough that it wouldn't appear in human generated text >> that consistently. >> Yes. Yeah. Yeah. If you want to share some examples, that would be amazing. Like some real or more real examples. >> Yeah. So, one like really common framework for a scheme is to divide all words into two equal-sized random lists. Call them a green list and a red list. and now modify your LLM to prefer to use words on the green list. And so now LLM generated text might have say 70% green words.

>> Um whereas human generated text will have 50/50 because you chose these lists randomly and to have equal sizes. Human generated text should be split right down the middle. Half red, half green. And so if you know this partition into red and green, this is like the secret rule that you can use to check for the presence of the watermark. >> This makes me wonder like are there already watermarks? I want to hear more examples here, but are there already watermarks? Because like why is it that that some text if you don't change the character that's speaking to you? I feel like generic chat GPT text will often use words that like we don't really use in real life like delve and thrilled. I we use them but we don't use them as

much as it seems to be used here. So yeah, so I guess I should mention that watermarking isn't the only approach to detecting AI generated content. And right now there are some post hawk detectors that work pretty well where post talk detectors are just classifiers that are trained to distinguish between like say chatgbt generated text and human text. And these work okay for now, but they're sort of inherently going to become worse because our goal in developing better LLMs is to make the LLM text seem as close to human text as possible.

>> Yeah. >> So without anyone even trying to fool the post talk detectors, like our goal in training better models is >> to reveal the signals. Yeah. that they're looking for% >> and and arguably all of the the current rage of verifiable rewards and RL models is basically trying to do that, right? Because you're you take a pre-trained model and then you make a detector and you have it play a game with the detector until it is able to fool it effectively. It's like in some ways you can view the it's not exactly that, but there's a version of the world you can view it. So, but actually maybe this gives us a good reason to to take a slight detour, which is what is the history of watermarking? So I like you know for me I I kind of remember there's this you know around the time chat came

out there was this like Scott Aronson post about hey like we need to figure out some way of statistically marking these things and you know it was like a blog post but if you read any watermarking paper everyone cites this Scott Aronson blog post. So it seems like that to me was the initial genesis. But what is the history? Like what is the history of the LLM watermarking industry? I guess at this point I would call it you know like the the set of stuff that people are working on like where did it start? How has it evolved? Where you know and how do we get to where it is now?

>> Okay. From what I remember I saw Scott's blog post first and I think that was nice because it was very easy to read and he gets a wide audience. So, I think a lot of people probably discovered LLM watermarking from Scott's blog post, but it was just a blog post, so it didn't have the full details. I think Kurchin Bower at all were the first to really flesh everything out in a paper. So, that paper gets a lot of attention as well. I'm not sure which came first and I think it's not really known because Scott was working on watermarking at OpenAI so he couldn't share a lot of what he was working on and also I know Google has their

watermark synth ID which if you look back at the patents for the patents were filed around the same time as Scott's blog post and Kersinbower and all but the synth ID paper came out much later like I think within the last So, it's not that clear exactly what the history was, but I think most people found out about watermarking from Scott's blog post or the Kchinbower at all paper, which actually won a best paper award at I think ICML, which also >> 2023, right? Yeah. >> Oh, wow. It's been a while now.

>> Yeah. Know that's the the interesting thing about this Baltimore. It's like new in the sense of 2022 to now, but it's not new in the sense of LLM. It's like, >> yeah, it somehow it came a bit before LLMs got really good. Um, like I remember when I was starting to get interested in watermarking, I tried to actively use LLMs as much as I could, but it it felt like a chore cuz they were pretty bad. But now, of course, they're much better. But yeah, I would say Scott's blog post and Kenbower at all were super influential in the field.

And then also the New York Times ran an article on both of them that was very nice. that had nice like interactive visualization and so you could actually see this red green list idea kind of like animated in a cool way and for me this was really helpful like I think I started talking about watermarking research without having read any papers but because I'd seen this New York Times article I had like a good mental model of how it works so it was actually easy to start thinking about it um so yeah those three things I think really helped the field so that was kind of the beginning of LLM watermarking. And for some reason, LLM watermarking got a lot of attention before other AI generated content. Like now people work on image

watermarking a lot more than they did then. >> Okay. And I'm not >> started just with text. I guess I don't think it started just with text. I think there was a lot of attention on text, but people have been watermarking all kinds of things for a long time. So maybe I should mention even before like Scott's blog posts and Kchinbower at all, people had been watermarking text and images and video for like decades. >> Yeah, true. For DRM stuff and for ID recognition reasons and Yeah. long long ago I worked on like a music recognition software product.

>> Oh wow. >> And it was all about metadata and these watermarks as like before all the this is ancient history. >> Yeah. So there's a lot of like influence from older work that people don't talk about so much. Yeah. For some reason AI generated text was more popular around when Scott's blog post came out, but now images have been getting more attention to and those are actually I would say easier to watermark. >> Um like you get better robustness guarantees. So robustness means that it's hard for someone to remove the watermark if they're really trying.

>> It's easier to hide stuff in images too, right? Yeah. You just have like more to work with. Yeah. >> Yeah. And so there was this initial excitement about LLM watermarking and since then it's taken a couple years for companies to deploy watermarks. So Google has deployed synth ID which watermarks text, images and video I believe. And Amazon I know has an image watermark that actually has a publicly available detector. This is the only one that I'm aware of. All other watermarks, only the company that embeds the watermark can detect. And so, as a user, like you can't see the watermark at all. But Amazon actually has a publicly available

image watermark detector. What about Open AI? Does it have anything? >> Oh, that's very interesting. Yeah. So, there's a Wall Street Journal article about why OpenAI chose not to deploy a watermark. Ah >> um so Scott Aronson worked on watermarking for them and in the end they didn't deploy anything. >> H >> and there are a couple reasons for that. One is they ran a user survey and users didn't want watermarks which I guess is unsurprising. So they didn't deploy a watermark and then also I think it was a big engineering task or there is a watermark but it's really really secret.

That's true. Yeah. So, we could just not even know if they're watermarking, >> right? Nice. I love that the ZK podcast now has conspiracy theories on it. >> I'd love to hear about more examples of types of watermarks. What you said before with the red green example, could you share a few more that you do know of that maybe have been deployed or have been experimented with? So, for text, but maybe also for some of the other mediums you mentioned. >> Sure. I know one image watermark pretty well. This is by my co-author Sam Gun and his colleagues at Berkeley, Shwan Dongja and Don. So this watermark is called an undetectable watermark for

generative image models. >> Um >> but this is a very recent paper, right? Like a couple months ago, one month ago maybe, right? Or is this older? >> At least six months ago. I think it appeared at a conference a few months ago and came out maybe six months before that. Maybe that's where I saw it. >> Yeah, I think it was at ICLR of this year. So, how does it work? It works with these kinds of image models that first sample random latent vector. So, these image models typically have a latent space which represents the higher order features of the images. So you can think of each component in the space as representing something like how bright the image is or how happy it is or how like natury it is something abstract

like that. And so these models will first sample these random features and then have some transformation from these features to an actual image. And they also have a reverse process. So you can take an image and transform it into latent which are typically close to the latence that we're used to generate them. But this transformation isn't perfect. There's some error here. And so this watermark works by embedding a pattern in the latence. So usually they're like random gausian centered at zero. With a watermark, you'll first choose special secret vector of say plus

or minus one. And maybe I'll talk more about how this is chosen later. But then they'll choose the latence to have signs corresponding to this vector. >> So you know if the first component of this vector is one, then the first component will be positive and so on. And so now if you know the secret vector then if you get an image you can transform it back to its latence and compare the signs to the secret vector. >> What's a latent? I actually don't know that term. Is that an image term or is that a vector term?

>> No, that's like a machine learning term for like variables you don't see that are sort of generate like like parameters that might not be interpretable but they like generate the process that's able to put the final in. >> What would an example of something like that be? Is that what you were describing with like the sort of bright happy like that or is it something else? >> I don't really know. I hear people throw this term around. I guess what Terrun said makes sense like with the latent space. It's something that as a user you're not meant to ever see. It's just like an intermediate step in generating the image. So, so I mean like I think the first time I ever ran into this term was like in hidden marov models where like I have a marov chain that I'm you

know I'm sampling a state from this like probability distribution but the thing there's like some small set of variables that changes the my sampling process. So I have my sampling process you know I'm generating the next text word. So it's like the quick fox jumped over the shed. And you could imagine that there's some other variables that change what the distribution of the next word is, right? And and so the oldest text models like from the '9s, I mean there's older than that, but like the ones that were popular in the '90s and 2000s did this type of stuff and culminated in kind of I would call it like the last pre- neural net language model, which was topic models. and and that topic models

are actually and the reason I'm bringing this up is it's actually very similar to the thing you just described. So topic models are the the main algorithms this thing called latent dear slay allocation and LDA what it does is it tries to learn distributions over tokens. So distributions over words that that correspond to clusters. So maybe like one cluster in your words is like music. So all of the words like saxophone, trombone, whatever have probabilities of being chosen that never go below 10%. But then everything else is like 10 the minus 9%. And then that that's that would be called a music topic. And then the video topic might be like camcorder, camera, light, and those are all high

high probability words. And so there were already these kinds of like models where people are like predict next distribution over next word. And the latent part was choosing how you cluster the the different distributions. >> Um >> and I think the idea in neural net land is that the neural net learns some representation like that that is this latent represent. You don't know exactly what it is but it seems to work in terms of like mean squared error or like how well it's performing. But the the usage of the word latent for like background variable that you don't know is kind of like in machine learning has existed for 30 years. But yeah, I think it's the same thing. It's like some set of features of certain types of pixels that

represent like something useful like >> so going back going back to what you said Miranda then it's like something sort of a predecessor to what the image is. You're going backwards to like the step before and it's that where you're kind of testing it for any sort of watermark. >> Exactly. >> Okay. The latence. >> Yeah. And so maybe to complete the description of the scheme, I said you use this vector of signs to embed the watermark and you need this to detect. And now if you actually do exactly that and have a fixed vector of signs, then you know if the first component determine the brightness of the image and you're always choosing that to be

positive, you're significantly skewing the distribution of images that you generate because they'll always be bright. >> So it's affecting the output actually. >> Yeah. So this I think is not great. And there are some image watermarks that really choose like a fixed vector and you know use the same vector of signs to alter all the images. But the insight in this undetectable image watermark paper is they have a way to choose lots of different vectors that look independently random. So they can effectively choose a brand new random vector of signs for each image. Yet, if you have like a master secret key, you can still detect the watermark. So, you can still tell that this random looking

vector comes from the special family of watermarked vectors. And to do this, you need an object called a pseudo random code, which was from my paper with Sam Gun. So yeah, I guess I'm biased in liking this scheme because it's like >> we we are we are going to get to to PRC's because I think to me that's where the connect the like if the thing that looks most like ZK stuff exists is like succinct proofs. Yeah. >> The thing that looks closest to that is this >> PRC. Um >> interesting in the work that you created or you talked about like are there public verifiers? Is there a verifier?

Does it exist? How does it deal with it? Is it like running that like anti- transformation in order to get to that underlying stuff or is it able to kind of cut through and just like figure out what the latents were? >> Yeah, good question. So for these image watermarks that are embedded in the latence, the detector does have to run the reverse procedure. >> Okay, >> which it has some computational costs. It's not as bad as generating an image. >> Okay. Um, and you also need to know something about like what model was used to generate the image because the reverse procedure is tied to the model.

>> So that is one downside of these image watermarks. >> I guess you know we've talked a lot about the kind of setup for watermarks. And I think you know one key thing to the examples that we just talked about is there's clearly some some features we want and some features we don't want out of a watermark. And so there are these kind of like fundamental properties >> almost like soundness in in CK or livveness in consensus. There there's a set of properties that a watermark must have and in order to be sort of like proven like embedding the watermark or verified by whoever is verifying. So maybe could we talk through the sets of properties that people study and and are important. You know, obviously the most

the one that's come up in all the examples so far is some notion of bias of like, hey, by adding in the watermark, I'm biasing, I'm changing the thing generated and I want to limit that somehow. But you know, if if we make this comparison to, you know, sort of soundness, completeness, things like that. What are kind of the properties you want in a watermark? >> I just realized a very explicit connection between these properties and ZK or succinct proof systems, which I didn't realize before. So this is good at a high level. The properties you want from a watermark are one that you can detect it in watermarked content. Uh if you know the secret key or whatever information you detect, it should

actually show up. >> The second property is you want a low false positive rate. So human generated content shouldn't be falsely flagged as watermarked. And then third, you want some kind of quality guarantee. So, you don't want to be harming the quality of the content by embedding this watermark >> by creating it. Yeah. Yeah. This is going back to what you were saying with where it's like you don't want it to impact the outcome. You don't want it to like make something brighter and always bright just because you want to have this this watermark in it. >> Exactly. Yeah. So, quality is super important, especially if companies are going to be deploying watermarks on top of like chat GPT for example. They really don't want to be harming the quality of the text that it outputs.

Yeah. And I would add a fourth property that's also important but I consider to be a bonus property of watermarks which is robustness. So you want the watermark to remain even if an adversary is trying to remove it. So these are the highle properties that we want. But in papers like in my work with Sam gun and or zamir it's called an undetectable watermark for language models. We formalized notions of these properties that we want and these definitions were inspired by cryptography. So the first property we wanted that the watermark is detectable. We defined this as completeness which if you're familiar with succinct proofs this is also a

property of those. Yeah. So this is the property that if the text you're generating has enough entropy meaning like there's enough freedom to embed the watermark then it should actually appear with overwhelming probability and this is you know draws parallels to the proof literature but I would say is a more general cryptographic property like you have this kind of property for encryption as well and you know I said earlier that watermarks are a bit like encryption so there are lots of parallels between these different cryptographic objects. effects.

>> Nice. And the false positive rate we formalized as soundness, which is a property of these proofs as well. It says that content generated independently of the secret watermarking key should be flagged as watermarked with only negligible probability. M and finally we had like I would say our main contribution to this paper was formalizing a quality guarantee which is very difficult because it's hard to even define what quality of text is. >> Um but the way we managed to get around this is we say that a watermark is undetectable or in my mind like the highest quality you can possibly get. if

it's infeasible to distinguish between the watermarked model and the original model. So this says that no computationally bounded algorithm can tell the difference between the original model and the watermarked model >> as long as it doesn't know what the like trick is, right? Cuz if it had the verifier component, then it would see the difference. But without knowing that, it shouldn't see a difference. Exactly. Yeah, that's super important. You need the secret key to sort of separate the power of the detector which you know should be able to see the watermark and the distinguisher which

>> PPT algorithm. Yeah. >> Yeah. Yeah. This can be any probabilistic polinomial time algorithm. So actually the funny thing the other thing about the undetectability definitions is it really resembles like the indistinguishability obfuscation definitions for like >> I can't detect the difference but one kind of at least like when I think about the you know the formal definition in that paper versus like a bunch of kind of like the Sahai type IO papers like the definitions actually look qu even like qualitatively if I look at the text they're like kind of similar which I thought to me was like kind One of the reasons I was like, "Oh, there's something like more formal here than like most LLM stuff." But I had one question that's kind of worth diving

into in my opinion, which is there's a a kind of statistical constraint you need in order to get your watermark to work and that you need a certain minimum entropy. And maybe why don't you walk us through that because cuz actually that's sort of something where it's in order to get these cryptographic like guarantees you still need some statistical property and like yeah it'd be great to kind of like talk through that. >> Yeah. So as I mentioned our completeness guarantee is that if the response has enough entropy then the watermark appears and the reason why you need this entropy requirement is that you can't hope to watermark deterministic responses. So, for example, if I prompt

the LLM to output like I don't know the first hundred digits of pi, there should be no randomness here if the LLM is any good, right? Like the LLM has no choice in what it's outputting. >> And so, it shouldn't be able to embed the watermark here. In fact, like soundness already implies that you can't watermark this text because then like this would be a false positive. M >> so there's some like tension between completeness and soundness which is that you know if you ask the LLM to output a human generated text then this should not be watermarked. Yeah.

>> So, the way that we get around this is you ask that the watermark only appears when there's enough randomness in the response that you actually have some control over this. >> Interesting. >> Um, and so you can like formalize this in terms of an entropy guarantee. And in this paper, I mentioned undetectable watermarks with SAM and or we also prove that you can only hope to watermark if you have a certain amount of entropy. So, we have like a lower bound there as well. Um, and so if you look at like the LLM watermarking literature, usually when they prove detectability, they'll have some kind of entropy bound. You can only detect the watermark in random enough text or like a certain amount of

text too. Like is it sort of or I guess it could be short long as long as it's not human. >> Yeah, people have different like different schemes have slightly different guarantees. For some schemes, you need a long enough text and a high enough rate of entropy throughout the whole length. >> I think for some schemes, you need enough entropy. It doesn't matter how long the text is. It could be like a very short text with highly concentrated like a large amount of entropy in that short text and that would be okay. In the image case, is there also minimums?

Like are there size minimums in that case? because you could also generate like a one pixel white box. [Laughter] Can't hide much in that. I feel I I agree. I'm not familiar with the guarantees they prove for image watermarks. I think in general it's harder to prove things there because you're relying on this >> model to go from the image back to the latence. So it's hard to prove guarantees there. But intuitively there should be some relationship between at least like the size of the latent space and how well you're able to embed. So you know now I think we've we've kind of talked about the properties that a watermark has which kind of resemble

cryptographic properties one might desire obviously with more constraints like you know I can generate a zk proof for any NP program but like here it's like no I actually can only generate the watermarks for some if they have enough entropy right which is like a slight different yeah you know not it's maybe not actually slight it's a non-trivial difference but an important thing that I think I only understood once I read your paper, the the undetectable watermark paper, is there's a sense in which the watermarks that are constructed are sort of they don't damage the model. All they're doing is changing sampling properties of the model a little bit like a blockchain samples randomness to choose a validator.

>> You know, that's a very that's a long jump to to to that. The interesting thing is the way that these models work is is I abstract the LLM to just doing two things predict and distribution over next tokens sample a token and we don't change the predict the next token thing which means I don't mutate I don't adjust the weights of the model I don't change I don't change anything I'm just changing the seed that I'm using to sample the next token >> which that is the thing that's like a blockchain that's the thing that's like a VRF a little bit M >> and so maybe it would be great to kind of talk through this because I think this is the reason watermark seems so much more powerful than like things like ZKML or other private machine learning

where it's like I have to put the entire model into >> private state or I have to mutate the model if I was doing differential privacy >> but here you're only changing the sampling and it's so computationally kind of a very relatively low effort thing and yeah like talking through that and understanding like how this kind of sample mutation leads to these properties that you want. >> Oh yeah, that's a really good point that watermarking is super lightweight and that you don't need to touch the weights of the model at all. I like to think of these watermarking schemes as just modifying the sampling algorithm of the LLM. And what I mean by that is like Trun said, these LLMs have two

components. one is like a neural network that takes as input the response output so far >> and generates a probability distribution over the next token. So this is really the computationally expensive part running this neural network to get this probability distribution. And now there's a very cheap step where you just sample the next token according to this distribution. And it's the sampling step that the watermarks modify. So the red green scheme that I described earlier modifies the sampling step just by increasing the probabilities of the green tokens and decreasing the probabilities of the red ones. So it's not touching the weights of the models at all. It's just changing the sampling

algorithm. >> Interesting. >> Um and so the red green scheme is kind of the simplest scheme you can think of, but even the more complicated schemes only change the sampling process. I didn't realize until you said this that the CKML models that we've talked to though require that the model be changed underneath. >> Well, I also thought of those as like >> it's not that it but you have to evaluate the whole thing in contained in this right? >> This thing is very minimally invasive. I'm like running the model as is. I'm only changing the sampling. >> I don't have to like put you in my enclave. I don't have to recompile your code with the ZK compiler, right? like I'm not touching it at all effectively except for sampling.

>> Interesting. >> This also means that you can watermark with only API access. >> So as long as you have access to these probability distributions, you can do sampling yourself or change sampling yourself to watermark text that's coming from say an API to chat GPT. So I actually did this to get some like sample watermark text for our scheme. I only had API access to Chad GBT, right? Like I couldn't run the whole neural network myself. But even just with this API access, it was enough to change the sampler and watermark Chad GBT generated text. Interesting. This leads me to like ask a little bit about the underlying

models and if like are you are there sort of groups of models where you'd have a very similar watermarking scheme for it and you can actually do what you just did which is like kind of apply it to all of them or do you have to create bespoke watermarking systems for every single one and let's like and maybe there's groups and you can do a few. I'm just kind of curious how that breaks down. >> Yeah. So, one nice thing about these LLM watermarks that just change a sampler is that you can use the same secret watermarking key across any number of models, >> right? So, with the red green scheme for example, you can use the same red green partition for as many models as you want. And since you're just increasing

the probability of the green words, you can do this for all of them and detect in the same way. Just count the number of green words. So this is a very nice property of these sampler based schemes where you can use the same secret key for any number of models which should be nice in practice if they're like updating these models often or have many different models deployed. I guess though there's a distinct like it depends on the medium. So going going back to like text versus images, obviously you're not really able to use the same technique across those, but would like all image generation models be able to be tested the same way that that was described in the undetectable watermark work that you described earlier. So that's a good point about the difference between these text and

image watermarks. for the image watermarks. Even if you're changing the latence in the same way across different models, since you also have this inversion process where you go from the image to the latence, that might be different from model to model, there's sort of nothing you can do to use the same detector for all of them. Like you'll have to use the correct inverter for the model that generated the image. And so even if you're using the same way of biasing the latence, you're going to have to use the model specific inverter.

Now there might be ways to like design global inverters that work for all of them. I'm not so familiar with this area, but it seems much harder to have like a an across model image watermark compared to a text watermark. >> Yeah. I mean, I think one of the reasons I was I kind of ended up learning about this literature was I basically was looking at how people in cryptography land were trying to do things and like all of them just seemed crazy from a computational standpoint and then I saw the watermark stuff. I'm like this is like no computation and you're getting like 90% of the benefits that people wanted from like zking the whole model or whatever. And so well you seek again I don't mean zero knowledge I mean

succinctness there >> we know we know >> uh but an interesting thing that I think I needed to read a bunch of these papers to understand was sort of the threat model and like how we think about adversaries attacking and like forging watermarks or trying to to make something that doesn't watermark detect as positive >> and I think it would be great to talk a little bit about the robustness properties and like how you formulate the threat model for attacks and like maybe now we can talk about the emoji attack and and maybe in order to describe the threat model it would be good to talk about concrete algorithms such as hashbased watermarks like your paper with Zamir and gun.

>> Yeah. So okay I mentioned this red green scheme a lot and one problem with this is that if a certain topic is on the green list it'll be talked about a lot more than it should be maybe. And so ideally you don't want to use the same red green list for all responses because this will systematically change what topics the model is talking about. >> And so one idea used in both Scott scheme and the Kchinbower at all scheme as well as mine with Sam and or is to use a hash function or in our case like a pseudo random function to derive randomness to effectively get a new red

green list per response. So the idea is to take the last few words generated by the model and evaluate some kind of hash on these to determine what should be in the red list and the green list for the next word or token that the model outputs. So maybe this is similar in spirit to like fiat shamir if you're familiar with uh cryptographic proof systems where you know you're taking a hash function on the transcript so far to derive randomness for what you're doing next in the protocol >> in the creation of this watermark basically.

>> Yeah. So maybe as a concrete example if the model has output like as a large language model you want to know which word to output next. This family of hashbased schemes would compute the hash of as a large language model derive some randomness which it uses to choose the red green partition for the next word in the response. And now to detect you need to be able to recomputee the hashes to derive this randomness to tell what the red green slits were. >> Okay. So it makes the verification process a little bit more complex.

>> Yeah. >> You have an extra step at least. >> Yeah. There's this extra step which is still computationally cheap >> but where it really hurts you is in robustness. So now if any word in this hashed portion changes then you can't recover the same randomness as before. >> Oh yeah. >> And you'll get a completely different red green partition. >> Yikes. So you can't it can't change. >> Yeah. Basically I mean it depends how much you hash but if you hash say a paragraph then this whole paragraph can't change at all. Otherwise you wouldn't be able to detect the watermark. And now Scott's scheme and the Christian Ber at all scheme hash something like the past two words so

they don't take a huge hit to robustness but my scheme we hash a much longer portion. This is to get this extra undetectability highquality property. >> Interesting. >> Um so yeah these hashbased schemes are very prominent and they motivate what's called the emoji attack. And in the emoji attack, you ask the model to output your response, but insert an emoji between every pair of words. And so now you'll get something that's like word emoji, word emoji, right? And you can easily get back an actual text by just deleting all of the emojis.

>> Mhm. >> What this has done is now if the watermark is embedded by hashing >> Oh, yeah. Will it also totally adjust everything because it has all these emojis in it? >> Yeah, exactly. Like if the emojis are used in the hashes now once you delete them the hashes are entirely different. >> Oh wow. >> And also you're not hurting quality by doing this attack at all >> because you get back exactly the same text you would have had without emojis. >> H wait I don't know if I fully understood this. Like are the emojis being added as part of this watermarking that you then like add and remove or is it if an adversary tried to like >> No. No. Suppose suppose you tried to cheat on your homework and you said,

"Hey, actually take the response you just gave me and replace all the spaces with emojis >> like within the model." So they're doing >> with the model and so now the now it's watermarked with the emojis, but then you when you turn in your homework, you just like go and manually delete all the emojis and put spaces and now you've ruined the hashing, right? Because >> so it's like it's like a very simple it's like a the first attack a kindergarter would come up with in some ways right of like >> yeah it's like super simple and completely defeats these hashbased schemes even the ones that only hash save the past two tokens because you have emojis everywhere. Could you add another quality to these hashbased ones that if they see sort of that kind of pattern of like just dumb obvious

injections that it doesn't hash that part? >> Yeah. So you can I think always play this kind of cat-and- mouse game where if you see a specific attack then you can you know add some step to the watermark that ignores things that are activating the attack. But I think if you fix even a broad class of emoji like attacks, people will come up with new attacks. >> Um, so it's very hard to play this game. In fact, in the limit, there is a paper by Jang at all group at Harvard called watermarks in the sand which shows that it's impossible to construct a watermark that's robust to an arbitrary polinomial time adversary. M

>> so they show that if the adversary is dedicated enough then it can remove any watermark. This is under some fairly strong assumptions. I would say it's more of a theory paper. It doesn't give practical attacks we need to worry about now I would say but what it does show is you can't have a strong robustness theorem like you can't say here's a scheme and prove that it's robust to polomial time adversaries that are trying to remove the watermark. Yeah, I mean it is interesting you made this analogy to fiat because I think about fatir as like hey random oracle model it's safe but oh like all of the recent papers that are like oh it's like kind of borked under some sort of like

something not quite like the robustness thing but like under slight pertabations okay there's not correlation inractability or whatever like fatir is out the window you know we talked a little bit about the robustness of schemes and one really I think the thing that kind of nerds sniped me personally a about this was the connection to like error correcting codes and sort of this idea that like I can error I can have these like error correcting codes that have randomness in them and if I you know make my secret like the path of the set of randomness I choose then I can get these kind of guarantees for watermarks and also there's sort of some sense in which I can handle additions and deletions up to some amount like you

know someone deleting one of the characters right like which is a form of robustness. And there have been other works obviously after yours, but they all seem to build build off this kind of abstraction you came up with for thinking about watermarks and their relation to coding theory and error correcting codes and kind of this natural embedding. So would be great to kind of talk about the relationship between watermarks and error correcting codes and sort of like the the way that once you go to error correcting code land you can suddenly talk about okay like you can you know this watermark is resistant up to some number of deletions in the text or some number of additions and you know to me that seems to be like the way people are kind of moving towards robustness unless I'm missing

something else. So, >> does that actually solve for the emoji attack too >> if you're using error correcting codes instead of the simple hashbased one? >> Yeah. >> Oh, cool. >> Yeah, the error correcting codes are super powerful. And this is like the favorite work that I've done in this area was the introduction of something called pseudo random error correcting codes. This was a paper with Sam Gun. And then I guess we have a newer follow-up paper ideal pseudo random codes that's maybe more theoretical.

>> That's the stock paper, right? >> Yeah. That just appeared at stock of this year that was also joint with Omar Alabia, Pvenan and Ygiodus. So yeah, what are these pseudonym codes? I guess they're exactly what they sound like. They're like error correcting codes that appear random. So usually error correcting codes work by introducing a lot of structure. >> Like one good error correcting code would just be to repeat the message over and over again. But of course this adds a lot of structure and isn't pseudo random. Pseudorandom codes are hard to construct because the code words need to appear random. So they still need to

have the structure so you can recover the message after errors are imposed. But somehow the structure needs to be hidden so that they still appear random. H this is what a pseudo random code is and at a glance there might not be an obvious connection to watermarking but actually Sam and I introduced them because of watermarking and kind of because of these issues with the hashbased schemes. So remember what we wanted from the hashbased schemes was to basically generate fresh red green lists on the fly.

>> Yeah. where we were deriving these lists from the hashes of previous tokens, but now we're hurting robustness because we need to recover the hashes to know what list we chose. And the minute anything changes or is injected, you're sort of screwed. >> Exactly. Yeah. And so we started thinking that instead of deriving the randomness from hashes, what if you could somehow just sample a different red green list for each token and forget about them? And now during detection, somehow magically if enough of the tokens are green when they should be without even knowing the lists, you can use a secret key to detect.

>> Um, and this is kind of like an error correcting code where you can think of sampling >> code word that tells you what the red green lists are for each token in the response. And now forget about the code word, but if you get back a response, you can do this like robust decoding to get back the message. Anyway, this is like super fuzzy, but this is the idea behind them. >> It's the sampling aspect and you're sort of accepting that like it won't like you're still looking for a pattern, but you're not looking as like direct. It's almost like the hash one is so hard.

It's like so finite. Whereas this is like a pro I guess it's a probability like you're still looking for like a probability of these green words or green list words or something. >> Yeah. Before in detection you were kind of asking is this response close to this specific sequence whereas now with pseudo random codes you ask like is this response close to any code word in this large family. >> Wow. And this sounds like a harder question, but we use the power of error correcting codes to have an efficient way of telling whether a given response is close to the code. In in that case with something like that, if you still like would there be almost like a

threshold of transformation of the end content at which it's like sort of like a cut off. Like if it's under this much change, it will still be recognized as watermarked, but if you really alter it past that, then it might get lost. Like could you lose the power of this error correcting code if it gets too different? Yeah, definitely. And this is kind of inherent. So these error correcting codes usually have some distance property that tells you how many errors you can tolerate. So we have something analogous which says that I guess our our detection guarantees are a little bit different from those in

coding theory because we're in like a computationally bounded setting and we have some randomness. But at a high level our guarantee says if the response is close enough then it will still be detected. If it's beyond some bound then we have no guarantee at all. This is also again where I think the ZK thing comes in because it reminds me a lot of Fry. >> Oh yeah. >> And Starks. Obviously there's a lot of nuance differences, but if you didn't know anything about the two and someone walked you through just like a highle description, you're like that sort of sounds like Fry.

>> Yeah. Yeah. I want to find more applications of pseudo random codes like especially in proofs. That would be super cool. So far, I think there are lots of like parallels, but I haven't found an application yet. So, if anyone if anyone thinks of one, that would be cool. But yeah, definitely a lot of like parts that are similar in spirit. Very cool. How is the verification affected by introducing the pseudo random error correcting codes? Does it change anything? Like we talked about like adding the extra step for the hashbased.

Is it the same extra step just a different technique or is there like more that's required to actually like see if this thing is watermarked? >> I would say it's a similar extra step different technique. >> Okay. >> Um now you're effectively doing a kind of robust decryption algorithm which for the schemes we have is fairly cheap. >> Nice. So I think the the interesting thing again of like what made watermarks really stand out to me especially in a world where like you're looking a lot of people in cryptography trying to figure out like something useful they cryptographers can say about machine learning and generally most of the things have been kind of pessimistic but watermarks are actually extremely

optimistic thing and they're not very theoretical like you can run them like there's lots of empirical papers that have compared the performance and looked at the bias And it's very clear they're very not only computationally tractable but like people are using in production and you know we talked a little bit earlier about Google synth ID. So I guess maybe could you just talk to us about how you see watermarks being used at production how you see their future evolving. I think you know obviously you know watermarks were invented in the world of like static LMS but like now we have these kind of like constantly evolving RL machines that are changing in some ways. Well, they were actually they were invented in the world of

static like material content >> products. And I think to to this point of this cat and mouse game, right? Like the watermarks of 1980s were like >> you know >> CD read Solomon code at the beginning on the like outer edge of the disc. Dating myself by telling you that. But >> uh and then you know like the '9s and 2000s it was like okay I have a hidden Marov model that like I reveal to you some of the preferences so like you can tell it was a machine. And now it's like hey this thing is generative it's like dynamically adapting the distribution so we can add this randomness and the seeds. Yeah. How do you see the future?

How do you see them applied? Like the broad very broad question. Feel free to take it whatever direction you want. >> Yeah. So, I remember you were asking me a couple weeks ago about adapting these watermarks to more realistic LLMs where it's not just looking at the response so far and then doing a next token prediction. They might be doing more complicated things like querying another model or going and fetching some information from the internet or like fancier RL things that I don't know too well. So, I think there's definitely work to be done adapting these watermarks to more realistic and more complicated like model workflows. Now, I

think that should be technically possible. I'm not familiar with too much work on it. I think some bigger questions are how to use watermarks given the fact that a dedicated adversary can always remove them. Um, so in addition to this theoretical result from watermarks in the sand that says watermarks can always be removed like in practice they are very easy to remove and especially as open models become better someone who really doesn't want watermarked content can just use an open model that's unwatermarked.

>> Um, so I think watermarks will be like one of many tools that we have for identifying AI generated content but they're not like a silver bullet. I don't think that means we should stop working on them. Like some people see the robustness issues or like the fact that you can always evade them using open models and say that it's a lost cause. >> Yeah. But I think we've seen in the past that ways of watermarking like static content can be evaded, especially you know just like your standard image watermarks that's just like some translucent text on top of a photo.

Those do stop people from just stealing the photos because they're annoying enough to remove. And so there is value in watermarks that can be removed but are a little bit hard to we just need to figure out, >> you know, how we can use them given that they can be removed. So that's a big question. And I also think a big question is how regulation will handle watermarks. So there have been many proposed bills that require watermarking, usually only images and video. Text is less common, but if watermarking ends up being required, it'll be interesting to see what

properties of the watermarks they want, like what kinds of robustness. And then we didn't touch much on unforgeability. That's the property that someone without the secret key can't forge the watermark. Um, >> oh yeah, I was about to ask about that. That's that's that was my next question. That's cool. >> Yeah, unfortibility is like a less common property. It's not clear whether people will actually want this because usually the watermarks are just used to weed out content we don't want. >> So there's no reason why anyone would want to forge them. So I think it'll be interesting to see what kinds of watermarks regulation calls for, especially in detection. Like right now, as I mentioned, basically all of the

detectors of deployed watermarks are kept private >> except for Amazon >> except Amazon. And if we want publicly available detectors, it's not clear how that'll work, especially if we don't want these publicly available detectors to make the watermarks easier to remove or replicate or forge. Yeah. Yeah. There there have been a bunch of papers on training a smaller model to remove a watermark from another model. >> So like you give it and like it has it it works like like empirically. So >> like like I don't think there's like a theoretical justification for like how many samples they needed like I don't but like the empirical evidence is like there is there are ways to train another

LLM to learn how to remove a watermark from a different one. >> Yeah. Yeah. And as you said earlier, like we're good at doing this training if we have some kind of discriminator type model. So if you actually have a detector, then it makes it even easier to do this training because you can see how well you're doing at removing the watermark as you go. >> Very cool. So I've been curious kind of through this we've been focusing really on these two techniques you know sort of the the more classic basic one of the red green lists and then the the one that you introduced on on images and with hashes and error correcting but like are there other types or like techniques of watermarking for different types of content for different types of

output? >> Yeah, so I'm less familiar with other kinds of content. I think the obvious ones that come to mind are audio and video. >> And I've actually seen one paper that uses pseudendom codes to watermark video. >> Oh, cool. >> Which I don't know the details of, but I would imagine that the technique extends well because in theory pseudendum codes work anytime you have some kind of randomness recovery algorithm. And what I mean by that is anytime you can take some AI generated content and recover an approximation of the randomness used to generate it then you can use a pseudo random code to watermark this content by in generation replacing the randomness

with a pseudo random code word and now it's detect you run randomness recovery you get back something close to the code word and you run your like pseudo random decoding algorithm. M and so like Sam and I try to present pseudonym codes as this general framework for watermarking any kind of content. I'm not sure yet how practical it'll be um for like video and audio. But in theory anything with randomness recovery can be watermarked using a pseudorandom code. I wonder with those two examples, what's different about them is they also have that element of time and so somehow the watermark has to kind of like I mean obviously you can put it at the very start or you can put it on a single

image like the third screen or the fifth note or whatever like or you know fifth second or whatever you can choose some moment in time if you want to do that test but I wonder if like time adds some element kind of like something where you could trademark on that feature the fact that there's like a timebased piece of content that it's like running and and changing. >> Yeah. Yeah. I think like the time dimension gives you a lot more to work with. So, it might be even easier to watermark these things. I don't really know, but it might be harder too because the generation is more complicated. And one surprising thing I learned about video is that maybe this is stupid to people who know, but a video is not just

a sequence of images that's stored. like you're kind of doing some checkpointing and then storing like the difference between different frames. And so when you get a video, you don't get like a bajillion images. >> It's not something more complicated. >> Yeah. It's not old film on the just like running slides of single images. Yeah. >> Yeah. So that maybe makes it more challenging, too. But this isn't my area of expertise, so I don't know for sure. I feel like the tell of that, by the way, was like when you had bad downloading speed and you started to notice that like the image would change.

It wouldn't just stop on an image, but it would like capture other colors, you know, like when like the green starts to take over half of the screen. This might really date me. I don't know if this stuff still happens. >> But yeah, I have a feeling that's sort of the tell that like there's something running those videos. They're not just static images like they used to be on the projector cameras. >> Yeah, that makes sense. That tell did not come through for me, but it checks out. Yeah. >> Yeah. Miranda, thank you so much for coming on the show, sharing with us this kind of at least for me like a pretty new space in AI and watermarks. It's not exactly ZK. Like I think what we found through this interview is there's tons of comparative techniques being used. A

lot of similar terminology and properties that we are very used to in the ZK world, but it's been really cool to kind of explore how these ideas are also entering into this watermarking space. So, thanks so much. >> Yeah, thanks for having me. I love talking about watermarking. Um, and I think it's a really exciting space both for research and for seeing where it goes in practice. >> Nice. >> Yeah, thanks for thanks for coming on. As I have have hinted, I somehow have have been kneede in trying to understand this literature because it feels like it's actually practically it's theory research that's practically useful in a way that I think it you don't find as often nowadays.

>> Yeah. Cool. Thanks again. I want to say thank you to the podcast team, Rachel, Henrik, Tanya, and Kai. And to our listeners, thanks for listening. [Music]