Voxbento: When 2 Seconds Matter — Building Real-Time AI Captions and Live Interpretations

Transcript

So thank you Vanch for the wonderful introduction and briefing us about wisdom and hope you guys really enjoyed that. So it was quite a bit of a thing. So do you think uh the next talk can top the wisdom? So we have another AI project which is walks bento. So hello everyone I'm Wengerkar. I'm a GSOC contributor uh at for Asia. I've been working at for Asia since a year now and um I'm going to talk about WX Vento.

So yes, yeah, I can see you guys better now. Yeah. And also my slideshow. So but first let's understand what walks bento is about. So the problem so multilingual events are very hard to run. So like I'm speaking in English but this conference is going on in Mumbai. So many people uh actually know Hindi as Hindi is the national language of India but there are even some more people who speak in Marathi you know so Marati the native language which is spoken in Maharashtra so it's really difficult for

some people who are you know native Maratti speakers they like to would like to hear the conference in Marati they would like to see it happen in Marati although this is a international event so it's in English so for you this is the solution So this this uh when there are such international conferences which are big like we there are a lot of different conferences like we have devcon there is the force Asia summit which is in Bangkok every year and yeah so s such conferences where we have audience from all around the world uh the conferences happen in English so to get them native we have a solution walks bento but you might say that oh at the previous Asia summit we already had Thai and we already had you know the

different Vietnamese and we already had different langu language is already there. So the thing is the setup is like this. So you know it's not really very easy to set this up. It's like a it has a massive footprint. There are huge number of people involved. There are a lot of cables. There are boots and they cost thousands of dollars. So walks vendor is something which makes this thing uh much much much easier. So what if uh you all the interpreters and the listeners need is a browser. So let's see how we can make how works bento makes this happen. So works it's an open-source browser first solution and uh in the audio is interpreted by a

human interpreter and we also are in the world of AI the age of AI. So it's either a human who does it or it's a AI which can completely do it and it's also uh can have the interpreters with the do it with the help of AI. So it's whatever you need, whatever your event needs. So like these these are some of the screenshots of works bento. So like for example this is the interpreter booth. So the interpreter speaks into this. They have multiple instances of this booth. Multiple people join. They see the live conference here. They can tune in. They can translate into their own language. This is the one one of the

listeners page and they see there for example this was a live conference going on in Paris. So this is the yeah so it's there in English French and obviously you can also have the text to speech there. So it's not only that walks bento is an independent solution it can be clubbed in with you know uh different different event management solutions. For example, DevCon have their own application. Uh many solutions, many events use event. So it can be plugged into event. So it it it has a very native support for all different kinds of uh you know event management software and different video streaming softwares. It can be plugged

into a YouTube stream. It can be plugged in anywhere. So it is very easy. So enough of screenshots. So let's try it live. Yeah. So you can scan the QR code here. You can pick a language. I think it's the main hall. So you can select the main hall and you can listen you can listen to this uh you can listen to this talk which I'm giving right now here in your own native language. Yeah. So you can scan it and pick a language. Yeah. So back to the slides now. So what just happened between my voice and your phone? So this is what is going on there. So there is the floor audio which is what I'm speaking. uh we use media MTX for you know the whip and web

so the architecture is pretty easy so the audio never touches any of the uh fast API or it doesn't go through uh any of the Python so initially it used to we'll go through that later on so yeah and the AI workers is all in the fast API side and it's as I can't simplify this even more it's pretty very simple. So yeah, like speaking of uh getting Python out of the media path. So here's the question. So where would you put uh Python in a real-time audio system?

Yeah. So the first answer was right in the middle. But it was also wrong, right? So we keep complet Python completely out of the audio path. So initially we used to use uh HLS. So the challenge with that was uh HLS you know caches the segments and the there used to be a lag and it it needs needed to be scaled vertically and you know it it wasn't really fast enough with very limited resources. So we swip to switched to whip. So now like it's for fast API is completely for the entire um you know the AI part and the audio is completely through medium takes.

So another another challenge that currently we are facing is that uh there are big glosseries there's a lot of vocabulary and there are there are a lot of terms which you know need a lot of context which are related to the conference. For example, the just the word speaker, it can mean multiple things or you know uh each in different languages uh a single word in English can have a lot of different meanings. So uh this is one of the challenge we are still trying to solve with local LLMs is trying to provide context into these different models. So this is one more challenge we are currently trying to solve. If you know the solution we can catch up later. Yeah. So it's uh let me ask you something. So which which LLM or

which model is the best Claude or maybe GB? Yeah. Yeah. >> Yeah. So the Yeah. Exactly. That's the correct answer. So it actually depends on what you what your what you need, what privacy setup you have, what um it it depends on the language. It depends on multiple factors, right? So the AI pipeline we have it has multiple stages, right? So each stage uh in Vox Bento has uh you can plug in your own models, you can plug in your own LLMs, you can it's just completely configurable. You can do whichever model suits you best for that purpose. So we have a set of models

which are already set for uh by default for these we have obviously bring your own key for these models as well as we have some models which are we are self-hosting and yeah so it's it's depends on you whatever you want to use for whatever so there are uh you know there are four stages of forbento so first we the audio which I'm talking it goes into that so the uh it is copied there so that that there is a different LLM for that uh you and choose your own. Then the second one is the one which converts my speech into text. Right? So this one again you choose your own model. We use whisper by default. And then for the translations and the last one is for the text to speech. So it's

completely configurable whatever you want. So another thing uh which works bento will is providing is a completely local translations. So it's like you can just download the application completely on your own uh you know computer and uh you don't need to send your data to any server you don't need to send your data to uh any different LLM or you don't need to send your data to the cloud. So some meetings they are you know maybe some private meetings or whatever maybe your organization needs has some meetings you don't want the data to go outside nowhere it completely stays on your laptop.

So as I said it's built to plug into you know different event platforms. So for example here's a screenshot of uh we using it with eventier. Maybe the quality could be better, but uh yeah, it's it you can plug it into your own ticketing solution, your own event application, your own website, and possibly into YouTube as well. So, it has a really good uh API documentation. You should definitely check it out. So, yeah. So the works we I started walks bento with uh like it started around a year ago at the for Asia summit which uh we realized that this solution which is there it could be you know made

really simple and we it the as you as I said in the first slide it's very complicated setup it's not so easy so we are building the solution since yeah So VBento didn't just remain you know a GSOC project which I started at Google Summer of Code but it turned into a entire community. We have multiple contributors. I see some contributors in the crowd as well. So so it's it it has now 120 forks. It has 700 commits and it has 70 plus contributors and it's a community. So thank you for everyone

to make this happen. Yeah. So what what is the next thing WS bento is trying to do now? So we are trying to make the audio for the organ make it organizer friendly. So what do you mean by organizer friendly? For example, we have the open tech summit here. So if uh you guys are watching on your phone the live translations or you can see that it actually uses the word open text summit in Hindi as well. So it's not that it changes the it it doesn't change it to donaguri or it doesn't change it there.

So there are some nouns there is the vocabulary that is glossery. So this is something we are trying to enhance and make even more better. Then there is hardened security and reliability. We were trying to make it uh more secure, more uh you know keep it as as much as we can on the client side. We don't try to send your data to your own vocabulary to on the server and better local AI is also something we are working on. So you can uh try VBTO at voxbento.com. You can contribute it to at github.com/fosia/wogbento.

There are a ton of uh good first issues labeled and you can best part is you can even run it at your own event. So you can self-host it. It's very simple. It's a single command setup and yeah so thank you and uh if you have any questions there's a workshop about Bento and Agentic AI at 4 p.m. at the builder area and also WBO has its own booth and the exhibitor's area outside. So thank you and let's connect. So before I step off the stage uh I get to introduce the next speaker which is very personal for me. Uh it's Shivatsv.

He is one of my GOC mentors and he's also a very good friend. We first met at the foria summit in 2026 in March this year. uh he's a open source contributor uh and a developer and he has worked extensively across the foria ecosystem uh majorly on event and um live event check-ins batch printing exhibitor tools and different platform integrations he's uh traveling across the world and AC try uh you know making events AI friendly now so I get to hand over the stage to shats and he will tell you about