Weeks instead of Years: Creating the Loungemesh Hallway Video Platform

Transcript

So this uh uh presentation is about how I created Lounge Mish a hallway meeting platform within like 5 weeks of a period uh which is supposed to take like a approx one or two years with the all the WebRTC engineers and so on. So it was a very nice experience. I'll I'll share a bit more. So how do we see like conversations happening inside uh something like a grid based talk system for example Google meet teams and so on one person will come they'll speak and then the they'll wait for other person to start speaking up which is not how you will have conversation informally at an hallway so we were thinking how we can bring that idea to the general

public and I thought of like why not have like a hallway style communication online. So what's going on? Okay. Okay. Yeah. So there's a issue with traditional video calls. You are logged inside a very rigid room. You are like a one speaker who's only talking other people are not able to like post their inputs and so on. So it feels like a bit forced on you. But what we fix by doing this is we walk up to like the join the actual conversation. We can uh like step up our boundaries and we can talk to

multiple people at the same time. So as you can see here you just step effortlessly away from your awkward conversations basically like how you join in a group inside a hallway. you join in listen to what is going what's the debate is going on about share your thoughts and when you are done with it when you feel like okay this this uh I have contributed enough to the conversation I want to discuss with someone else you can just take them into a private column that's also something that lounge mush helps us achieve so here's our team foria team he's saying hi via the platform you can see a lot of uh contributors maintainers and so on saying hello so this is the core idea. So every one of us can join and

join the live. This is like a flow where you join and you join into a pod which is like a circle in between. So inside the circle the conversation happens as you walk away from the center the volume gets lower sort of how like you will feel inside a hallway there's some murmurings happening in the background and so on but it's digital that's the core thing whenever we try with different platforms like Google meet and so on this is like gets very difficult to like have this hallway style conversation lounge m fixes that and it is very nice uh in terms of like how you and just walk away with like two people and have a separate discussion in the corner of the room. It kind of remind me like how you are at a community hall

similar to what we have just outside the hall. We just meet like five people talking in a group and we join in the conversation. Now the core of this thing is acoustic proximity. So the we engineered gain nodes and off thread system which do not work on server side things. So instead of like having all the compute on the server side and having like a bloated software which is like not very nice in terms of uh cost and so on, we added a gain node system which works on the client side and this is what helps judge how much of the audio you will be able to hear. For example, you are in a circle, you are away from the circle,

you you are trying to get out of the circle and so on. So it determines how much volume should be audible to you and uh for example there's this some people at the very center and you have gone to the very center of the cluster now you are the loudest person in that room then one of the engineering challenges here so that's bandwidth and culling for example we have 30 streams for in case of Google meet you can actually see like six seven streams at a time and then there's streamings uh also going on like screen shares and so on, right? So for screen shares it gets a bit tricky and so so what we did anyone who's off the screen is not loaded and since we are

not using very big avatars as you saw like something in Google meet like you show like a full uh person perspective and so on inside here we just show a small canvas and there's a like avatar thing so that does not need heavy uh videos. So we just limit the quality of those to 240B, 360B depending on the network uh connectivity that's available. That allows way more streams to fit in the same room at a smaller server size also. So and other thing that helps make this all together possible is screen sharing.

We limit screen sharing to 720p. In case more people increase, we can give options to go down and there's recording options as well and that recording can get done in a high quality. So after this we arrived at the audio physics. At first I thought like the audio physics is going to be weird because implementing a hallway style conversation was not something that was uh very good uh like very well done before. So I was a bit worried like how we can get the audio engine and so on.

So apparently if you uh like define architecture files very well with AI, it is quite good at it. But like there's some manual intervention involved always. But uh there's peak g their peer-to-peer gain nodes as you can see here. There's uh ramps and zero pops. What does zero pops mean? You were wondering. So when you get out of uh audio stream, right? Sometimes it breaks. there's a pop sound that you hear. So we uh added like a layer that checks if there's a pop sound. It just eliminates it so that it does not feel weird like when you are getting out of a hallway style conversation.

There's a media control plane. So the media control plane was designed in such a way that it's very easy for us to develop on top. For example, there's a requirement a certain feature is required and now that involves changing to the media control plane. If we just contact directly with the back end then it's an issue that it can break other parts of the platform as well since there's a control plane involved in between there's a boundary adapter so it checks whether everything is in place or not and there's a only a single round trip so what the single round trip means you have a audio stream that is going on you have webcam stream that is going on and then there's your screen sharing thing right so this puts it in a one uh

packet and transfers So rather than having like three separate connection streams and bombarding it with the server, it just goes inside one stream and then split up at the other way as well. So that's the idea behind uh media life cycles. So whiteboard interesting thing about the whiteboard is like one of the uh contributors proposed an issue that it will be very nice if we had a whiteboard because in meeting uh like meeting halls also there's like some form of in informal communication there that some people are sharing their opinion and uh at the back there's a one person presenting and other people are it's sort of like how you do Q&A so he proposed that there should be a

whiteboard and I thought about it for a moment and I said that's a good idea but it's also complicated because say someone is in Europe or Americas and we are here in India right so there can be latency gaps and so on so we engineered a web RTC based stroke syncing system completely with AI to be very honest I have very manual like very zero kind of manual intervention and then the stroke sync was completely designed by and it worked flawlessly that was kind of surprising thing to me but there were some uh small nitpicks that I had to fix but like sketch workflows and diagrams for example you see here I am like influ like including a diagram how this is supposed to go there's the spatial flow

that's what you see in the background and there's the media service which takes your webcam streams and so on and then it passes over to proximity gain nodes there's how you can present uh your streams for example here's one screen being shared there's chats going on you can share links to documents and interact like how you interact in a hallway maybe share like a telegram link for example to a message a discord message attachment or some form right or reactions one of the biggest things I feel like missing in Google meet since I am there in a lot of Google meets to be honest but I feel like I'm not able to

express myself because the emoji choices are very small. So lounge mesh has a lot of emoji choices. Basically the whole pallet of emojis is there because I feel like why restrict a person from expressing themselves because that's not how what happens in a hallway. If I am free to express myself I feel like we are bit more open and this is a open source community. So it's better that we are able to express ourselves freely. Now you will wonder who can just land on the floor. There should be some form of moderation involved because this gets tricky. Maybe some person is not authorized to be in that hall and it sees inside the hall and that makes people uncomfortable or something right.

So we have also designed a moderation based system. You can uh lock their mics cams uh moderate their chats and so on. There's access to features that can be managed by moderators. So there are three people inside the hall. There's host then there's moderator and then there's attendee. So attendee does not have like feature controls in the sense uh he can control this uh stream the camera and the microphone and the other feature he's allowed access to. Host can enable these features. Moderators are not allowed to disable these features.

They can just disable them for an individual user if they are misbehaving or they can just knock him out of the room. So there's options here to knock out from the room, turn off their cameras if there's something inappropriate on the screen as well. So how we build it that's that's a very nice question and some of the people I think reached out uh in the hallway uh when like before the talk there were two girls who were asking like how did you accelerate this so much with that you were able to do this with uh within weeks. So first of the things is you should always show clear boundaries around what is that you want and then any complex media any engine that needs to go through and

will require changes at a later point you should decouple it all together. So there should be a layer between the front end and back end which handles that communication and at if some pointed there is a failure there's not a failure across the whole platform. And then there's autonomous agents for uh multi-line refactors. That's the agents we use today. How did we accelerate this test workflows and so on? So we added 100% test coverage as a safety rail for these media engines. not for the whole platform but the media engine has 100% test coverage always. So there's no media failure in terms of like a webcam stream failing. If there's a webcam stream fail, we get immediate feedback

and we start iterating on how we can maybe sometimes it can happen like they're doing it on a very small server. There are a lot of people's but we believe like uh we should optimize and make things efficient as possible so that this is not very hard thing to do. We encode architecture rules in domain skills. What is domain skills? You ask. There's agent MD files and so on that you declare in order to have like your AI code better according to your needs. So we did that and we validated early with opensource community. So I think you saw in the slides that the whole foria team kind of said hello to you right. So there were similar sessions then that uh that session that you saw was at the G-Shock mentor summit also. I

think Shivasta was there at the G-Shock mentor summit also and we said hello to everyone. So that's kind of the interaction this platform can bring and it was very uh wonderful in the sense that we can communicate these ways. Now what is the synergy you ask like what is the foundation of the project? How can we achieve something similar or like help contribute to the project? So these are the technologies that you need to be aware of before contributing to these projects. This is Jity libjity meet go. These are libraries open source libraries that are available to the general people. You can read the documentation in order to start

contributing to similar projects. And we use vue3 pia and typescript typescript for type safety. We have pia to managing the stores and there's a high performance web audio api that is selfgenerated in our application. So that's a separate documentation we have available on the lounge mish repository. you can check out later. So you can read the documentation around how you can contribute to the high performance web audio API because uh the audio was a bit tricky part for us because there's no existing solution that is doing it uh well then with AI agentic pairing like with even though we were doing like this multifile refactors and so on it was not

particularly good around these audio engines and so so with audio engines it gets a bit tricky because there's no sol uh there's there's a solutions but they're not that efficient. like how we like it. We want everyone in the room to be like audible even though they are moving away and so on. So this this needs to be taken care of. So we designed like custom uh files like architectural decisions and so on. So we had to manually intervene at this stage for uh audio thing a bit but like for other things the AI was very helpful for me.

And then our open source community I'm like very proud to be part of the force Asia community. It's a very nice open source community and uh they helped me through the way during the whole G-Shock period and we tested it in the live of uh lot of lives forious sessions a lot of Google meet alternates we have a lot of Google meets usage for uh meetings and so on for some meetings we replaced it with lounge mish we tried it out and got feedback from the whole team people even who are like newer onboarded that that feedback kind of matter because some people who are like from a lot of time when the community have some biases maybe but a new person does not have any bias towards anyone right so in that sense we also collected their feedbacks

and this is communitydriven road maps we shared so at uh like I think it was around second to third week uh there was five people who were very interested in the project and they started contributing and they were asking like how is this designed this was a communitydriven road map for them so I gave them some diag diagrams and so on. This is how it is supposed to be structured and that's how we develop it. So how is AI gated here and how are the PRs being kept honest? There's a strict adapter boundary as I told earlier. So there's no direct import of a web RTC.

There's no direct communication with the back end front and there's a layer in between. So a failure is detected early and uh there's agent skill files that are defined for this uh adapter boundary specifically and there's a 100% test coverage rails and we later this week added playrite e2 suits that actually test these webcam streams and so on. So here's a live hallway room link if you guys are interested you can join in maybe have a chit chat with people who are here and uh there's a link to launch m repository so you can explore the project and contribute in any case if you have any doubts you can uh connect with me on GitHub or LinkedIn here and

there's a workshop happening uh in the afternoon with event workshop you can also reach to me there so