When Agents Build and Agents Judge: Lessons from an Agent-Native Hackathon

Transcript

All right. All right. All right. GM GM GM. Cool. So, I'm Denver. I'm the co-founder of this neighborhood friendly hackathon platform called Devfolio. And with me is our back-end and AI engineering lead Swarnim from from the Devfolio team. And we want to talk a little bit about a hackathon we did earlier this year, right? Uh so, this is very interesting. Like, so if you have known about if you know anything about Devfolio so far, um we've been doing hackathons since a while. And so, Devfolio has grown to like a million developers so far on our platform, over 1,500 hackathons hosted.

But um for us, for the team, right? We always like to push the envelope, right? It's not interesting for us otherwise. Um so, we have done hackathons ranging from 100 to 2,000 people in person. Um the 2,000 person being the biggest Ethereum hackathon in the world at that time called ETHIndia. Um and we've done hackathons ranging up to 10,000 people online. Um I think Base Onchain Summer was 1,200 projects or something.

Um and then we've done like unconventional formats over the last few years. We've done like hackathons in in villas, hackathons and like gyms. Uh we did one at Net- Network School. What? And of course, how could I forget? On on on last year we did a hackathon on a train, which came back to this this year as well. Um with H City. So, very creatively, um 33-hour hackathon from Bangalore to Delhi last year was called Blocktrain. Yeah. And this year we followed it up with um a partnership with H City and called it like Edge Blockchain. Um and brought like a bunch of folks over

to um Devcon. Yeah, but what I want to talk about today about is Synthesis. Um so like a few months back, we got this message like from Binji and Sophia from I think Foundation wanted to do a hackathon, agentic hackathon, but then we thought, "Okay, it would be interesting if the hackathon is uh the participants are AI agents. Uh the the project submissions are done by AI agents. The submission platform is agentic and the judging is also done by AI agents, right? So, everything agentic. Uh you don't need to do anything. Just point your agent at the problem. Uh it reads the skill.md file and does everything.

Um yeah, it was 10 days. We had 28 uh seller partners. Uh had 1,500 builders come in and over 680 projects submitted, right? No UI. Uh the agents called uh the API directly. Every agent got an on-chain identity uh using ERC 808004. Um and we minted it for them, so there was no crypto barrier for them, right? Uh and just before publishing a project, the agent just had to move the identity to the wallet it come from. Um so, with that, I'm going to uh hand it over to Swarnim to talk a little bit about how we got through this.

>> Right. Perfect. Building Synthesis changed how we think about designing platforms for digital tools. Synthesis was built to be agent native from the get-go. The primary interface through which agents discovered and interacted with the platform were skill files and API docs. Individual API endpoints were designed to be as atomic primitives, which were all retryable, and more complicated flows were outcomes that the agent retrieved by working in a loop and composing various endpoints together. The main skill file was a map of everything the platform supported. And the agent queried what it needed when it needed it to work out which stage of the hackathon it was in and figure out what it needed to do next.

Judging was agent native, too. Each of our 28 partners raised to the judge agent, which also had an identity on ERC8004, and set up two rubrics by interacting with the agent. The product rubric captured the partner's taste in uh products, and the technical rubric uh gave judges an intuition as to what uh the agent like like what the partner wanted uh the builders to work on. Uh Judging was also a bit Judging Judging also took place in two stages. Uh a cursory look for the product uh side of the rubric, which identified the problems that the products were trying to solve, and you can get to general intuition of what kind of a project it

was. The technical rubric included more uh architectural constraints and technologies that the uh partners wanted their builders to use. For each project, uh the judge agent spun up the project in a sandbox, went through its code, ran tests, did static analysis, and created a report, which would was then sent to the human partners for them to make the final call on which project won which prize. That's fine. >> All right. Uh so, judging really is an attention problem, right? It's not It's simply not possible like I don't I think I think now that we have agents, uh it's a problem that is uniquely suited towards agents. Why? Because they can

inspect every project within uh the timeline and give it the same rigor from beginning to end. Human judges just can't do that. Like we are we're going to get tired at some point of time, right? Um like you know, get brain fog, get like you know, um not not be be no ironically not have the best judgment, right? Um so, what so, what was hard harder for us to codify was this the taste, deciding what matters, and weighing the trade-offs. Uh we tried to do a good job with that, and uh we got to the final decision.

Uh and yeah, like you want to cover the rest? >> So, I definitely remember when we decided to build judging as a first-class product or for our judging offering, making judging fairer and more consistent across multiple runs on large scale was one of our primary intentions. So, the agent is responsible for inspecting the project. It does some of the same things that the synthesis judging agent did. It clones the repo in a sandbox, reads the code, does static analysis. It does all of this and creates a very thorough report again for the humans to make the final call.

So, these are just like some of what the screenshots from our product. So, you on the on one side of the screen, you can see the agent's report and the other you can see the judges panel where they score the project based on the various parameters that they defined earlier in their rubric. In the next screenshot, again, this is how the organizers or the sponsor representatives can decide a rubric for their agent. So, they could decide like on different parameters in which you want the agent to assess the project on and you construct this rubric by having a conversation with our sort of chatbot style interface, which helps you through the whole process.

While the agent is working, you can also see its faces in real time, so there's transparency and you can know exactly what the agent is doing. And this is sort of the final a few final screenshots of the report. Um Yeah. >> And so, in all agents gathered the evidence and humans essentially decided what to make of it, right? So, it changes things for every builder so that every project gets the same attention to detail whether the a judge gets to it first or last. We bringing bringing this to all hackathons on Devfolio and yeah, couldn't be more excited for all of you to try it out.

>> Thank you. You can find us on Twitter on these handles.