Freeze Commit Security Reviews Can't Keep Up with Modern Development

Transcript

[music] [music] >> She's spoiled in there. I hope no one takes it. So, what is a freeze commit and why is it a pain? Freeze commit is the standard way that most security firms do a source code review. So, we get the code, we pick whatever the most recent commit is or whatever commit would like to be reviewed. The security team comes in, we review the code at that commit. We will then send the issues that we found back to the client.

The client will then perform fixes, mitigations, what they need to do. Then finally, we come in for a retesting and this way we review that all of the fixes are correct, have solved the problems and everything. The problem with the freeze commit methodology is that it is extremely sequential. There is no parallelization that's happening here. If you look at this diagram, you'll see development cycle, security review of time, development cycle, security review of time.

This is fine when people aren't trying to rush out the code. If if people doing smart contract updates that occur very infrequently. Smart contract updates may happen every 6 months and in which case you don't mind a little bit of sequential. You don't mind losing a little bit of time on the clock. However, there is certain times when this is going to be a problem. The problem is happening when code is generating at a much faster pace. So, software development teams now have access to AI and they can produce code at many X faster than they've previously been able to. And therefore, when we freeze a commit and the software

development team continues their work, they end up putting weeks worth of code on top of the commit that were frozen. And so, our freeze commit is often no longer relevant because development has moved so far. and so when we come to the retesting it's um a huge amount of work for us, and that needs to add even more sequential time to the end of the review. So, what is the solution to this? Well, the solution is a rolling commit review. The rolling commit review is changing this uh pipeline to make it parallelizable. So, if we start with a fix, we call it the baseline commit, and then we do the rolling stage, which is where we look at PRs that are coming

through from the development team, and then finally uh at the end we do an integration commit. We really want this integration commit because that's what ties all of the review PRs, all of the fixes, all of the mitigations together into one final commit, and then we give this final commit to the development team that they can say, "Hey, this is clean code that has all of the fixes, and it is ready for deployment or release." So, what would that look like as a uh time map? As you can see right here, the uh parallelization is coming in.

Development Developers are continuing develop at the same time security reviewers are continuing to security review, and the entire pipeline shortens from 4 to 6 weeks to about 3 weeks in this example. Uh why did we actually implement this? Uh we implemented it this way for one of the clients who wanted to make a hard fork every 2 weeks, and that amount of time, if they were wanting to release every 2 weeks and we needed to make a 4 to 6 week review, uh the entire review process is over doubling their development cycle, which is just infeasible. Here, however, our security review review process is probably only adding about 1 week to their review cycle. Uh so, the benefit here is

clearly wall clock time. However, uh for security reviewers, this is us taking on the pain that developers used to have. Uh we end up with a little bit more admin because we're having to continually review some pieces of code multiple times. Uh that is things like, for example, if one PR touches a function, we'll review that function. Then if the second PR also touches the same function, we review it again, and so forth. So, we do actually add a little bit more time to the security engineer, which is a little bit less efficient. Uh, on top of that, there's admin overheads because we now have to track multiple PRs and see what they're what code they're tracking and make sure these PRs aren't interacting in weird and funky ways. And finally, um, is the the reduction of the concurrency.

So, um, Sorry, lost my train of thought here for a second. Coming back. >> [laughter] >> Yeah. No, the the concurrency just reduces the weight. Uh, I'll leave it at that. So, what is the difference between the two and when should I pick one and when should I pick the other? The freeze commit is generally good when you can stop code, and you want your code to be, uh, well scoped, well bounded, and tightly fixed, and efficiency matters. As I mentioned, if you're on a tight budget, the freeze commit is going to be a little bit better for you because you'll spend a little less on security.

I I estimate the difference between the admin overheads and the extra repeated work to be maybe between 10 and 20% if you take a rolling, uh, review commit versus taking the freeze commit metal. So, there is still a lot of time in the world freeze commit, and I'm going to recommend this for most of your smart contract development to continue on the freeze commit methodology. Where would you want to do rolling commit reviews? It is generally in your off-chain protocols where lots of code is coming in. This code is coming in fast, and the developers are not stopping, uh, for clean releases. For example, if you have a client who's releasing a hard fork every 2 weeks, this this is the type of time when you need to run the rolling commit review.

All about a faster review. Now, time for any questions. >> Thank you. Thank you, Kurt. >> [applause] >> I think we have time for maybe one question. Yeah? Maybe we have Yeah, in the front. Look out for the box here. >> Thank you for the talk. What What do you think about the quality of the security of you in this ongoing process? Because from my experience, you know, auditors say that it's really stressful.

You know, this process because you have a lot of information, a lot of conflicting information, and it you know, your context window, your human context window gets messed up really quickly. >> Yeah. Yeah, that's a really good question. And so this is where the admin comes in that it's very important for us to track these things well, like where these new PRs are conflicting with other PRs. And then after that, at the end, it's very important to have a final integration commit to make sure you review the code when everything's merged in together, and you can do that holistic review without having to try and keep all the little pieces in your head. So between those two, it helps to reduce the problem, but obviously, yeah, there's there's a little bit of an extra

over overload on the reviewer's mind if we do it this way. >> [music]