Project Hummingbird - Minimal Images, Maximum Trust

Transcript

[music] >> Hello everyone. Uh good morning. How are you today? Uh so, I'm here to talk about something called as Project Hummingbird, uh minimal images, maximum trust. Uh my day job is working as a security engineer at Red Hat. If you are not aware of Red Hat, you might have probably used Linux. Uh Red Hat provides the uh the de facto stable uh distribution for it called Red Hat Enterprise Linux.

You must have used OpenShift or automated your DevOps workflows using Ansible. >> [snorts] >> Uh this talk is focused on security. So, if you are a developer who had to triage security reports, uh this will be of value to you. If you are a security engineer, this is of also value to you. Yeah.

Uh how many of you are aware of the PHP incident that happened uh a couple of weeks ago? Okay, almost everybody. That's amazing. So, I happen to be one of those lucky ones who picked up this ticket even before it was announced to the world. Um So, if you look at this date, 21st October is the date when it was made public. These were two high-profile vulnerabilities found in the PHP code base uh with a CVSS rating of 9.8, which is really, really high.

Uh it's been almost 15 days since this was the patches were released. Can anyone take a wild guess how many systems are still affected by this? 1 10,000? >> [laughter] >> That's a good guess. I just happened to check this yesterday evening while I was at the venue. You can go to this website yourself. I'm pretty sure the numbers have increased since yesterday. This is the data from yesterday evening. Even after the patch was released almost 15 days in, we have 29 million systems that are still vulnerable to it. So, there is a problem somewhere, right?

Let's see what the defenders are doing. If you haven't heard of OpenSSL, I highly encourage you to follow them. They are the open source security foundation. They come up with a lot of projects that improve the security of open source projects. They give a lot of guidance. They have a lot of discussion around this. They recently released a blog just a couple of days back again on the PHP incident. What they say is this is just a wake up call and fixes are just the beginning okay? I'll I'll just give you a gist about it, but I highly encourage you to go and read this talk. So, what they are saying is the maintainers or the upstream developers are in charge of creating the

fixes and releasing the security fixes. The distributors, vendors, and everybody else who ship this software then incorporate these fixes, provide more guidance around what's happening, are you affected or not, what are the mitigations, and so on. The operators at the extreme end who already have systems deployed will assess their exposure, will decide if it affects them, will decide the risk, and they will be in charge of, you know, deploying the mitigations and the update.

The gist of it is upstream do not control downstream deployments, okay? The middle is where all of it happens. Of course, they do provide some practical steps, which are namely like before raise the security baseline, have more contribution in upstream, or you know, easily share the vulnerability information. Um PHP was just a beginning, but if you are into open source, what's the status of open source security right now?

And how do we really find it? Uh let's pick another popular project, which is really nice, curl. Everybody uses curl, right? It's bundled in a lot of billions of devices. It's probably the most used open source project after the Linux kernel. Uh Daniel is, uh I would say, the benchmark for how an open source maintainer should be. He regularly writes about his experience with AI, with securities, how he manages it, and all. So, I look at this as a bar about what open source security currently is.

Uh in this blog, he admits that he's making a complete U-turn on AI slop. In last year, 2025, AI slop was so bad that he was at a point where he stopped accepting any new vulnerability reports. He literally shut down the program. Two, current April of 2026 is again where he did a complete U-turn, where he again opened it up. He admits that AI slop has reduced significantly. Uh we will see some more highlights from here.

The good thing about Daniel's blog is it's backed by data, which he publishes regularly. You can just compare 2024 and 2026. 2025 is all about AI slop, where everybody had access to, let's say, LLM of their choice, and all they had to do is find security issues. Don't mess up. That's it. And people would print PDFs and large text files and send it to maintainers saying, you know, this is a security issue. Fix it. Give me a CV. Give me acknowledgement. Um Here you can see the number of hours between security reports. So, read it slightly differently.

2026, the count is low, which means the reports are so much that there is no time left in between to triage them. Compare it to the previous years. This is the most important slide. What he also admits is the quality of the vulnerability reports is higher. Notice confirmed vulnerability rate. So, what he's talking about is not just reports which have been thrown at him. These are triaged issues which are accepted by the curl project to be genuine security issues. They are assessed based on their quality, what risk they provide, they are giving a rating and a CV.

So, this is very important. Finally, Daniel being Daniel has it goes into prediction mode where he predicts that, you know, by the end of 2026, which we are almost there, curl will have at least 50 confirmed vulnerabilities that will be coming in. I just checked the number, but I'll leave it as an exercise to you as to what the current curl CVs are right now. And he's pretty close on his predictions. Okay, this is about curl, but what about the larger open source operating the space, right? So, Daniel has been pretty gracious. He did a poll where he shared his findings and asked other open

source projects to kind of give a rating or you know, discuss what they're feeling. They tend to agree with it. If you look at the open source projects, HTTPD, bind, Django, power DNS, Python, URLlib, these are all the standard open source components that we all use, right? And everybody's agreeing with that. The point I'm trying to make is we can no longer make an excuse for AI slop. AI is getting surprisingly good at finding security issues. People are getting surprisingly good at using AI to find those issues.

And we [clears throat] see models being released almost every other day, which brings something to the table that was not available earlier. This was about the open source projects. Excuse [clears throat] me. Or the maintainer side of it. What about false users or in to put it more broadly, what about the enterprise users? How are they kind of dealing with this? Here are some quotes from some of the customers or people who are using some of the open source projects from Red Hat.

If you notice one of the biggest challenges that they have is the time spent on investigating vulnerabilities. Security scanning is of course a critical part before releasing a project or releasing a software. But when you try to do it at scale, it really really slows down teams. So a lot of valuable engineering time is spent doing security. >> [clears throat] >> Let's look at another one. So the another common challenge across the industry is the container images, standard container images that we have have a lot more than it's actually needed. That's how the container images are built.

>> [sighs] >> What this ends up doing is increasing the attack surface really badly. Okay. And as a side effect, the scanner alerts are ever increasing, which is correlational to the effort that is required to you know, triage all of that. A final one is again identifying whether these security reports are genuine or a false positive, right? You could be using a component which might be tagged as you know, having a CV which is very important. But is it really the issue with that part of a function which your code is actually using? Okay. Finding that out takes a lot of time and that's what's dragging us down.

So um with this background, what's the primary driver for project Hummingbird, right? The first very important one is the overwhelming complexity of modern vulnerability management. The second part is scanners have become increasingly thorough. Even a small application if you point it at a scanner, it will give you thousand pages of reports with multiple issues found in it. Triage in them will take a lot of your time. Then there is an overwhelming you know maze of security data from let's say the project itself to the downstream vendors to third parties like OSV or you know NSD.

And in some cases for critical installations the government. So all of these are not the same. There will be some differences with each of these reports. If you are using a component you have to go go through all of them to get a clear picture of you know where you stand in terms of risk. This followed by the security data is kind of standardized but not really. There are many different formats which it comes with CSAF, Wex, Oval. What do you standardize on? What do you use? There is kind of a ping-pong between the developers and the security teams. What do you focus on? Do you fix to mitigate the risk or do you prioritize rapid feature delivery?

And you could just pick one. >> [clears throat] >> So part of the solution lies in project Hummingbird in terms of minimal images. Let's see what that is. So what does actually Hummingbird provide? It's a catalog of minimal hardened container images. Uh, what do we provide? It we provide language runtimes because you will be developing application in a certain language, right? It could be a .NET stack, a Go stack, a Java stack and so on. You will need a database, right? Of your choice. Of course you will be interacting with the web. That's where the web servers and proxies. And any foundational components that are required to build applications.

So is this just about an image? Well, everybody tends to think that the industry focuses just on the end result which is the image. Uh, but what we truly believe is the real moat lies in the process behind it, not just a end result, which is an image. Zero CVEs. Uh this is an important concept. You might have uh recurringly heard it in the context of some compliance, let's say FedRAMP or any compliance where this is really important.

But if you look at it, zero CVE is kind of a myth. Anybody who has worked in the security for some time realizes that this is a moving target. Uh you cannot have zero CVEs always, right? If you use a component and if it says zero, it doesn't have any CVEs, you've mitigated all of them this morning because of AI or other uh methods, by evening, you will have some CVEs in it. So, you're never really at zero CVEs. >> [clears throat] >> Okay. The real problem is just isn't just about the CVEs, it's about the trust okay?

Engineering teams or teams that are building the software need to answer this really critical questions. Where did the container come from? How was it built? Can we prove it was not tampered with? Or are we in a position to respond quickly to upstream fixes? These are far more important questions which show foundational security rather than just fixing or, you know, working on one issue. Uh there's a really great uh blog post from our Red Hat team. I highly encourage you to have a look at it. >> [clears throat] >> The second part is about maximum trust, which I feel is more important.

Uh before we go into trust, uh I'd like to introduce to a concept of reproducible builds. You might have heard about it earlier. Let's say you do a Docker pull or a Podman pull and just pull an image, right? You get a bunch of data, its dependencies, and everything on your machine. Um I would say, "How do you prove it's secure?" Someone might say, "You know, there's an S-bomb with it. I can just go and check, and it will tell me what all is there, right?" But you're not just implicitly trusting the image, you're implicitly trusting the S-bomb as well.

Nothing is stopping from an innocuous S-bomb showing something else while shipping something else, right? Uh end of the day, you're also trusting the S-bomb without verification. What reproduces reproducible builds do is it makes uh tampering completely impossible. Users can rebuild the entire images themselves and verify it bit by bit. What it ends up doing is establishing trust and verification of artifacts. Uh All of this is around a secure software factory, which Red Hat has started doing many years ago. It's built on something called as the Conflux build system, which is built on Tekton pipelines. What it provides is tamper-proof salsa, which

is a framework for supply chain artifacts, level three provenance. You could build uh and verify it uh because we are providing signatures for it. You get a complete uh S-bomb along with it. And we do a bunch of things in the background, which the teams would have to do it themselves. Like SAS scan, malware scan, CV scan, and all of that. Uh so, what's special about uh you know, Hummingbird images? We provide near zero CVs. Focus on the word near zero, not zero CVs. We of course provide what is needed for application development. We provide components that can be easily verified, and there is a huge community uh backing for this project.

This is a gist of how it looks like. I won't go too much into details. >> [clears throat] >> Uh before what's coming, all of this was assuming humans using AI, but we are rapidly moving towards a shift where agents are doing the choosing. They are choosing the image, they are choosing the OS, the libraries, the entire stack. Agent are producing the code, the software architectures rapidly. Agents build and run software themselves, and agents do not stop. This is the next evolution of project Hummingbird, which is Fedora Hummingbird Linux. I won't explain it here, but I will leave it to you.

And do we use rail versus Hummingbird? Well, it's best of both worlds. You choose what do you want, stability or compliance, and choose one of them. And here are some references. Thank you so much. >> [music]