#264 Nick Sullivan: Cloudflare – The Internet's (De)centralized Security Blanket

Transcript

[Music] this episode of epicenter is brought to you by Microsoft Azure configure and deploy a consortium blockchain network in just a few clicks with pre-built configurations and enterprise grade infrastructure spend less time on blockchain scaffolding and more time building your application learn more visit AKA ems / epicenter 5 welcome to epicenter my name is sebastian with you my name is sunny got there well hey honey how's it going going well happy

here yeah pretty good it's been a while since we've done this together yeah definitely I think I think mo is the last one okay you said you've done some grand episode since I congratulations on the coil episode it was terrific I I listened to it twice that's a good one yeah yeah so today we're speaking with Mick Sullivan and Nick Sullivan is the head of cryptography at Cod flair now CloudFlare is not you know one of your typical companies that we usually cover on the podcast it's more of a traditional Internet company but they're doing some like really really interesting stuff with cryptography that I really wasn't quite aware of before recordings podcast yeah and they're like

do a lot to like especially like help bring like a lot of like essential ization technologies like tour and I'd be a fast and what not to like sort of the masses like you know a lot of like blockchain companies are like you know building really cool tech but like it's really hard to get this stuff into the hands of like everyday users and you know cloud fairsley like making that a lot of this stuff a lot more accessible yeah absolutely I mean I think just the fact that a company like CloudFlare is is writing the blog posts now quite like long and detailed blog posts about what is ipfs and how they're using it and you know these posts are read by you know probably tens of thousands of people outside of the crypto space it's great

for the the system I think and yet make make was a great great great guest very articulate and I mean I really am a huge fan of like I've always been a huge fan of like decentralization projects that aren't necessarily like blockchain focus and so Cloud Fair has been working a lot with tour and IPF a so it's like you know really excited me yeah so hope you'll enjoy this episode with Nick and that we do have a couple of announcements so I think in the last episode I was on I mentioned that be at the hyper ledger Global Forum it is from the 12th to the 15th I'll be there on the 12th and the 13th and I have the discount code now and so if you're interested in attending it is in Basel

Switzerland it's in Brian's hometown and mayor's old city where he used to live I've never been there so I'm excited if there's a discount for 15% it is hgf 18 news so hgf 18 news you can go to the events page if you if you search for a hyper logical forum you'll find it and if you can't remember this discount code we tweeted it a few days ago so you can see it it was tweeted on on the 23rd of November so you can always go back to our Twitter feed and see it so hopefully if you're there come

say hi we've got to say to see you and sunny you mentioned you were also attending some events yeah so next month December 11th there's a company called Dora hacks which has hosted a bunch of like really cool blockchain hackathons throughout the world in in China and Berlin and Toronto and so they're actually holding their first event here in SF on December 11th it's completely free to anyone to attend I will be speaking and mentoring at this event so if you want to come hack with me on some cool stuff I definitely check it out the free and just look it up on Eventbrite Dora hax SF I got a question for you I've never attended hackathon mm-hmm

what would you what would you suggest like for some I mean I'm particularly like I know how to code I don't code like everyday anymore but you know at some point in my life I was like a front-end developer and I have some experience with stuff like node my smart contract development skills or in near zero but you know practically speak a you know if if I'm interested in like learning how to do things is is a hackathon a good place to just sort of like just you know jump in both feet first or is that not recommended for someone like me I would say that there's usually often like two classes of people who attend hackathons that's people who are there to like you

know to go win like they want to build a cool project and end up with something at the end of though at the end of the weekend and like actually let you know or there's often people who are there just to like learn something and you know I I've worn both hats throughout my hackathon career if you will and so you know some so sometimes like I know I go in with a project I really want to build and I'm like I want to get this done in this weekend and so I'll do that but then there's some times where it's like you know I just want to learn a new piece of technology and so I just like try to choose like a very very simple product project and that's like and honestly like when you at hack a Thon's like when you're experimenting a

new technology like anyone who's done this before you know that like half the time get goes into just like installing the software which is you know it's not not fun but yeah definitely like you know look at check out like tutorials and like you know I would say I would say you expend a lot of the hackathon instead of like building the product first spend like half the first half almost like going through tutorials and then the second half if you're feeling comfortable then start to like try to work on a project directly yeah right okay so it may be a good exercise to like properly take some time and go do some tutorials around people that can perhaps mentor you and the sort of thing a good opportunity to learn if you're not going to build like a proper project

and one of the you know at hackathons is often a lot of mentors there and I think that's honestly sometimes one of like the most underused like amenities that hackathon to offer so definitely talk to the mentors and then also you know especially doing one of the more learning style I really like to not show up with a team and I really like to like get to the hackathon and like find new people there to work with it just makes it a much more fun experience in my opinion cool thanks thanks for the tips oh uh I think I'll maybe I'll look out for some hackathons to attend then sure alright so without further ado here's your's Nick Sullivan of CloudFlare hi so rehearse today with Nick Sullivan who's

head of cryptography at CloudFlare Nick thanks for joining us today absolutely thanks for having me yeah we're really excited about the show when I found out that the CloudFlare was sort of dabbling with ipfs it led me to like do a bit more research about what you guys were doing in the area of cryptography and it turns out that you guys are doing a lot of really really cool interesting stuff and I'm always really fascinated when like companies and the more traditional web space and sort of intersect with like companies that were more familiar with in the blockchain space and projects and companies like in the sense of my PFS and so that's why it's really really happy to have you on so let's maybe start off by talking a bit about

your background and how you got involved in cryptography and how you landed it as a head of cryptography at Koffler sure well I have always been interested in math and mathematics and solving problems and puzzles and cryptography in general so when I went to school in Canada at University of Waterloo I did a pure math degree and was really kind of enthralled by the abstract notion of you know taking understanding the the mathematical world understanding how objects fit together how prime numbers worked how you could you could take something like as simple as as you know two and three and five and

seven and and you have this sort of infinite number of interesting problems and challenges to to go through to discover this end after I did a master's degree in cryptography I got into the computer security world and worked for a little bit at Symantec I some documents basically on the Internet's Internet security in general they have this thing called the Internet security threat report that kind of help analyze what's going on online online so my kind of two passions were the Internet and understanding you know what people are doing in this really kind of amazing interconnected network that we all enjoy as internet and and cryptography which is this the science

of secret information and so after after leaving Symantec I joined Apple where I worked on a lot of some sort of secret cryptography related efforts for about six years or so and eventually I learned about this company called CloudFlare which was a very young start-up at the time but was doing some really interesting things for example they had withstood what was at the time the largest distributed denial-of-service in history and so a lot of what CloudFlare was doing was really interesting to me because they were offering a free service to help accelerate the web as

well as protect it from threats and their kind of kind of at the center of of everything that was going on all online so when I joined cryptography I was the first kind of security engineering focused person at the company and I've been here for about five and a half years growing the team the company has grown tremendously since then we're now you know a big start-up if you will still private company but um so I started the cryptography team at Club Flair in order to use this really interesting tool which is cryptography encryption hash functions all this sort of really cool math science that that

lets you protect information online as well as provide properties like integrity and non-repudiation and I started building a team to help take cryptography and apply it to some of the bigger problems that CloudFlare was facing and to basically we spearhead new research in this area and so this is what I've been doing ever since so when you were in college studying or university as us Canadians say and study in cryptography in Waterloo and you're you know you know getting into the your career did you have any idea that like cryptography would become such an important thing like today I mean just just if you think

of block chains it's such a central it play such a central role and the functioning of that technology and also just generally the web did you do you think that this was something that would become so massively important for the world well it's very hard to heart to see what happened right it's hard to predict what happened like for example my thesis was on elliptic curve cryptography which at the time was barely barely ever used for anything in production it was sort of you could you could use SSL for your website right you'd have encryption for your website but everything that people were using was based on diffie-hellman and RSA which were the two standard algorithms developed in the

70s and elliptic curves were this kind of new thing and now this is actually the fundamental glue that holds together Bitcoin as well as aetherium and you know and it's also the the most fundamental cryptography for protecting information online when you're browsing the Internet so it was very hard to see at the time that you know that this interest of mine would become one of the one of the key technologies to enable technology in the 21st century could you like give us a little bit of a brief lowdown on like what CloudFlare is overall you know it's like you know it's our traditional blockchain companies or some listeners I'm sure I've like heard of CloudFlare but maybe don't know quite

exactly what they do and you know it's a relatively young company actually right like I think only nine years and like somehow it's grown to become this like almost like centerpiece like very integral part of the entire like web infrastructure so could you tell us a little bit about what are the different kind of things cloud Ferrer's working on and whatnot sure yes well claw muffler is a internet security and formants company the mission of CloudFlare is to help build a better internet and that's really what we're trying to do is folks who operate websites and who operate web services and who offer services online whether you're sort of a small the smallest sort

of individual hosting your own blog to a very large corporation and a large enterprise that has you know massive massive sets of customers and you know very very high requirements what kossler dozens just help make your site or your your property faster more secure more available and to give you insights so the way that Clough lured does this is using I guess the two main traditional protocols on the Internet HTTP or HTTPS the encrypted version and DNS so cloud ler has data centers distributed all around the world over 150 I don't have the exact number now but um and

basically every continent except for Antarctica and so the way it works is if you sign up for CloudFlare rather than visitors to your site going directly to your site which could have to you know travel across the entire world which due to speed of light considerations can actually you know slow things down you connect to the nearest CloudFlare location and if we have if you have sort of static content on your site we can serve it directly from there so we can also apply rules so rules to protect against different types of attacks so if you think of people doing sequel injections or cross-site scripting attacks or all these sort of web security things by being able to inspect

the traffic we can block these attacks and the part that's closer to to my responsibilities is that we also can improv provide encryption so in the early days of the web and and some of the more challenging things that a web administrator has to do is set up encryption an encryption security for your website so to move from HTTP to HTTPS you have to buy a certificate or get a certificate issued and you know manage the figuration and and do these sort of things that are better little tricky and Koffler makes that kind of dead simple and and handles it on your behalf so CloudFlare as a service has grown tremendously and one of the reasons for

that is that we offer a free service so there's over 11 million domains or so they use cloud slaves free service which is probably why so many people have heard of it and so yeah you can sign up for CloudFlare and get you know denial of service protections so if someone's trying to knock you off the internet I'll sit in front right and we can see in bad traffic and we can kind of keep you online while other people are trying to take you off and so it's it's great because having all of these different customers gives us some visibility into what's really happening on the internet and we take what we see from the general set of customers and and if you see an attack

against one customer you can use it to protect other people so it's a it's a real it's a real center of the internet kind of thing where things go through us and we learn about it and we help make the internet better and we're not only involved in just providing this service we're also we really care about making the internet scale going forward and to make make making the internet better so we're involved in standards for example TLS 1.3 which is the recent encryption standard for for websites we were closely involved with that and and my team we do a lot of research on the cryptography side to see what new ways we can we can change things so that in

the future using the Internet is safer and more secure faster than it is today and are using your own dark fibre between data centers no we use the we use the Internet which is why we we rely on strong encryption so much so every one of cloud fleurs data centers is independent and I guess you could say technically decentralized although administrative Lee sent relies and we communicate over the internet over different interconnections with network so Koffler is actually the most connected network on the internet we have more peering sessions with other networks then than anybody else online

yes I use cloud fair on a website and we use the the paid service and I also use it on some other websites like the free service and I kind of see CloudFlare as it's like nice blanket of security but that also provides like a bunch of optimizations like it serves your CSS and JavaScript super fast and your HTML and and it has these like built-in you know like this built in fortress like that you can call upon it will if you're if you're being attacked that you know sort of like come into action if certain rules are being are being triggered so yeah it's a it's a really great service and like no wonder that a lot of people are using it and it does show up in a lot of places on the internet I mean you

very often see CloudFlare landing pages and like CAPTCHA landing pages quite a bit online so we'll come back to the CAPTCHA thing but later but in September you and and some some colleagues of yours wrote a series of blog posts and we'll link to these in the show notes and I strongly encourage anybody listening to this to check out these blog posts because the really terrific so it's called crypto week so welcome to grifoll week in which you describe sort of all the different things that CloudFlare is doing with crypto with like sort of innovative stuff right so like with ipfs with a rest at or like DNS SEC and reading these blog posts I was like these these are great primers

for anybody that's really looking to understand fundamentally how this stuff works like how does it DNA like how is your HTTP request function like when you call a website like what who are the different parties at play here or where the trust points you know where the vulnerabilities and how is cloth layer doing it better so I thought these these posters are really terrific but in in this post you mentioned so the trust relationships that one has to engage in when using the intranet so whether that's like visiting a web site or you know chatting online or it's like using social media what are your thoughts about how we trust the internet sort of a broad scale do you think most people

have a good understanding of where the trust points are on the internet and if and if not how can how can companies like cloud fly or like help make that better yeah so I would say in general people don't understand the trust relationships online you enter in a website and you go to that website and it comes to you you enter in a host name or a URL and it goes through you click on a link or open an app and you just get content but there's a lot of interesting things that go on behind the scenes and a lot of these have to do with trust and trusting and and actually the implicit trust that is built into

the technology that you're using to browse the Internet to to show you what you expect and to make sure that what you're getting is something that you're intending to get and so there are a lot of parties that are involved in this and some of the very obvious ones are registrar's so a registrar is a company that you use to buy a domain name and so if you if you if you buy Google calm or mysite.com then you have a registrar and you kind of work with this registrar to to make sure that your website is advertised and your web registered is connected to a

DNS provider and so when you type in CloudFlare com into your browser behind the scenes you have to know what IP address is CloudFlare ramazan so there's this entire system called DNS which is a name system which which is managed by a lot of different entities around it's sort of one of the first decentralized systems or I guess hierarchical systems out there so you have to look at where who comments and then comm tells you who example.com is and then you talk to example.com and then it'll tell you what IP address you actually use to connect to to example.com so

from just a just a names to numbers perspective the Internet is is based on IP addresses your numbers DNS is kind of the phonebook that goes from your name to to a number other pieces that you have to have to trust involved when you're doing encrypted connections so if you're going to an HTTPS version of a site that site has a cryptographic key and this is embedded into a certificate and so they present you a certificate and you do this sort of handshake and then you have a secure channel and so one of the things that your browser has to do is know how to trust which certificates correspond to which

websites and this is a another system another sort of system of different organizations that make up something called the public key infrastructure and so your browser trusts a bunch of certificate authorities who are the only ones that are allowed to mint certificates for different host names and so the system has been around since the 90s and there's been some problems with it over time certificate authorities have been compromised and that's put a lot of people at risk certificates themselves need to have an expiration period or else you know certificates from the 1990s using old cryptography that's been broken would still be valid so there's there's a lot

of challenges with with trusting this and and and we don't even need can't even go into this even more but um but even at the lower layers of the internet IP addresses the Internet is a set of you know hundreds of thousands of interconnected networks that have to actually exchange data so when you when you're one network and you say hey this IP address - two - two or one two three four belongs to me then well you need to be a thorn you need to actually you know trust that when someone says yeah you know send that traffic to me that it actually belongs to you so there's there's multiple different layers and

and the the intra blog post really goes into this and in depth and so as a general user all of this is happening the scenes and and you really have to trust it there's you know there's the very minimal thing that you have in browsers which is that padlock which does imply some things it implies that you know the certificate it is that you're getting is valid for the site and this is the site the youth that you're trying to go to but there's a lot of threats out there and there's a lot of ways that people try to manipulate this and hijack this and you know steal people's traffic but um generally this is not a well understood thing by the public so companies like CloudFlare are

investing in various technologies that you know help simplify this for folks like help make it so that if we are connecting with other entities around around the internet that we can trust them and we have to agree on protocols to do this and define these protocols and implement them and get everyone to kind of agree on standards and so that's that's one of the that's one of the interesting organizational challenges and inter-organizational challenges that that we have to deal with right now but luckily for our security and for people's privacy online is that there are a lot of organizations who do care about this and who you know are impacted

when malicious things happen so companies like CloudFlare and others are working to help improve the situation if you've listened to previous episodes with marlene gray and matt Koerner you know that microsoft is committed to providing enterprise-grade tools and infrastructure for blockchain developers well the azure blockchain workbench is perfect for organizations building consortium networks take the etherium proof of authority template for example it's ideal for permission networks where consensus participants are known and reputable aetherium on Azure has on chain network governance that leverages parodies extensible proof Authority client each consortium member has the power to govern the network or delegate their consensus participants to a

trusted operator and parodies webassembly support allows developers to write smart contracts in familiar languages like C C++ and rust as your blockchain workbench was created on the same principles that drive all production services in Azure so you know you're relying on secure redundant infrastructure that can scale and with built-in services like authenticating api's off chain databases and secure key management services you can scaffold your infrastructure in just a few hours to learn more about Azure blockchain workbench and how Microsoft is advancing blockchain usability and enterprise check out a kms epicenter and start building today we'd like to thank Microsoft Azure for their support of epicenter a lot of these authorities that you mentioned like you

know for example the certificate authorities or you know you mentioned DNS is like a hierarchical system where are these where do these authorities come from like sort of who decided them and like you know was it just happened to be like all the companies were around like back in the 80s like they just happened to like be grandfathered in or how did how does that process work well yeah the Internet has evolved over the years in various different ways and you know original we can go into like the origins of the Internet as a DARPA project and and the the switch to tcp/ip in the 80s and the the evolution of the DNS but it's really sort of happened organically over time and then some

organizational bodies have been put in place to help guide this and so for example internet protocols there's a volunteer group called the internet Engineering Task Force IETF so if you've heard about rfcs when people say oh RFC whatever whatever this is a certain protocol like DNS is is a set of RFC s that's what the IETF does there's I Anna which is an organization that is associated with on with managing names and numbers and they have lots of processes around that there's I can there's a there's the set of regional registries so there's a the entire IP space North America has a group called

Arin and they distribute out the IPS to different organizations by different bids so these these are often organizations that are a mix of profit nonprofit but generally have a mandate to be good stewards for the Internet and to to make sure that this is this technology that we all rely on is something that is available for everyone in the world that enables kind of equal access and that you know continues to to grow in terms of having both commercial and non-commercial uses so one thing I find interesting is often when people are talking about like cryptography / blockchain things there seems to often

be like three somewhat independent goals that often get like correlated together but I think actually should often be thought of by somewhat independent and I think the three here what I see is like privacy security and like decentralization and the third one desensitization is just like very vague kind of concept that came up in the last few years along with the blockchain space and so you know the reading through your blocked through your blog post are the welcome to crypto week you talk about like you know a lot of the stuff about like the immutability that ipfs provides which kind of goes along with security you talk about the privacy that tor provides but not too much talk about like decentralization and so you

know would that be a fair characteristic to say that like when you guys are approaching this like cryptography on the internet you guys are really much more focused almost like you're willing to accept this like these like authorities and centralization that exists on the internet but are trying to focus primarily on improve and almost you know like kind of becoming one of the central authorities on the Internet but really trying to focus on pushing the security and privacy side of things would that be like a fair characterization well I would say that CloudFlare is trying to serve its customers and cloud fleurs customers are not only websites and web services that use CloudFlare but you think of the users of the internet as a whole and if the internet becomes more functional and

if people are happier online and are more likely to to do business online then then it leads to the growth of the entire industry so security is one of the one of the very very very most important things for the company is if you get hacked or somebody steals data from your website or someone tries to mess around with your users this is going to impact trust and it's going to impact the bottom line for a bunch of businesses and and same with privacy if you if you think of how people are really waking up to privacy online and you know what you share and what the motivations of organizations that are based on monetizing individuals actions online have done and how that

how that's grown I think it's it's a it's another really big really big salient thing to human so security and privacy I think are things that human beings understand and relate to and businesses understand decentralization is is it's more of a more of a second order goal right I mean if you don't have decentralization you have these if you have if you have sort of fully centralized systems you have these really really inherent risks to to to your system so if you sort of think back to the mid 20th century the telephone system in the United States there was a bell had this massive monopoly over over

over the way that communications telecommunications happened and that led to a lot of really fascinating and amazing innovations you think of the transistor like a lot of radio communications and and all the sorts of amazing things they they created and they actually did connect everybody online but but until bail was broken up we didn't have this this ability for all of these internet companies to to kind of come out of nowhere and be able to to compete with each other so you have centralization and I guess if you think in the corporate terms monopolies are are ways to ways to kind of build wealth and and make something really good but

it also leads to the ability the tendency to kind of abuse the abuse power and having a diversity of participants a diversity of views and a diversity of components in a system and I guess decentralization is one component of that is is I guess a result of having a lot of different participants is is something that actually really helps innovation helps competition and helps things grow so it's it's less relevant to individual customers and people but I but it's it is a second-order goal and it is something that that we think about as well and when when talking about the cloud computing space and how people are

running services we we do worry about companies that are kind of massive central points of lock-in right I mean if you think of the AWS reinvent conference is going on this this year it's the largest trade conference in United States and that's that's a company that you know wants everybody to put all of their computing workloads on to a single company and there's there's a lot of lock-in associated with that so I think from a cryptography perspective decentralization is important but I think also from a business perspective having a lot of different options is important for a healthy ecosystem yeah

there's a thing that you might be familiar with with the Zuko's triangle Zucco Wilcox Ahern the founder of C cash and the sequel's triangle looks like you have security decentralization and human readable names and I think there's a lot of overlap with with this question here where I think like I think user experience also plays a big role or should be considered in like how we build systems and so if you have a system that's like secure and and easy to use but where you don't have this robustness which is it's meant to be brought on by the centralization then really you know you you might have to choose between two of those three points

on the triangle I don't know if if someone will actually solve that but it seems difficult yeah it is difficult and there are there are trade-offs and you can make in any one of these little corners and and finding the right ones are fighting the right trade-offs are hard to do but um considering where we are as a staff there's there's always improvements to be made to try to you know square Zuko's triangle if you all yeah so let's move on to the the core topic today we want to bring you on to discuss and that's ipfs so within this crypto week series of blog posts there was a two blog posts

about IP fs1 that sort of explained what IP FS is for the for the you know the average person who doesn't you know necessarily know but idea this and another post that described this experiment which based on this concept of end to end integrity so could you describe like wise cloud for experimenting well yet with ipfs and what would you guys what are you guys doing here yeah so I think one of the important things that clouds are trying to do is to well as I mentioned make the internet better but one of the aspects of this is connecting users of the web to some of these new networks that have values and have property is that the

current web doesn't have an IP FS is one of them as a content addressed network every piece of content has a hash as a specific unique fingerprint associated with it and unlike the web where you look things up by a names with ipfs you can look things up via a fingerprint of what they are and so the the traditional web is is is not necessarily immutable you you have different things that can happen you have a lot of very dynamic web pages and you you have services like CloudFlare that can see and detect things going wrong and sort of modify and optimize things on the fly

which is great but with ipfs there are certain use cases that people have for this where it they they just want things to absolutely be guaranteed that you're getting exactly what you were sent and if you think of things like package managers or image sharing or things like this where you have something that's static that's never going to change then ipfs makes a lot of sense so the ipfs gateway though is is the I guess the first of what we're calling the distributed web gateway which is a way to access ipfs as a network through HTTP and so people

have web browsers people don't necessarily um on I guess the broadest sense there's a lot of there's a lot of experts and and and people who are interested in the space who are really keen on on these decentralized networks who run nodes and and are happy to do these sort of things but the general populace has a web browser and they know how to how to use a web browser and so what this gateway does is allows people with web browsers to you know connect directly to ipfs and because ipfs is static cloud flairs a really really great service for that because we can do caching we can keep copies of data really close to people we can distribute data all around the world and so you mentioned the experiment that we did

which is a browser plugin for intent integrity i guess one of the the purists complained about having a gateway to something like ipfs is that as a question is you know what if the Gateway changes the value I mean the the value or it changes the content and the value of ipfs is in the fact that its content addressed so if you build a website it's guaranteed to be the same for every single person who sees that there's no censorship there's nothing like that it's it's just you publish one thing once and then it becomes you know there in the universe forever and this is why it's called the interplanetary file

system or one of the reasons is that you know publish something once it's available at all times and if you have a gateway HTTP is as I mentioned it's not really based on this intent integrity concept but with an IP FS gateway you can put the hash as part of the URL and with this extension you actually can validate that that hash in the URL matches the that you expect and the way that it's actually chained into way that it's actually chained together is with DNS so if you have a have a website you can say in the typical sense you have here's my hostname and it gives you an IP address

and that gives you the address so this is this is about routing with our IP ff6 parent you have this is the hostname this is the hash that represents the content on this website and so what the browser extension does is it just valid make sure that you know what you're seeing on the site matches exactly what was published in the DNS and it's kind of ties in with our with our other efforts of the week especially DNS SEC which is just signatures in the DN in the DNS itself so if you trust the DNS and you've trust the DNS central authority then this is a way to you know put the put I P FS into an existing system to help kind of validate the

integrity from within the browser how is the adoption been of this CloudFlare like I PFS see you know you can always consider like I'd BFS as a CDN of sorts like a content delivery network and so have you seen that like cloud flares offering has like helped increase the adoption because you know I actually tried to put my website on to ipfs it's been a while a she's probably over a year now so the technologies have a little bit more immature and you know I had a quite a hard time doing so and so like you know you guys have built a lot of a tooling to make this easier and stuff has been like the public reception and stuff to this yeah I think people are really excited about the IPO fest

gateway and they're really excited because of the possibilities that that it unlocks and the content hosting site on ipfs I agree it's it's relatively immature so if you want to host something on ipfs you can you know host it from your your your local laptop or you can use one of these services that's a pinning pinning service but um well yeah the the publishing side of it I think needs need some development but um actually integrating the access side is is where the Gateway really shines so we've we've seen all sorts of different customers or websites or properties or that that

really you know believe in decentralization and believe in having a source of truth for their data that is distributed beyond to their own data centers this is actually good for things like disaster recovery and they need a way to bootstrap their app or they need a way to bootstrap their application and like the fundamental you know belief is that you know we want to build this in a distributed way but we don't necessarily you know it one of the drawbacks of ipfs as it is is is it's relatively slow to actually get content and so having this gateway is a way to to speed things up you get all the benefits of CloudFlare

in front of this network that you have integrity protection and you have decentralization so it's it's been coming up we've we've definitely seen a lot more adoption since since we launched this several months ago and and not just from the distributed application space but also from more traditional companies as well that have have have an interested in decentralization so if I could just sort of rephrase what what you guys are doing here is those different components I think that need to be separated out so yeah the first is a Nike FS gateway and there are tons of ipfs gateways out

there and I think most of our listeners are probably familiar with them so there are these websites that you go to this URL so example - gateway com I think is one of them you go this website you you pop in a you just add the like EFS hash to the URL and it serves like this gateway is in the backend connected to an IP FS node and it is serving to you the content on the ipfs network and then the vulnerability here is that you know perhaps this website is is sort of doing a man-in-the-middle type of attack where it's serving you another piece of content than one you initially requested and you haven't really no way of doing there knowing that unless like you you know do like an mp5 or they verify the hash of the

content once you've downloaded it they verify that it matches with the hash of the address yup that's right well you guys are doing is like a step beyond that you're actually putting one of those gateways in the CloudFlare sort of wrapper so all the ipfs all in content I like EFS is now available superfast in one of these 150 data centers that you mentioned earlier yeah that's right so that's the the CloudFlare ipfs gateway is yeah it's like you take any typical gateway and then you clarify it okay so so that's great because all of a sudden you have this really fast content network it's content delivery network that's serving up ipfs content it's kind

of similar I guess like it reminds me of this project we had a few weeks ago called blocks route there's this like content delivery networks for block chains but yeah it's sort of similar that but then the issue with that is that if you're using it and maybe you trust example Gateway calm because you know some nice crypto persons hosting it or I don't know something you made me that a conference is hosting it you might trust that person the issue here is that people might might not trust CloudFlare where at least CloudFlare would like to prove that the content that they're delivering to you is actually the content that you requested so what you build here is in a browser plug-in that checks the ipfs network and make sure that that content matches the

hash that you requested well more specifically it checks that the Hat if you're using the clever gateway as you know CloudFlare - ipfs comm slash your hash value it checks the value of the content against that hash the the really really cool part that I didn't actually go into detail but um so just sorry so he checks the value of the content against the hash so it doesn't the browser it there's no like browser right it doesn't sort of like go and do an IP FS request you know a parallel request to verify it checks the hash and it does sort of the the hashing algorithm internally and verifies that it matches it's kind of like doing an Emmy and md5 verification it's it's it's like md5 but better hash

function md5 is a little breakable right now well the other really cool thing that you can do with Cobblers gateway is bring your own hostname so rather than have CloudFlare - ipfs comm slash whatever you can just have you know my website comm and you just say my website comm is on ipfs here's the hash of the root file of my website ok so this is the third thing I want to talk about so there's that there's the the Gateway there's the verification tool but then what wraps us together is saying ok now as a CloudFlare user as someone who has a website hosted on cloud flower-like like epicenter for instance what you can

you can you can set this thing up on your website or in your cloud floor account the Telus CloudFlare index might like your guys are running an IP FS node and and it creates basically a copy of your website and all the web pages on your website static content on this ipfs node that is now available to the entire ipfs ecosystem yeah it's it's sort of like that it's more it's more that you have to put your content onto ipfs someway and if someone tries to access your website through cloud layer we will fetch it from ipfs so CloudFlare as a service doesn't host content and and

this is this is sort of a very important key part of what clauser does is we cache content and so we need a place there's sort of some route of source of truth and so if you're going to use this service you can run a local node on your computer and say I'm going to host them here and we will grab it from there we'll keep a copy around as long as we can and and and serve it from cloud flares cache alternatively you could pay a service to keep a copy of your content on ipfs and that's that's the host and then CloudFlare just goes onto ipfs fetches your content puts it around the world and anybody once it can can get it through us that make sense yeah yeah that make sense we're not actually hosting things on

ipfs although if you fetch something through ipfs our node will have a copy of it so it actually helps improve the the duplication of content in ipfs which is really important because if there's only one copy of your content in the ipfs network then it if that that copy goes offline then you know the contents no longer available and so essentially what you guys have done is you know allowed for that third part of this like project it you kind of allowed ipfs almost integrate well with the existing DNS system right so I can now like I can have my website accessible by ipfs

hosted website accessible through like my own personal like domain name but still going through cloud Flair's like ah CDN yeah that's right and because we are so good at issuing certificates and kind of managing that then you also get get encryption for that website so sort of automatically how do you like see like the future of like ipfs do you see like it being like sort of a complementary service to like HH or protocol to http or more it's more competitive do you see like you know maybe websites will be served over HTTP but like certain assets on the website are over IP fast how do you see this like amalgamation of these two protocols

going forward it's an it's an interesting question because nobody really knows the hope is that you know ipfs provides a specific niche and a specific property that HTTP doesn't and that the expectation would be that they would both kind of live in parallel you can't necessarily do a lot of dynamic stuff with IP IP FS but but the integrity protection that it has and the actual distributed nature of the hosting I think makes it useful for specific applications so I think you'll find applications that are mostly HTTP applications that are mostly ipfs and applications that are sort of a a mix of the two and it really depends

on how well browsers and other other technologies adopt this so if you have like a mobile app that has native ipfs support or a mobile sdk that comes up with that native ipfs support then maybe it'll it'll become more popular in apps that would need this but um yeah I tend to see them as complementary they both have their advantages and disadvantages yeah I was I was speaking with Juan Bonet at level 3 I kind of bumped into him and of asking him about about this very thing and from his perspective I mean like browser support is at least partly possible so I guess like chromium is supporting it now and like their dev

versions and maybe Firefox will support it soon or I can't remember exactly but you know browser support is coming and I was actually quite surprised to see the how fast that had come I mean when we had him on I guess was episode 100 well 160 3 weeks ago or something like that I thought this is the years to get integrated in the browser's but it seems like it's moving much faster than yeah my understanding is that the path that they're taking is first exposing ipfs as a first order protocol so you have HTTP colon slash slash whatever you're gonna have IP FS colon slash slash whatever and the transition to get there is is that if you have ipfs colon slash slash

you can you can register a plug-in that is able to handle that for you and that's that's sort of the first step and then eventually down the line the ipfs node will potentially be native in the browser but right now it's it's all browser extensions so what did you learn from this well I guess we learned a lot first that latency is is really an issue when it comes to user experience so if you are the first person to ever fetch something through the IP FS gateway then it has to go back to our node and then our node has to search on the

internet and find it and then get the copy and then send it to the cache and then it can eventually take you know a long time for content to show up and so for certain applications it's the user experience is potentially problematic if you if you don't have a lot of caching and a lot of ability to serve things immediately the other thing we learned is that you can build some pretty interesting unexpected applications even on a platform that is for essentially for static content as ipfs is in its current incarnation so one of the

examples that we did with the ipfs gateway is we built a searchable mirror of wikipedia so you can actually link to this site that's on ipfs and you can build essentially search type capabilities into ipfs because a search is essentially a table which is a static file and then you can you can put JavaScript into there so you could you can do some really cool interactive things with ipfs it's not just about serving up static images it's uh it's a fully fledged platform so I think those those are the two things that that we learned about ipfs the other thing that we learned is just just the the interest

in this area is huge a lot of people are really trying to figure out how to engage with and take advantage of and you know have reaped the benefits of new technology and that that provide provides new features like having having resilience to single to failures is a big thing having integrity and and and people are really really thinking about trust and hosting websites and hosting web services and and running things

on line it's it's more and more important for people to to be able to trust what you're doing and as the infrastructure grows there's just so many more participants that it's hard to heart to to actually you know implicitly trust everything that you're doing online so we have to have to build these technical measures and and there's a lot of interest in this from watts Michael's cool so um what are my favorite stories actually regarding like ipfs and gateways um is I was talking to Jeremy Johnson from protocol labs about a year ago last DEFCON DEFCON 3 so November 2017 and this was like right around like

right after like the whole like Catalonia and Catalan referendum around independence I was going on and so what was happening during that process was the Spanish government so you know there are a lot of like Pro referendum websites people like websites showing people like how to go vote and like you know just like reasons why like you know just general Pro websites and the Spanish government was like sort of censoring these and shutting a lot of them down and what was really cool was um ipfs was actually being used to keep some of these websites up and so people were like hosting them on ipfs and i thought it was really cool because it was one of like I don't know I think one

of the first times that like this generation of like decentralization technologies has really been used to like cause ago physical like a tangible impact on like current of unlike world on world politics or whatnot um but then there was something interesting happening where the websites were being hosted on ipfs but everyone was accessing them through the ipfs IO gateway and what the Spanish government essentially ended up started doing was actually censoring the ipfs not IO domain gateway and so now people weren't and most people weren't even aware of any any other gateways and people didn't have the soft you know it's not easy to

install the IP fi software and so it just suddenly became very inaccessible to them and so this kind of like leads into the other one of the other kind of centerpieces of your krypter week that you had was about tour and so how do you see this like interesting relationship between ipfs and tour and like what can I be a fast gain by being served over tour yeah so I I think I think of tour as in the same family of technologies as ipfs and and a lot of these new blockchain distributed web type technologies because it really is a lot of independent nodes that work together to provide a property that you

wouldn't get with a with with the regular web so with tor what it does is it provides you with routing anonymity and it uses a layered encryption approach to do so and and and in terms of their trade-offs latency is one that they they just don't really care about it's actually anonymity is much more important than than getting things quick so the the typical web I mean the the unencrypted web and potentially even ipfs if you're talking about distributing this content it's the opposite of an anonymous right you're you're connecting directly with another person and requesting a very specific

thing and they know what you're asking and they know who you are but but it provides integrity so you have one network that provides integrity and one network that provides anonymity then it's it sort of makes sense to if you want both you can kind of put one on top of the other and what CloudFlare launched during crypto week was essentially a way to access the Tor network it's kind of like cloud ler put an IP FS node into the ipfs network cloud flow put a tor node into the Tor network as well and this Tor network tor node is is used to route any traffic to any site that's on CloudFlare so if you connect through cloud flares tor node

which is a dot onion address we've got about ten of them if you connect to any one of those and make a request for any site that's on cloud layer it kind of goes through and so the yeah the the the bottom of the diagram that I think you're you're referencing on the page shows a user going through tour and then connecting out the tour point through CloudFlare and then to the CloudFlare ipfs gateway and then to ipfs so I think if you're doing so you're going to get a very slow connection but it's going to be very private um even CloudFlare doesn't know who you are but but you also gain you know the antenna integrity properties of ipfs so I think they're pretty cool

complementary technologies if you're okay with things being extremely slow I see so this whole like Onion Routing service that you guys built that week you know I know in like the past year especially on like hacker news and stuff there's a lot of people like to like lame cloud flares like like especially the recapture features for something sort of like the degradation of the user experience on tour III always thought that it is a bit of like an unfair blaming but could you explain a little bit of why this whole recapture system is like so necessary in the tour in tour and then how your Onion Routing Service protocol like helps resolve some of those pain points yeah absolutely so as

I mentioned people come to CloudFlare for security insights acceleration things like this security is one of the main things and if you talk to you the average webmaster or the person running a website they actually don't really have a very favorable opinion of Tor because as an anonymity network it's very easy to send abusive traffic through it and not have to deal with the consequences so a lot of the traffic that actually comes through tor and comes through exit nodes is is attack traffic and we it hits our Web Application Firewall and we say what is this and and sort of block it so the way

that the clef ler is currently set up and and we're hoping to improve the system is is to use something called IP reputation and IP reputation databases help make a determination as to how likely a phtt request is going to be malicious or not or part of a flood or not is this an attack or not and so the what we do is is we use a CAPTCHA to kind of prove that it's a to force the user coming through to force to prove that they are a human or at least able to solve one of these human interaction puzzles and sort of once they prove that they're a person then we say okay great

you can come through do whatever you want with this website but where you're coming from seems to have a lot of bad requests and so the kind of danger level gets elevated and and this is something that our customers expect is that they have to pay for bandwidth they have to pay for you know what it takes to administer a site and run it and deal with comment spam and and deal with all these sort of things and and you know this IP reputation is a very coarse way of lowering the amount of crap that you get if you will on the site so because of how tor works is that there's there's a couple there's a small set of computers that are called tor exit nodes

where the traffic goes into the Tor network and then exit out of those exits out of those nodes into the internet these IPS tend to be given a pretty bad reputation because there's so much bad stuff coming from them so this is kind of the crux of the reason why people see so many CAPTCHAs while using tor and why CloudFlare is is sort of being blamed for the degradation of this network and we didn't like that we think that you know towards a valuable tool we still need to protect our customers from attacks and we and like these are just this is this is who we're building the service for and this is these are the people who you know we want to use CloudFlare we still want to give them

that service but we also think that the the secondary effects on the Internet as a whole its are important as well so having more people use an anonymity network make having people use gain these properties of these alternative networks if they choose to use them and not be punished for it is is something that we're really interested in so what our tour gateway does is it allows folks who are browsing websites on tour to actually access CloudFlare websites through as I mentioned a node that's running in the Tor network that has an onion address and I guess every time

that you connect through the Tor network to an onion service you connect through a circuit so there's an entry node there's a transit node there's a third node and then you then you can connect to the site so every one of these circuits is unique for every person and when you run an onion service you actually get a circuit ID you get to know whether or not two different connections to the same to the same service are from two different people and because of that you can actually apply policies on a very selective basis right so if someone is actually sending a lot of comments BAM then you can say you know this circuit the circuit is bad we can block this without walking legitimate people and I

think this is this is one of the one of the great things that we helped put together with this with this tour thing we work with the tor browser team as well to help implement this so if you visit a site that's on CloudFlare we'll send an HTTP header that says hey by the way if you want if you're going to reconnect we have all these onion addresses and you can just use these and connect through tor instead of connecting through an IP address and and this has been very very successful actually we turn it on for all CloudFlare sites with all of this and I mean you guys are it seems like quite quite involved in so the open source space in fact sunny was mentioning

earlier and I wasn't really aware of this but you guys have quite a few crypto libraries that that are open source in fact some of them are being used by aetherium and a bunch of other websites like the most happy internet is using your comfort libraries how does this and this experimentation with ipfs and this Torah stuff you guys are working like how does this all fit into like you know your business model is there other specific businesses here that you're looking to develop or is it more just sort of being at the cutting edge of these technologies and allowing for the you know the experience of everybody using the web to be improved well it's part of the mission statement of the company which is to help build a better internet and open source is is something

that's core to to what we are I think cloud flairs doesn't necessarily have secret sauce in the software right almost everything that we use we try to open source because it will be usable for for other folks online so for example four years ago we released a library called CF SSL which was a go based certificate authority and you can use it to you know build certificates and build a PKI inside your own organization and it actually got picked up by let's encrypt and now it's the core of the let's encrypt certificate authority as well as you know sales force and a bunch of other really big

companies are using it and and we've contributed code to the NGO standard library so the P 256 which is one of the most well commonly used elliptic curves one of cloud flares engineers you know we optimized it because we do so much cryptography so you know why not share this with the world and I think it's it's um there's no drawback for everybody having a better version of cryptographic tools and if you have a faster library that's secure and safe put it out there for people to use I guess so so far we've talked a lot about you know two major

decentralization technology which is ipfs and tor the one that we haven't really talked too much about yet which is probably one of the you know ones that's the most well known is block chains right and so I was wondering how do you guys think about block chains you know I know you have this one protocol you kind of like dubs clock chains like as a joke but you know in that one you talking about like you know a timing system for like SSL Certificates so you could you know like you know synchronize clocks but you know another option is I actually worked on a project where like you know instead of like doing SSL Certificates expiration you can do a system where like you know a

public a blockchain acts as a public bulletin board where you like lists expire are like compromised signing systems another use case for I think like block chains within Europe within the web infrastructure is throughout this entire thing you guys have talked a lot about like using the DNS system right so you talked about how you're using DNS for like the ipfs like resolution or you know with the your you have this other project called encrypted sni which you're trying to basically create like a PKI and so you know you're using the you're kind of using the DNS system to do that as well and like we mentioned the DNS system is a very like hierarchical system have you ever

thought about maybe exploring the option of using block change to do so but you know so we mentioned Zuko's triangle as earlier as well and so you know the cool thing was so sucrose triangle was this whole thing about human readability and centralization and security but Aaron Schwartz actually had this like you know he actually made this observation that a blockchain actually is a way to get around as you goes triangle and so that kind of led to projects like namecoin and handshake and things like this so I guess my overall question here is like how do you guys think about like integrating like blockchain technology into some of your offerings or into just in the general web infrastructure as a whole yeah so I think there's there's another kind of trilemma that our CEO

Matthew Prince put out in a in a blog post about tor a few years ago about you know making things usable secure and and having low latency and I think when you're in the web context this is something that's very underrated is the ability to get things fast and to get things immediately and so when it comes to certificates and time and a lot of different things if you're connecting to a website 100 milliseconds is going to kill you and so there's there's a number of initiatives that we're interested in that are locked

chain ask that are blocked chain that sort of seem blog shape blockchain asked and one of those is certificate transparency and so one of the I guess the one of the main differences here is that in a lot of the blockchain technologies that we're talking about it's we're talking about fully trust list decentralized systems where you have a lot of different peers and then you have to this is this is why consensus is so important is being able to have all these impairs and all sort of agree on a specific thing I think in the web PKI and at least in the web website situation that's fine but that's sort of a step too far or at least it's

a step that's a little bigger than the technology is willing to take us right now so um a certificate transparency is an example of of one step so it's a it's essentially a hash tree of all the certificates that have ever been issued and for certificate transparency to work you need independent groups to manage these these certificates as well so you end up in something that's sort of the analogous of like a permission block chain and with certificate transparency you have to it you actually don't have to do the lookup on the machine and you don't have to run a node on your machine and you don't have to synchronize with the blockchain and so

the cost of latency to a system like this is not big enough to to slow its progress so I think that the main challenge for integrating web PKI traditional web technologies and block chains is really about being fast and being able to synchronize things fast and being able to transfer data fast and be able to have fast consensus and having a fully trust list system is is not Sara Lee conducive to that although we've seen some some pretty good experimentation experiments in that direction I see cool and earlier you had mentioned that this ipfs gateway is just one of

the first projects in this larger decentralized web gateway series of projects almost so you know what are what are some of the other projects that are like you have planned in sphere of decentralized web one that I thought I think it would be really cool was like it would be like you know maybe in your cloud fair DNS like 1.1.11 maybe integrate like namecoin resolution well I thought would be a really cool idea but I don't know what are some of the other ones that you guys are thinking about so we've talked to the namecoin folks we've talked to folks at aetherium we're we're really kind of testing the waters at this point and right now we're mostly investing in you know how can we make the ipfs gateway better and that's what the short term roadmap looks like

but but down the road there's so many interesting technologies in this space solving different problems and you shouldn't be surprised to see any one of those pop up down the line yeah so you mentioned one dot one dot one dot one and so that's a basically a free DNS service that you provide so I guess similar to like Google DNS or Open DNS so like that but but with with privacy apparently and your I was reading your website earlier and I guess KPMG is auditing your servers to make sure that you're not actually like logging anything and so privacy is sort of a big deal here I'm curious like what goes into buying would people sort of

hear about your flipping domain names and paying and enormous amounts of money for domain names what goes into buying the IP address that went up 1.1.1 well we we didn't buy one dot 100 101 it's actually um I mentioned how there's different registrations the different authorities and manage IPS and the one space is actually owned by AP Nick which is the asia-pacific region for distributing IPs and they they never thought that they would be possible to even to give this IP address to anybody because it was so so bad in terms of the amount of garbage traffic that would come to it so anybody

who's building any sort of test for an IP address in any documentation it's gonna say 1 1 1 1 it's just the simplest example that you can you can use so there's an enormous amount of background internet background radiation hitting the 1 dot 101 IP address that they were sort of like we can't allocate this there's no there's no reason anybody would ever want to use it it's just so it's basically constantly under DDoS from from just the the background internet radiation and cloud ler was one of the organizations that in the world one of the few that could actually you know that's no no big deal to handle a bunch of unexpected traffic so we made a deal with aap neck and and they're lending us the IP address for for the

for this project and and and it's been a pretty fruitful collaboration with them so far and a really successful project that's pretty funny so it kind of shows off your DDoS like capabilities as well protection capabilities yeah absolutely and one other thing we should mention earlier but I guess in your office in the maybe like in the lobby or something there's a bunch of lava lamps there like generating entropy can you tell us a bit about that sure yeah so anybody who saw the first episode of NCIS this year might recognize they kind of stole the plot idea from from cloud Flair's office

but yeah so we have a wall in our front lobby that has you know about a hundred lava lamps and we record it with the with a digital recorder and we turn that data stream into a source of random numbers that we actually you know send out to our data centers and our and our servers and feed it into the as an additional random source so is there any academic research or anything like that that the switch suggests that lava lamps are actually random well lava lamps themselves are are pretty unpredictable the main thing is it doesn't really matter if you have a sufficiently advanced camera there's going to be

enough noise in in it to to actually to actually create enough entropy to it to be a useful useful source and also the lighting is also is a big part of it at any time of day you're gonna have different different sources of light and people walking in front of the camera and there's there's enough entropy and in like an HD HD film to to use for a lifetime right and I'm sure the temperature fluctuations in the room also affect the lava lamp as well yeah it's very hard to predict the levels of the lava lamp but it's to protect everything else all the other atmospheric conditions basically

impossible and even if they were predictable we mix it in with other sources such as hardware random numbers okay well with that Nick I want to thank you for coming on the show today it was a fascinating discussion and I look forward to seeing what comes out of clouds flow in the future I think now that things are so easy thanks to cloud flare might like look into making our website available because an onion domain or like available IP FS you know do something like that yeah absolutely thanks for having me on thank you for joining us on this week's episode we release new episodes every week you can find and subscribe to the

show on iTunes Spotify YouTube SoundCloud or wherever you listen to podcasts and if you have a Google home or Alexa device you can tell it to listen to the latest episode of the epicenter podcast go to epicenter TV slash subscribe for a full list of places where you can watch and listen and while you're there be sure to sign up for the newsletter so you get new episodes in your inbox as they're released if you want to interact with us the guest or other podcast listeners you can follow us on twitter and please leave us a review on iTunes it helps me behind the show and we're always happy to read them so thanks so much and we look forward to being back next week [Music] [Music]