The Modern .NET Show

S08E22 - From Pentest to 1.7 Million Downloads, Part 1: The Headers I'd Never Heard Of

Sponsors

Support for this episode of The Modern .NET Show comes from the following sponsors. Please take a moment to learn more about their products and services:

Please also see the full sponsor message(s) in the episode transcription for more details of their products and services, and offers exclusive to listeners of The Modern .NET Show.

Thank you to the sponsors for supporting the show.

Embedded Player

S08E22 - From Pentest to 1.7 Million Downloads, Part 1: The Headers I’d Never Heard Of
The Modern .NET Show

S08E22 - From Pentest to 1.7 Million Downloads, Part 1: The Headers I'd Never Heard Of

Supporting The Show

If this episode was interesting or useful to you, please consider supporting the show with one of the above options.

Episode Summary

In this solo episode (the first half of a two-part story) Jamie steps back from the usual interview format to tell the story behind OwaspHeaders.Core, an open source ASP .NET Core middleware that has now been downloaded over 1.7 million times.

The story starts in 2016, with a failed penetration test. Jamie’s first solo project at the company came back with pages of CRITICAL findings — almost all of them for HTTP headers he had never heard of. X-Frame-Options. Strict-Transport-Security. X-Content-Type-Options. The documentation he found at the time assumed he already had the vocabulary. The gap he was standing in didn’t even exist on the page.

This episode lays the groundwork for why OwaspHeaders.Core exists in the first place. Along the way, Jamie covers:

Featuring quotes from previous guests Tanya Janca (episodes 77 and 105, and S07E11), this episode is the why behind the package. Part two tells the story of the how.

Episode Transcription

Hey everyone, and welcome back to The Modern .NET Show; the premier .NET podcast, focusing entirely on the knowledge, tools, and frameworks that all .NET developers should have in their toolbox. I’m your host Jamie Taylor, bringing you conversations with the brightest minds in the .NET ecosystem.

Today’s episode is a little different from the norm. I waned to, temporarily, take you away from the “AI is the best thing ever/worst thing ever” news cycle, and talk to you about an open source project that I work on called “OwaspHeaders.Core”

This is the first in a two-part series about OwaspHeaders.Core and lays the ground work for why I chose to create it. Part two will be more about how I went about creating it, and the lessons I learned along the way, which are not just about HTTP headers and building your own NuGet package, and some of the amazing contributions that the community has put in place.

The three main goals for this two-part series are to talk people through the real-world problem that exists in all web frameworks and how we can all work together to solve it, to talk you through what I believe good open source stewardship looks like, and to talk through what creating and publishing a NuGet package looks like.

So let’s sit back, open up a terminal, type in dotnet new podcast and we’ll dive into the core of Modern .NET.

Early in my career, I was working on my first solo project at the company. It was for a public service. It had a hard requirement that it had to be secure. So we had a penetration test done before we shipped.

When the results came back, most of the failures were marked as CRITICAL. And almost all of them were about HTTP headers I’d never heard of.

This is the story of how that moment led me to OWASP, changed the way I think about web security, and eventually became an open source package that’s now been downloaded over one point seven million times.

But more importantly, it’s about something I think a lot of developers aren’t taught. And probably should be.

The Pentest Story

So. 2016. My first solo project at the company.

It was a public service application. The kind of work where the consequences of getting security wrong aren’t theoretical. Real people, real interactions, real data flowing through the system. I was proud of what we’d built. The team had been good to me. I’d taken the brief, understood the constraints, shipped a thing that did what the spec said it should do.

And then the pentest report landed.

I opened it expecting maybe a couple of low-priority findings. Things to tidy up before we went live. What I got instead was page after page of CRITICAL marked in red, with header names I genuinely did not recognise. X-Frame-Options. Strict-Transport-Security. X-Content-Type-Options. I read those names, and I felt the floor go out from under me. I’d been writing web applications for a few years at that point. None of these had ever come up.

So I did what every developer does when they’re suddenly out of their depth. I opened a browser, and I started searching. And the documentation I found assumed I already knew what these were. It assumed I already had the vocabulary. It assumed the gap I was standing in didn’t exist.

Now, what saved me at that point was my CTO. He looked at the same report and said something close to: “some of these don’t actually apply to us, given how we host the service. Let’s prioritise the ones that do, and triage the rest.” And that was a really important early-career lesson. A lot of developers, when a CRITICAL lands on their desk, will try to fix every single one regardless of context. Learning that triage based on context is itself a skill, that was the gift I got out of that moment.

But the bigger gift was the rabbit hole the research opened up. Because in among all the docs I was struggling with, I started seeing the same name come up. OWASP. The Open Worldwide Application Security Project. And alongside OWASP, a couple of names from the security community. Troy Hunt’s blog. Scott Helme’s tooling at securityheaders.com. And for the first time it clicked that there was a whole field of knowledge that nobody had ever sat me down and said “right, this is the bit you should care about.”

And it turns out I was very far from alone. Tanya Janca said something on the show, back on episode 77, that has stuck with me ever since.

I went to college in the late 90s. And there was nothing on security. And people tell me now that there’s still almost nothing or if there is it only has to do with access management, and maybe a little bit of network stuff. But there’s never in their software development classes, how to do the right thing.

That was 2021, and she was talking about a degree from the late nineties. The pentest results that landed on my desk in 2016 were saying the same thing in a different language. The gap was real. It wasn’t just me.

Before I get into what those headers actually do, just hold a thought for a second. Have you had a moment like that. A moment where you discovered a gap in your own knowledge that nobody had ever flagged to you. Where something everyone seemed to assume you knew, you genuinely didn’t. Hold that. We’ll come back to it.

What Are HTTP Headers, Anyway?

Before we go deeper, I want to take a step back. Because I’ve been talking about HTTP headers as if everyone listening already knows what they are. And some of you absolutely do. If that’s you, hang on for a couple of minutes. I’ll be back to the security side shortly. But for everyone who’s never had to think about this layer, this part matters.

Here’s the way I find easiest to explain it.

Think about a delivery driver bringing a package to your door. The label on the outside of the box tells the driver everything they need to know to handle it correctly. Where it came from. Where it’s going. Whether it’s fragile. Whether somebody needs to sign for it. Whether it can be left if you’re not in. The label is not the package. You can’t eat the label. But the way the package gets handled depends entirely on what the label says.

HTTP headers work the same way.

Every time your browser asks a server for a web page, the server sends back two things. It sends back the content. The HTML, the CSS, the JavaScript, the images. That’s the package. And it sends back a set of headers. Those are the label. The headers are instructions to the browser about how to handle what’s inside.

Now, most developers, most of the time, spend their attention on what’s in the package. We care about the HTML structure. We care about whether the CSS renders. We care about whether the JavaScript does what we wanted it to do. The headers, the instructions on the outside, we tend not to think about. They feel like plumbing.

Security headers are a specific class of those instructions. They’re the ones that tell the browser things like “only ever talk to me over HTTPS.” Or “don’t let any other website embed me inside a frame.” Or “don’t try to be clever about guessing what kind of file this is. Treat it as what I told you it was.”

Quiet, structural protections. None of them affect what the user sees. None of them change how the application behaves under normal use. They’re just instructions on the label that the browser reads, and acts on, and gets right.

And here’s the key point. The browser is already a pretty good sandbox. It does a lot of work to keep your users safe from things you might never have thought about. But with a few extra headers, that sandbox becomes meaningfully stronger. And the effort to add those instructions, especially in modern frameworks, is genuinely surprisingly small. I’ll come back to exactly how small for .NET developers later.

So if that’s the shape of the problem, and that’s the shape of the protection, the obvious next question is who actually decides what good looks like. Who maintains the list?

That’s where OWASP comes in.

Enter OWASP

OWASP stands for the Open Worldwide Application Security Project. It’s a non-profit foundation. They exist to improve the security of software, full stop. They don’t sell anything. They aren’t owned by a vendor. The materials they produce are free, freely available, and broadly trusted across the industry.

Within OWASP, there’s the Secure Headers Project specifically. It’s a maintained list of HTTP response headers that web applications should include, with recommended values for each one. And critically, it’s a living document. As browsers add capabilities and as new attacks get documented, the recommendations get updated. It’s not a static piece of advice from 2014 that nobody’s looked at since.

Now, why does this matter for your education as a developer? Because most of the formal training developers get focuses on writing code that does things. Code that handles requests. Code that queries databases. Code that renders pages. Security tends to get handled as either an afterthought, or a specialist concern that someone else is going to deal with. Which is how you end up with a pentest report full of CRITICAL findings on your first solo project. Like I had.

The OWASP Secure Headers Project is one of the most accessible entry points into application security I know of. The headers themselves are concrete. The recommended values are documented. And the result is something you can actually test. You can put a header in, scan your site, and see the change. There’s a feedback loop, and feedback loops are how developers actually learn things.

When Tanya came back on the show in season seven, talking about her new book on secure coding, she put the problem more sharply than I ever could.

From the very first lesson of ‘Hello, World’ they teach us to make insecure code. So the first thing with ‘Hello, World’ is how to output to the screen. That is fine. But the second part of ‘Hello, World’ is: you ask them their name, you take their name, you don’t validate it, and then you say ‘Hello,’ and you reflect their name back onto the screen with no output encoding. And then you just made cross-site scripting.

Once you hear it framed like that, you start seeing the same shape everywhere. Tutorials optimised for the easiest possible path, with no follow-up on what production-ready would actually look like. So if you take only one thing from this episode, let it be this. Go and look at the OWASP Secure Headers Project. Bookmark it. The link will be in the show notes.

A Quick Tour of the Headers

Right. With that out of the way, let me give you a quick tour of the headers themselves. I won’t go deep on any one of them. The OwaspHeaders.Core docs go further. The OWASP project page goes further still. What I want is for you to come away with a feel for what these things do.

And before I do the tour, I want to give you the framing I now use whenever I explain these to someone for the first time. It came from Tanya,P on episode 105 of the show.

I feel like security headers are just like seatbelts. But for web apps. So seatbelts are not sexy. They’re not exciting. They’re also not that much effort. And when the crap hits the fan, you’re very pleased when you do not go through the windshield.

Seatbelts. That’s the shape I want you to hold onto for the next few minutes. Quiet most of the time. Material when it matters. Easy to put on. Right. Let me show you four of them.

First. Strict-Transport-Security. Often called HSTS. This header tells the browser to only ever talk to your site over HTTPS. Once the browser has seen this header from your domain, it won’t even attempt an unencrypted connection in the future. That closes off a class of attacks where someone on the same network forces the browser to downgrade to HTTP and then watches everything that goes past. With HSTS in place, that downgrade attempt never even happens.

Second. X-Frame-Options. This one stops your website from being embedded inside a frame on someone else’s site. Why does that matter? Because of clickjacking. The classic version goes like this. A malicious site loads your site invisibly, layered behind something that looks innocent. The user thinks they’re clicking a button on the malicious page, but they’re actually clicking a button on yours. Confirming a transfer. Deleting an account. Approving a permission they didn’t mean to give. X-Frame-Options says no. Other sites cannot frame you. The attack stops being possible.

Third. X-Content-Type-Options. This one prevents the browser from second-guessing the type of content it received. Without this header, browsers will sometimes look at a file you’ve labelled as plain text, decide it looks suspiciously like JavaScript, and execute it as JavaScript. That’s called content sniffing, and it’s been a real attack vector. The fix is one short header value that says “trust what I told you this file is. Don’t try to be clever.”

Fourth. Referrer-Policy. This controls how much information about where a user came from gets sent along when they click a link from your site to somewhere else. Why do you care? Two reasons. The first is privacy for your users. The second is that internal URLs leaking out into other sites’ analytics dashboards is a small-but-real reconnaissance signal for anyone planning an attack on your infrastructure.

None of those four are obscure. None of them are theoretical. They protect against attacks that have been documented for years and that are still actively used. And most web applications still don’t include them. That’s the gap.

Now, my CTO was right about something back in 2016. Not every header applies to every application. Some of these don’t make sense in particular hosting models. But the only way to make that judgement is to know the headers exist, and to understand what they’re for. Triage requires options. And until OWASP put it all in one place, most developers didn’t even know they had options.

There’s one header I deliberately left off this tour, because it deserves its own treatment. And that’s the Content Security Policy.

The CSP Exception

The Content Security Policy. Or CSP. This is the one that genuinely is harder than the others, and I want to be honest with you about why.

The way I explain CSP to developers is to talk about a bouncer at the door of a club. The bouncer has a list. If you’re not on the list, you’re not getting in. CSP works exactly like that. It’s an allowlist that tells the browser, in detail, which sources of content are permitted to load on your page. Scripts from your own domain? Fine. Scripts from a CDN you trust and have explicitly named? Fine. Scripts from anywhere else, including anywhere clever that an attacker might try to inject content from? Blocked.

When CSP works, it’s powerful. It can shut down whole categories of cross-site scripting attacks because the browser refuses to execute the injected payload in the first place. The script literally never runs.

The trouble is, somebody has to write the guest list. And most developers don’t actually know everywhere their application loads resources from. There are inline styles. There are third-party analytics scripts. There are fonts loaded from Google. There’s a CDN serving JavaScript libraries. There’s a payment widget. There’s a customer support chat tool. There are tracking pixels you may not even have personally added. If any one of those isn’t on the list, the browser blocks it, and something on your site stops working.

And I’m not exaggerating the pain here. Tanya, who, remember, works in application security professionally, described the first time she rolled out a CSP on episode 105 of the show.

The first time I remember I put it on a website and these devs I was like, listen, you know, you’re not using any security headers. These are the ones I would like, I’m like, but do you have a list of your outside stuff? And they’re like, ‘Oh, we’re not using anything that’s not from within our own domain. We don’t do that.’ We turned it on, and the entire website’s just this ugly mess.

If a security professional gets caught out by what an application is actually loading, the rest of us shouldn’t feel too bad about it. The lesson is that you have to look. And then you have to keep looking, because the dependencies change.

It gets more complicated still with frontend frameworks. React injects bundles at runtime. Different pages may need different policies. Single-page apps and server-rendered apps need different shapes of policy. There isn’t a one-size-fits-all CSP value I can hand you and say “off you go.”

Which is exactly why OwaspHeaders.Core does not include a CSP in its default configuration. The other headers have sensible defaults that work for almost any application. CSP doesn’t. Including a default CSP would silently break things in production. And breaking things in production would be the opposite of falling into the pit of success. (Yes, that’s a phrase the .NET community will recognise. It’s there deliberately.)

What the package does provide is a set of extension methods that make CSP configuration easier once you’ve worked out what your policy actually needs to be. The documentation walks through the shape of a policy. It’s a starting framework, not a magic wand.

So here’s a small homework task. Open your browser’s developer tools. Go to your application. Open the Network tab, reload the page, and just look. Count the distinct domains your application talks to. You might be genuinely surprised. I certainly was, the first time I looked properly.

The Tutorial Problem and Better Defaults

Which leads me to a bigger question that this episode keeps brushing up against. If these headers protect against well-known attacks, and most of them have sensible recommended values, why aren’t they already in place by default?

There’s a tutorial problem. Example code, getting-started guides, the official 101 walkthroughs. They almost always pick the easiest possible path to the working result. Which makes sense for learning. You can’t teach everything in lesson one. But the side effect is that production systems get built on insecure foundations. And I’d argue this is a major reason behind a lot of the breaches we’ve seen over the last couple of decades. Developers follow the 101 guide. The 101 guide sets things up insecurely. The result goes to production. Nobody flags the gap because the gap is structural.

The framing I keep coming back to here came from Abel Wang. He had a phrase he used a lot. Don’t accept the defaults. The idea is simple. The defaults in any tool or framework are somebody else’s opinion about what’s good enough for the average case. And good enough is rarely the same thing as secure enough. Be intentional about your choices.

Now, you might hear that and immediately spot a contradiction. Don’t accept the defaults. And here’s a package that is itself a set of defaults. Fair point. So let me sit with that contradiction for a second.

The defaults in a brand new ASP .NET Core project today are insecure on this dimension because the headers simply aren’t there. The field is empty. There’s no good default to override. OwaspHeaders.Core fills that empty field with values that have been researched, agreed by OWASP, and are known to be safe in the broad case. Calling that blind acceptance would miss the point. The aim is to make the starting position a secure one.

Tanya put it best when she came back on the show last year, talking about her new book on secure coding.

In the book, actually, I have a big section about secure defaults and building paved roads. And in it, I’m like, ‘I know this is manipulative, but what if we manipulate our users for good? What if all the defaults are super secure?’ Most users aren’t going to change the default and we can trick them into being more safe. Come on, let’s do it.

Manipulating users for good. I love that framing. That’s the whole design intent behind OwaspHeaders.Core in a single sentence.

And for the developers who do want to be intentional, exactly as Abel would have wanted, the package is fully configurable. Every header. Every value. You can exclude what doesn’t apply to your hosting model. The documentation explains every default and why it was chosen, so you can make an informed call. The pit of success is there for the developers who need it. The understanding is there for the developers who want it. That, to me, is what honouring both principles looks like.

Call to Action and Close

Which brings me back to where I started. The pentest. 2016. The CRITICAL findings. The header names I’d never heard of.

When that report landed on my desk, I thought security was complicated. And some of it genuinely is. CSP is proof of that. But the majority of the OWASP recommended secure headers? They’re straightforward. The values are documented. The reasoning is clear. And for .NET developers specifically, implementing all of them is one line of code away.

So two things to take from this episode, depending on where you’re sitting.

If you write .NET, go and read the OwaspHeaders.Core documentation. It walks you through the shape of the package, what it sets, why it sets each value, and how to customise it for your particular case. The link will be in the show notes.

And if you don’t write .NET, the principles do not change. The headers don’t change. Your stack almost certainly has a way to implement them. A package. A middleware. A few lines of configuration. Go and read the OWASP Secure Headers Project page. Pick your stack. Find the equivalent. The work is small. The protection is real.

In the next episode, I’ll tell you the other half of this story. How a blog series I wrote about the ASP dot NET Core middleware pipeline accidentally turned into a NuGet package. What it was like learning CI/CD with AppVeyor before GitHub Actions existed. And what eight years of maintaining an open source project has taught me. The good bits, the awkward bits, and a few that genuinely surprised me.

Until then, take care of yourselves. Take care of your users. And I’ll see you next time.

Wrapping Up

Thank you for listening to this episode of The Modern .NET Show with me, Jamie Taylor. I hope that you found this monologue intersting and that you’ll come back for part two.

Be sure to check out the show notes for a bunch of links to some of the stuff that we covered, and full transcription of the interview. The show notes, as always, can be found at the podcast's website, and there will be a link directly to them in your podcatcher.

And don’t forget to spread the word, leave a rating or review on your podcatcher of choice—head over to dotnetcore.show/review for ways to do that—reach out via our contact page, or join our discord server at dotnetcore.show/discord—all of which are linked in the show notes.

But above all, I hope you have a fantastic rest of your day, and I hope that I’ll see you again, next time for more .NET goodness.

I will see you again real soon. See you later folks.

Follow the show

You can find the show on any of these places