The Modern .NET Show

S08E11 - From Chaos to Control: Anton Moldovan on Load Testing with NBomber

Sponsors

Support for this episode of The Modern .NET Show comes from the following sponsors. Please take a moment to learn more about their products and services:

Please also see the full sponsor message(s) in the episode transcription for more details of their products and services, and offers exclusive to listeners of The Modern .NET Show.

Thank you to the sponsors for supporting the show.

Embedded Player

S08E11 - From Chaos to Control: Anton Moldovan on Load Testing with NBomber
The Modern .NET Show

S08E11 - From Chaos to Control: Anton Moldovan on Load Testing with NBomber

Supporting The Show

If this episode was interesting or useful to you, please consider supporting the show with one of the above options.

Episode Summary

In this episode of The Modern .NET Show, Jamie Taylor sits down with Anton Moldovan, founder of NBomber, to explore the crucial world of load testing. \n\nAnton shares his 19 years of experience architecting high-load systems (spanning sports and iGaming domains!) and demystifies how to effectively test your .NET applications before they’re hit with peak traffic. They discuss pragmatic functional programming, blending paradigms for effective solutions, and the importance of testing within your existing tech ecosystem.

NBomber: Discover this open-source .NET load testing tool, designed for flexibility and ease of integration with your current workflows. Learn how its simple lambda-function approach allows you to test any protocol without being locked into a specific DSL.

Load Testing Strategies: From end-to-end user journey simulations to isolated microservice testing, Anton explains how to build a robust testing strategy tailored to your needs.

Chaos Engineering & Load Testing: Find out how to combine load testing with chaos engineering principles to build truly resilient and fault-tolerant systems.

Pragmatic Functional Programming:** Understand Anton’s approach to applying functional principles in real-world .NET development, focusing on value and avoiding unnecessary abstraction.

NBomber Studio: Get a sneak peek at the upcoming private cloud offering, providing a cost-effective and flexible load testing solution.

Episode Transcription

Another thing which I also observed is that there is some benefit to be able to run your load test in your native… using your native platform, libraries, protocol access; those type of things. Because in our case, for example, we use Orleans and it’s a proprietary protocol which doesn’t exist in in Java in Scala language. The same about, almost the same, was about Signal R: Microsoft released SignalR for Java, but the quality of this library was different.

- Anton Moldovan

Hey everyone, and welcome back to The Modern .NET Show; the premier .NET podcast, focusing entirely on the knowledge, tools, and frameworks that all .NET developers should have in their toolbox. I’m your host Jamie Taylor, bringing you conversations with the brightest minds in the .NET ecosystem.

Today, we’re joined by Anton Moldovan to talk about load testing, advice for testing strategies, and how NBomber can help you to load test your applications. Are you sure that your application can handle 4 million users at once? Better load test it before you start boasting.

We call this type of test, like, "user journey." Like, end-to-end simulating user journey across entire applications. So end-to-end, end-to-end flow, end-to-end tests. But this type of test they they have some downsides.

- Anton Moldovan

Along the way, we talked the different types of testing involved in getting your application for production, the many different ways that NBomber (or other load testing suites) can help you prepare for that, and Anton helps us understand a little more about functional programming.

So let’s sit back, open up a terminal, type in dotnet new podcast and we’ll dive into the core of Modern .NET.

Jamie : So Anton, welcome to the show. I feel bad. Because, you know, you reached out a while ago and I missed the email. So I want to apologise to you in the episode so everyone can hear me. I’m having The audience hold me accountable to apologising to you.

Anton : Sure. And thank you for for having me here.

No worries, no worries at all, no worries at all. It’s gonna be a ton of fun. We’re gonna talk about something that I know I don’t get to do a lot of. And I feel like a lot of the average .NET developers probably don’t get to do so much, but it is something super important. I will leave the teaser at that, although it’s not really a teaser because folks have probably read the name of the episode and figured it all out by now.

But before we get on to the topic at hand. I was wondering, could you give the folks who are listening in a bit of an intro about the kind of things that you do, the kind of software stuff that you’re interested in that you like to develop? I know you do a lot of stuff in open source. We had a brief conversation before we hit record talking about open source. So like it feels like it’s a bit of a passion of yours. I wonder if you could just let the folks know.

Anton : Uh sure. my name is Anton Moldovan. I am founder of NBomber company. I have about 18, maybe 19 years of experience in development. For the past ten years I have been working as a software architect mainly on the project related to high load. Usually it’s some sports or iGaming domain. those type of projects they require require a low level latency request handling, scalable, fault tolerant, cost efficient solutions. Yeah, I have this background. And from programming perspective my main interest distributed systems architectures databases internals algorithms, and I also am a big fan of pragmatic functional programming. I do some open source contribution and, from time to time, I also speak at local meetups and conferences. Yeah, that’s about me.

Jamie : Awesome. There’s a lot to unpack there. Especially with the, you mentioned "pragmatic functional programming." I know what functional programming is. I know what being pragmatic is. But what’s pragmatic functional programming?

Anton : Yeah, I understand your question. So basically, to me at least, it means that I can apply and adapt functional programming to this messy real world. Basically what I mean by this is that instead of just blindly apply(ing) the functional programming paradigm you would do a bit different things. You’re using functional ideas where they add value instead of just blindly start adding abstractions like monads, call monads, free monads, and so on and so forth. And in cases, for example, if you see there is need for mutating the state, there is nothing wrong with this for some performance critical sections.

You can use these techniques and you basically can blend functional programming with other paradigms in a pragmatic way. This is what it means to me.

Jamie : Oh, I see. I was worried that you’d discovered some brand new way of doing functional programming that was like light years ahead of everybody else and I was like, "wow, I need to know about this." But no, I also fully appreciate that it is often quite difficult to be pragmatic in any programming paradigm. And I feel like functional is one of those things that certainly requires a mindset change from the standard—standard is the wrong way to put it—from the often used object-oriented way of thinking about things. Especially since most of the the books and documentation about being pragmatic tends to be from an object-oriented perspective. So yeah, I fully appreciate what you mean.

Anton : Actually it took some time for me to develop this sort of pragmatism or pragmatic way of thinking.

Yeah, when I start with functional programming, I learn a lot of different abstractions, but at some point I just realized that the code that I write is so complex and I don’t see big value in it, then I just did like two steps back to the roots of functional programming, and I just found that I just need a few things usually like better matching, pure functions and immutability. This is all I need. And then for some cases I can mix it with object-oriented programming where it makes sense. For some cases I can just write very clean pure logic and cover it by some, let’s say, property based testing. Yeah, it, this makes sense to me.

Jamie : Yeah, and I think you raise a really good point there. And I’ve seen this happen with a lot of people. When they learn a new technique or they learn of a new way of doing something or, you know, a new language feature is enabled, or a new package is discovered: a lot of people try to shoehorn that into their work, right?

So Let’s say the the example that I usually give is IAsyncEnumerable, which is a. NET thing, I believe available in both C# and F#, but I know of it from a C# perspective. When that was first announced and released, a lot of the pushback was, "now I have to rewrite all my code to use this?! I can’t believe it. That’s really horrid. Why could why would you release a new thing?!" I’m greatly reducing the online discourse down to that, but that’s what it came to. And it felt very much a case of, "hey, just be pragmatic with it, right? If you don’t want to or you can’t think of a use for this particular thing, don’t do it."

Especially with software engineering as a subject, right? We need to be pragmatic because there are so many rules and guidances and all sorts of different advice floating around that sometimes, just like any other engineering practice, we can’t use it all because not all of it fits, right? You wouldn’t necessarily just use a hammer to build a bridge, right?

Anton : Yeah.

Jamie : So we’ve danced about it a little bit, having a bit of a dalliance into functional programming and paradigms and guidance and things. But we have a specific topic we’re going to talk about today. And that is NBomber, which is your open source project; and how it allows folks to do load testing.

Now load testing is something that I don’t get to do a lot of. I tend to be quite safe in that the industries that I’ve worked in, and the types of software I’ve written haven’t needed a million concurrent users because I haven’t worked for Netflix. But a lot of people do have that requirement, and most of the time I just kind of… I’m happy with making the system scalable and then going, "hey cloud provider, I’m happy for you to figure it out if you want," and then just hands off, letting the cloud provider do it. But this is obviously for folks who want to test what their system can do and maybe find those limits themselves, right?

Anton : Yeah.

As you mentioned regarding the the Netflix case, back in the day, I believe it was 2017 or 2018. They started designing a new system which was supposed to be (a) stateful system that handles a big massive stream of background events, and also in parallel it handles incoming requests from the users. And this system requires a lot of concurrency, high-availability. I actually wanted to run some PRC for my tech design to to prove that the architecture can handle such a load.

And on that time we were using Microsoft Orleans for keeping the state in memory, and also for the WebSockets we used SignalR to be able to send push updates to the users. So the thing was that on that time I just discovered that, first of all the landscape of load testing was different, and I remember there was JMeter; which is good maybe for basic load testing where you can just click buttons and start bombarding some endpoints.

But for my case I wanted something more flexible, something more close to the code itself, where I can use the code to express my load test scenario, and I remember on that time the Scala Gatling framework was quite popular and to me it it looked as a solid solution. So first of all this solution is GVM-based. and it provides you it gives you the nice DSL to express your your scenario. And at some point I realized that this DSL is actually very good for very basic things to express, but as soon as you wanna do some dirty (?) things: introducing some custom hash tables, dictionaries, while true loops, this type of logic, you need to learn the DSL. I understand like today probably with help of AI, ChatGPT, you can probably ask ChatGPT and it can provide you equivalent in this Gatling DSL; but on that time I just found that I start fighting with the framework. I need to learn this concept, and this is not what I want.

Another thing which I also observed is that there is some benefit to be able to run your load test in your native… using your native platform; libraries, protocol access, those type of things. Because in our case, for example, we use Orleans and its proprietary protocol which doesn’t exist in in Java, in Scala language. The same about… almost the same was about SignalR: Microsoft releases SignalR for Java but the quality of this library was different, and we’ve seen this in our load tests that we were able to open maybe five, 10 web sockets connection and that’s it. Like we reached some state where we see some exception, a weird exception we’re were throwing constantly, but on .NET everything worked as a charm. And I recognize that, "hey man, I need something more close to my platform because I see the client drivers for database, they could behave differently depending on the platform."

And basically this for me were was turning point where I realized I would like to build something for .NET, to have like native .NET load testing tool which I can use, and be able to express my intent. When I started designing, I decided that since I was already in functional programming, sort of camp, and I wanted to have to build a like minimalistic framework with a few abstractions, I don’t want to provide a lot of stuff that need to be learned. And in NBomber, for example, you need to learn mainly two abstraction scenario and step that’s it. Other than this, you can reuse your knowledge. If you know C#, you can express anything. You can use dictionary, if, while loop, it’s just regular code which you write every day.

Also, I was thinking that having native debugger which comes with Visual Studio brings a lot of benefits because now, especially for complex tests, I can iteratively debug some stuff, and be able to fix something more quickly, let’s put in this way. Additionally, I also realized that when tests are growing, when it is not just, you know, pinging simple endpoints, but something more bigger, then you need type safe language. So comparing writing tests in in JavaScript and .NET: it’s day at night. So therefore I just decided that, yeah, it makes sense.

And also the thing which I really like about and NBomber and having native load testing tool is how it integrates in my ecosystem. For example, I am developing my software in Visual Studio or Rider I can easily navigate to my load testing project, click run and be able to run my test just from Visual Studio, because the load tests in NBomber is is just (a) basic test which you do with xUnit or nUnit. So you create a a test project, you have the xUnit method, and within this method you just put the logic of NBomber, that’s it. So you can reuse your knowledge, you can reuse your CI/CD jobs for this. So it plays nicely. That was like, key points which triggered me to to start building NBomber.

Jamie : And all of that makes perfect sense, right?

Like the the point that you made really early on into that about using your… you didn’t use these exact words, but you said around the idea of using your own tech stack, right? Because you mentioned JMeter. I remember very early in my career doing stuff with JMeter—I know I did say that I don’t tend to do a lot of load testing, but when I did need to do load testing, I did use JMeter—and that was inside of a container, because no one on the team knew anything about Java. We just knew that if you Googled enough, you could find the arcane command to start JMeter with the particular parameters we needed. None of us understood the parameters, we just knew that if you ran it and pointed it at your app, you would get a chart at the other end. That was it. That was all we needed to know, right? Which was enough for that moment in time. But then when things went wrong, we had no idea.

So no, I completely agree that we should always be using tools, and frameworks, and techniques within our own ecosystem, because then when things go wrong we have a little bit of knowledge about how to reuse them; and especially your point about because if it’s written in F# you can reuse some of your C# knowledge and some of the C# types because guess what folks? Both languages use the same .NET type system.

Anton : Yeah, yeah. Additionally to this, I found that I don’t want to limit the user with this. I don’t like the DSLs in, for this particular for the load testing. Therefore I wanted to provide the freedom for the users to define his load test scenario just using regular code; and with NBomber there’s no dependency on any specific protocol or semantic model, pull or push model. So NBomber provides you just basically a short lambda function which it measures whatever you put within this lambda function, you can put the code which does HTTP request, you can put the code which does WebSocket connection or gRPC call. NBomber just measures how long this function executed. That’s it. So you don’t have with NBomber, you don’t have the dependency. But I do understand that for some use cases it’s still better to have some sort of maybe plugins for specific protocols and NBomber provides, I believe, about like maybe 15 different plugins for HTTP, MQTT, WebSockets, GRPC, this type of MQP protocol. So there is plugins, but you still have this freedom to do whatever you want to do.

Jamie : Right, right. And the simplicity of how you described it earlier, with it being effectively just a lambda function, a pure function, and we just test how long that runs for. Like, that sounds almost too simple to be true, right? But when you’re doing any kind of testing to do with load, that’s one of the things you want to know about is: how long did it take to service this request or to do this functionality or something like that, right? How long did it take to run this process?

The fact that it is such a, on paper, incredibly simple way of working that out and and implementing that: that’s genius.

Anton : Thank you. Thank you for your feedback.

Jamie : So you talked a little bit there about the, you hinted at, I guess, about the extensibility there. I’m trying to think of the best way to put this, right? Because I know that F# is a functional language and the idea behind functional language is that everything is supposed to be a—and this is this is coming from my understanding of functional programming, and is probably not correct—but like everything is supposed to be a pure function, right? We don’t affect the world state. And so is it the equivalent of MapReduce? Like you have a function that measures some process running time and then you can then immediately pipe that to something else by calling some other Lambda function that you’re interested in testing.

Like how does, from a ten 10,000 foot view, I know we’ve gone really deep into the weeds and said, basically it’s just a lambda function; but what does that look like for for a developer, right? If I’m looking at some NBomber code, what… do I write like… I’m asking a million questions here. I’m really sorry, but like if I’m using NBomber, am I using F#? And if so, am I diving deep into writing my own lambda function? Or if I’m using NBomber, is it just "call a thing from a thing, and it will give you a chart?" Like where does it sit on that spectrum.

Anton : Yeah, sure.

So first of all, F# it’s not pure functional language. It allows you to do side effects. And you can think about this as a stronger language than than C#, but still what whatever you do in C#, the same you can do with with the F#. Regarding NBomber and lambda function: so NBomber basically provides you the abstractions of scenario which contains the name of scenario and the lambda function it’s a body of your function which will be executed, and within this body you can put whatever you want. But there should be something for composition. Within NBomber you can create steps. These steps could be created within the scenario as additional lambda functions. And a step also has a name, and lambda function, and a body. Basically whatever you put in this body Nbomber will compose together and execute like sequentially, step by step.

After finishing all steps execution within the scenario, it treat this as a like successful iteration and writes, under the hood it writes the calculates metrics, calculate statistics. And if it’s needed, it can voice them to any popular observability platform, DataDog, InflexDB, TimescaleDB, whatever you decide.

Jamie : So I can add a number of things, we said steps, and then once it’s fully finished processing those, I really like the idea that I can send the data, the report off to whatever, a number of not whatever I want, but a number of different information collection services. That’s that’s really useful as well. Yeah, I like that.

Anton : Yeah, in terms of extensibility NBomber provides a rich reporting capabilities. It builds, at the end of the test, it builds the HTML report with a single page of HTML which you can attach even to, like, to (a) Jira ticket. You can open it on any device and and see the results of your test. But also it provides extensibility points, one of them is “Reporting Sink”. It is interface which you can implement NBomber will call you to the statistics data which then you can store in any observability store which you use, for example, in your company.

NBomber comes with a few things one is InfluxDB, Datadog, TimescaleDB, and I believe Prometheus. If you need something different, you always can implement. There’s about maybe three, four methods which you can implement for saving data into your store. That’s about it. Additionally to this NBomber provides clustering support, you can run your test in distributed mode. And also you have a very, I would say, tunable distribution of load. So there is dynamic load distribution. One of them is for example Zipfun. or multinominal distribution where you can specify, "I have several scenarios. I want that 50% of cases, this scenario should be invoked. But in 20% another one." So you can have this option to specify such type of distributions and to simulate basically any workload.

So when I designed NBomber, I actually always tried to sort of mimic the workloads that I needed to implement or to simulate back in the day. So NBomber is basically based on my experience in building high-loaded systems.

Jamie : And that’s one of the best ways to build a, I’m gonna use the word "product," but it could be just an open source thing, just a thing for folks to build. And for the joy of building something, right? Thinking of the times when you’ve probably thought in your life, "gee, I wish this existed." Well guess what? You’ve got the skills to go make it exist, right?

Anton : Yeah.

And it means that you’re then more likely to, you know, the the phrase is dog-fooding, right? So you’re you’re more likely to then go back and, "can I can I solve this problem now with what I built? And I will then see," and this is the most important bit, "I will then see the pain points of using this in a production-like setting. So then I can help the user. I can empathize with the user and sympathize with the user and see where I can make my application or my library way better," right?

Anton :

Right, right. Yeah.

Jamie : We’ve said so far that we can have an action of some kind that we want to measure, and we can put it inside of NBomber, which will do all the measurements for me, but we haven’t said, like at least I don’t think we have. I mean, please correct me if I’m wrong. But I don’t think we’ve said whether it’s like HTTP, whether it’s GRPC on any other three-letter acronym, right? is it technology agnostic or is it technology specific?

Like, let’s say I build a REST endpoint that uses HTTP and I want to measure that, "fantastic. Now I’ve built a GraphQL endpoint that uses HTTP. Fantastic." And now I go ahead and build something with GRPC that I’ve personally have never built anything with GRPC, so I’m just going to assume that I can build anything with it. Can I test all three of those or am I locked just to REST?

Anton : So in case of any other technology, you usually have some client. If it’s GRPC, you will have GRPC client. If it’s Microsoft Orleans, you will have Microsoft Orleans client. And basically you initiate the connection, you take this client, wrap it inside this lambda function, then do whatever you do: invoke some method, send some message, receive response. NBomber will measure it, and based on the content you receive you can say, "this is response is correct successful," or, "this response is fail." And NBomber will measure and track successful requests and unsuccessful; and basically at the end you will receive a statistics plus you will be able to apply assertions on top of the statistics results from NBomber at the end of the test you receiving overall statistics.

And if you, for example, integrating NBomber with xUnit or nUnit framework, you can use their assert libraries or methods, basically helpers to do assertion, on top of NBomber statistics. And in addition, NBomber will print out threshold results in the in source results in the HTML report. So you will see how many times some of your threshold failed. And, yeah you will have this information.

Jamie : Right. So it’s pretty much any protocol, pretty much any communication method, as long as it is, "I, as a client of something, push or send or call or something like that." And then on the other end, "I receive a response, and I can track the beginning and end of that process. I can then Test it," right?

Anton : Yeah, yep, yep, yeah. And if you have, for example, CI/CD pipeline and this test has failed, if it’s xUnit or nUnit, you will see that, "okay this test failed. What I should do about this?" It’s easy integrated with with a .NET native ecosystem

Jamie : Right. Cause that that was actually going to be my next question. Like, who’s doing these these tests, right? Is this an automation test? Is this a QA based test? Is this a, "I’m a dev and I’ve written something, and I want to test it to prove that I have test coverage?" Or is it just for everybody? Like anyone can do this as long as you have access to the tooling.

Anton : So actually when it comes for load testing in general, we usually suggest applying two strategies.

One effective approach to load testing is simulating real-world user flows that interact with multiple services. And this end-to-end test replicates how users actually use your application, from logging in in a browser to making purchase or performing other key actions. Basically, by executing these flows under the load you can check how your system, microservices, whatever you use perform collectively. This helps identify bottlenecks in the orchestration of services and evaluate system behaviour under realistic conditions.

So the key benefits of this is you understand how servers interact under the load, you uncover integration or orchestration issues. Potentially in your testing also you may test also autoscaling configuration to avoid unexpected overloads. And you validate that the system meets performance expectation from the user perspective. We call this type of test like user journey, like end-to-end simulating user journey across entire application so end-to-end end-to-end flow end-to-end tasks.

But this type of tests, they have some downsides. First, high infrastructure cost, because running such load tests across the entire system typically requires spinning up all microservices, databases, caches, queues and other dependencies. This can be resource intensive, especially if you need to scale components just for the purpose of the test. For big companies, this type of task can lead to significant cost. Another downside of it is a slower feedback loop. Because of the orchestration involves these tests take longer to set up, prepare data and and and run, making them less suitable for fast iterative development cycle. So as a result your developers might only discover performance degradation at the last moment. Basically we recommend using this end-to-end strategy when, I would say, often in preparation for some major big events like Black Friday or monthly releases. The reason to run such tests less frequently is due to their cost. Depending of course on the scale of the company, this test can be expensive to run.

And answering your question about who should write this type of test, usually this this particular this type of test is written by automation QA engineers or performance engineers. Usually it’s not just developers. Usually it’s special QAs, performance folks.

And another, like in addition to this, there is another stream, another approach; we call it "load tests in isolation," or where you testing some microservices in isolation. So while end-to-end testing provides a macro view on the system, it’s equally important to test microservices also in isolation. So this strategy helps you to, like, pinpoint which service may struggle with performance independently of the rest of the system. And this test brings a lot of value.

So key benefits: it’s low low infrastructure costs. These tests are usually very easy to set up and run, even on developer machine. So fast feedback loop; you can quickly identify performance degradation in specific specific service. Isolated tests can be easily integrated in your CI/CD pipeline and treated like a regular unit or integration test, and they identify performance bottlenecks within individual services.

The downside: these tests are not fully representative. So they typically cover only single service. The worst case scenario is when multiple services each performing well in isolation, but fail to handle the expected load when run together in production environment. This is what I’ve seen many times to be honest with you. And this, like local isolated test, this type of test is usually written by developers and not not QA engineers.

Jamie : I really I appreciated your point there of bringing up, "I have four, say, microservices and they all run really well when I’m testing them in isolation. But when I test them together, that’s where I have the problem," because you don’t run them in isolation in production.

Anton : That’s true.

Jamie : Yeah, that’s a really good point. I know that it feels like there’s not much else to say about that, but is that something you can expand on? Like, maybe some hints and tips for folks who maybe are doing micro—I worry about saying microservices because then we’re playing buzzword bingo and all I need to say is "Bitcoin", and "AI", and "a genetic workflow" and then we’ve won at SEO.

But if folks are building apps out of several different services, what are the top tips from your perspective for load testing those services together? Is it do things like NBomber and get an idea of what it feels to run this application in production? ‘Cause, like, what are your top tips for that kind of thing? ‘Cause I feel like there’s some gold there that we need to dig a little bit deeper for.

Anton : Sure, sure.

Usually I suggest to start with end-to-end flows, even if if they maybe will be more expensive, but still they show a lot regarding how your system works, interacts with other services. But then I would say you should try prioritize services that interact directly with that database. Because if these services perform well, usually upstream dependent services will typically also benefit from it and will work also well. It’s usually what I seen from my practice. So since database scalability is is often the main bottleneck, it’s more effective to focus on services that work directly with a database rather than those that depend on the other services in the chain. This is what I usually recommend to companies.

And of course important to be able to easily touch specific microservices, be able to overload specific database. Because I’ve been in situation where the company tried to, you know, develop a new flow based on new database, and in that case it was scylladb. It’s a very fast NoSQL database. And I found that in order for them to run the load test, they need to spin up everything, every microservice, because this is how they have to develop their end-to-end services. They didn’t have any like more like a localized type of test that could just hit this scylladb to validate that workload. So I found that you still should be able to have some solution in your toolbox which will allow you to quickly develop such type of test, just to test specific database, specific flow, be able to run some generator on the background which will produce load for your system. So this is what I suggest usually.

Jamie : Right. All good hints and tips.

Anton : Yeah, yeah. Sorry, sorry. Yeah, I’ve I forgot to mention one one thing about this isolated type of test which also I seen several times: is that there could be cases where your microservices can depend on other services, especially those outside of your ownership. So you should consider mocking them. And you should be very cautious about this. You should choose for mock it API. You should be able to get pretty good controllability in terms of configure the latency delay, be able to configure the throughput capacity for running such type of tests. You definitely will need something like this.

Jamie : Yeah, you need to be able to mock any kind of big dependency. Because, let’s say you have a third-party dependency for your application, right? And you want to be able to run any kind of load test against your application. You don’t want to be sending a request through your application as a load test to that third party, right? Then calling them up and saying, "sorry, we just sent five million requests to you. Please don’t use up all of our API usage for this year in the last five minutes."

Anton : And, by the way, in my experience, we found that, for example, if you take some generic framework which provides you mocking API. They usually do not work well under high load. So and we end up with building our own custom locking services using just ASP .NET Core and just deploy them because they handles a lot and we can we can rely on their performance.

Jamie : It makes sense. Because your mock to be able to behave somewhat like the real thing without having to put loads of engineering time into making it as performant as it, right?

Anton :

Yeah, yeah, yeah.

You know that moment when a technical concept finally clicks? That's what we're all about here at The Modern .NET Show.

We can stay independent thanks to listeners like you. If you've learned something valuable from the show, please consider joining our Patreon or BuyMeACoffee. You'll find links in the show notes.

We're a listener supported and (at times) ad supported production. So every bit of support that you can give makes a difference.

Thank you.

Jamie : I appreciate we’re starting to run out of time, but I also appreciate we’re starting to hit at some really interesting stuff. Because we were talking about your end-to-end user journey testing and how we can test individual microservices in isolation, but as an entire group. Because, you know, my experience of doing microservices based stuff is: it’s way more complex than you think it’s gonna be, folks. Even if you’ve done microservices for the last twenty-five years, it’s still super complicated. For sure.

Anton : For sure. Yeah.

Jamie : I wonder, just as as we get towards the end, there’s something that we’ve got written down in our notes from when we were planning this out that I’d love to cover if we have the time. And that is: load testing with chaos engineering. Just before we get to that, chaos engineering is one of my favourite things ever, because it is literally letting a monkey loose in your server room. But also like describing it to non-techie friends as, &quot’guess what I got to do today? I got to do some chaos engineering." And then they think that I’m some kind of evil super genius. So I love that. I love the phrase and I love that we get to do, and we’re supposed to do some kind of chaos engineering because we need to know what happens when we switch one of the services off, right?

Anton : Yeah. Yeah, it’s a basic switching the service offline. It’s a one of the useful, by the way, useful examples of applying chaos engineering.

Yeah, so chaos engineering is (a) discipline where we trying to introduce controlled experiments into our system, and basically when I say "experiments" I mean some fault injection or some unexpected condition. And based on this we can understand how our system handles such type of, survives such type of conditions. This discipline, in general, it helps to improve reliability, resilience, and find some hidden vulnerabilities in the code. So I, to be honest, I don’t know how you can build especially for some bi companies, let’s say Netflix or maybe a bit smaller, but still I just don’t know how you can build reliable system without applying chaos engineering.

So and to say this, I want to add that load testing is actually brings a lot of value if you start considering applying chaos engineering, because at some point you will realize that you need to simulate the traffic, and maybe even specific traffic patterns. And this is where a lot of testing will help you.

Usually what I seen is that people start from running the special load on their staging environment, and in parallel they start simulating different experiments. One of them is just killing nodes, shut down some zone, killing network between services or between your Kafka and database, and so on and so forth. Or introducing additional load on CPU or on memory, introduce some restarts of pods in the Kubernetes, and so on and so forth. Based on this experiment they can understand, analyze what went wrong and improve this this. Yeah, that’s in short what I can tell about chaos engineering.

Jamie : Yeah, you’re absolutely right. If you’re building a system that is big enough that you’re considering doing load testing, you definitely need to be doing chaos engineering, because otherwise how are you gonna know what happens, right? There’s no way to know, right? And I don’t mean on the daily basis. I don’t mean, "hey, we’re developing this thing and then suddenly we turn on Chaos Engineering for Production." Don’t ever turn on Chaos Engineering and Production. Turn on Chaos Engineering and Production then watch the logs light up with everything is broken, and customers are cancelling their accounts. Don’t ever do that.

But you definitely need to know what happens when this particular service is disabled. Like, "can we do exponential back off and retries, and things like that?" How do you know your retry policy is working if you don’t switch off that microservice to see whether you can retry the the request, right? How do you know that your queue system is working if you’re not putting things onto the queue and then switching off the thing that reads from the queue, right?

Anton : It’s yeah, I agree. It’s absolutely super important to do some chaos engineering from every now and every now and again.

Jamie : So does that mean then that I can do an element of Chaos Engineering with NBomber? Like is that something that is is supported out of the box? Or is that one of the… I know we talked a little bit about how there is some, what’s the phrase I’m looking for? Some like extensibility built into NBomber. Is that something that is extensible through the marketplace or anything like that?

Anton : Not yet. By the way, a good question. We have this in our backlog regarding adding some API which will allow you to do in parallel with load testing, allow you to do chaos injection into your into your deployment.

Jamie : Yeah. Okay, okay. So there there’s some some stuff that can be done. That’s interesting.

What we’ll do is we’ll leave that as an exercise to the listener to go ahead and check out the website, of which I’ve got down as nbomber.com and go and check that out. Get a demo of it, and see if you can, see what you could do with your stuff. See if you can break your stuff. Don’t do it with live things, folks, because you will never hear the end of it, and you may actually get in trouble. But like anything you’re working on in development you pull down locally, maybe throw NBomber at it and see what what it does, right?

Anton : Yeah.

Jamie : So let’s say folks want to learn about NBomber and give it a try. Is the website, nbomber.com, the best place to start? Is there like a demo or like any docs or videos and things on there?

Anton : Yeah, yeah. There is basically you will find everything on the on the NBomber website. There is a link on documentation, and there is a link on our "Hello, World" tutorial. And the link on the GitHub repo where you can find the demo project, which contains a lot of demos for different technologies. You can run it locally, spinning up containers if it’s needed, because some of the test comes with a dependency on on Redis database or if it’s Redis load test, or on the GRPC with a GRPC server, and so on and so forth.

So Yeah, in addition, I suggest to look at integrations on plugins. Also, we’ve been working on the tool, what so-called NBomber Studio. It’s a UI, sort of, UI dashboard for your load tests, which provides you observability for your load test sessions. But also it soon will provide the feature which we call "private cloud", which will allow you to basically create a test and deploy it any cloud. It will be integrated first with Kubernetes. We are almost finishing and polishing this layer. So it basically provides you full end-to-end load testing flow which you can get from the cloud providers. But from the cloud I would say SaaS solutions, but with NBomber Studio you can deploy it within your private environment and run your load test in your environment with unlimited durations, unlimited number of machines, and so on and so forth. So This is quite appealing solution.

Jamie : That sounds really good, because like yeah, if I’m wanting to, without having to spin up something on Azure, to hold my resources whilst I do my test, I can say, "hey, NBomber Studio, go create me one of these these private clouds, put my thing there and test it for me at the same time." That’s That sounds brilliant. Yeah. Love it.

Anton : Yeah. If you start comparing prices, you will be very surprised because NBomber provides very attractive pricing, just flat price for the whole company. Currently is like 2k and a half per year for the whole company, unlimited number of users can use it, one, whatever test.

So you’re basically receiving the same quality as you could buy in some cloud provider, in cloud I mean SaaS provider, for load testing, But here you you have the privacy, you can run them locally, you can run them in Kubernetes. So it’s a super, super attractive feature.

Jamie : Amazing.

So folks are gonna go check out the website. By the time that this comes out for sure, they’ll be checking out NBomber Studio and taking a look at the private cloud offering. What are some of the other ways to catch up? Like the you know there’s— I did some Googling ahead of time. There is an NBomber blog, which I will have in the show notes. But like if folks are going, "hey, this Anton chap, I wanna know what he’s talking about."

Anton : Yeah, yeah. I’m on LinkedIn, it’s easy to find me there. And also if you are interested in NBomber, there is a lot of projects on GitHub which you can contribute. Plus, you can subscribe on updates on the LinkedIn page of NBomber Company. And you also can find me on on Twitter. My nickname @AntyaDev.

Jamie : Yep. I’ve got those. I’ve got that written down. Those will be in the show notes, folks.

Anton, this has been a fantastic conversation. I say this with everyone, but I genuinely mean it with everyone. I’m walking away from this conversation knowing loads more about load testing as a subject, but also I’m gonna go ahead and try and put Nbomber into at least one of my apps and see what happens. Because I’m very interested to find out.

Anton : I appreciate your feedback. Feel free to ping me if you need my help. Thank you very much.

Jamie : Thank you.

Wrapping Up

Thank you for listening to this episode of The Modern .NET Show with me, Jamie Taylor. I’d like to thank this episode’s guest for graciously sharing their time, expertise, and knowledge.

Be sure to check out the show notes for a bunch of links to some of the stuff that we covered, and full transcription of the interview. The show notes, as always, can be found at the podcast's website, and there will be a link directly to them in your podcatcher.

And don’t forget to spread the word, leave a rating or review on your podcatcher of choice—head over to dotnetcore.show/review for ways to do that—reach out via our contact page, or join our discord server at dotnetcore.show/discord—all of which are linked in the show notes.

But above all, I hope you have a fantastic rest of your day, and I hope that I’ll see you again, next time for more .NET goodness.

I will see you again real soon. See you later folks.

Follow the show

You can find the show on any of these places