S07E02 - CQRS, System Maintainability, and Pragmatic Tech Choices with Dejan Miličić

Sponsors

Support for this episode of The Modern .NET Show comes from the following sponsors. Please take a moment to learn more about their products and services:

• Metalama, reduce your boilerplate code by up to 15% with Metalama’s C#-to-C# template engine and reduce your code complexity today!

Please also see the full sponsor message(s) in the episode transcription for more details of their products and services, and offers exclusive to listeners of The Modern .NET Show.

Thank you to the sponsors for supporting the show.

Embedded Player

The Modern .NET Show

S07E02 - CQRS, System Maintainability, and Pragmatic Tech Choices with Dejan Miličić

Supporting The Show

If this episode was interesting or useful to you, please consider supporting the show with one of the above options.

Episode Summary

The interview features Dejan Miličić, the head of Developer Relations at RavenDB, a NoSQL database producer. He shares his extensive experience in software development, including his long history as a software consultant and his involvement with open-source and mentoring startups. Dejan emphasizes the value of learning new ideas and being open to new thoughts within the software development community.

The conversation transitions to the architectural design pattern of Command Query Responsibility Segregation (CQRS). Dejan explains that CQRS separates a system into two distinct areas: one that handles commands (manipulating data) and another that deals with queries (retrieving data). He emphasizes the underlying motivations and benefits of this separation, such as improving maintainability and simplifying the complexity of the system. The concepts of essential and accidental complexity are introduced, where essential complexity arises naturally from the domain and accidental complexity is created through design choices. Dejan discusses how adopting CQRS can help manage accidental complexity and lead to better long-term system design.

As the discussion unfolds, the importance of not blindly adopting architectural patterns is highlighted. Dejan expresses his scepticism towards the notion of rewriting existing codebases to implement new patterns like CQRS, arguing that maintaining and adapting an existing system is often more beneficial than starting anew. He also explains that CQRS can complement other patterns, such as event sourcing, and that while they are related, one does not necessarily require the other.

Dejan shares insights on the application of CQRS and event sourcing in real-world projects. He encourages an understanding of the system as a whole and the interdependencies between its parts, countering the trend of microservices being viewed solely through the lens of scalability. He emphasizes the importance of pragmatism in choosing the right technologies for a project and ensuring that the system’s architecture aligns with its operational requirements rather than adhering to trendy frameworks.

The conversation also touches on the necessity for developers to be aware of the historical context of their craft. The importance of systems thinking, understanding principles over picking trends, and learning from past experiences are all underscored. Dejan reflects on how knowledge of fundamental principles can guide developers in making informed choices rather than just following the latest fads.

Towards the end of the interview, Dejan shares resources for further learning, including workshops focused on NoSQL databases, and recommends engaging with him on Twitter and LinkedIn for continued discussion. The interview concludes with mutual gratitude for the engaging conversation, underscoring the continuous journey of learning within software development.

Episode Transcription

Essentially, when you look at it, what we are doing, we are not building software. We are changing it, we are maintaining it. Because when you look at the typical lifecycle of the application, enterprise one I would say, or any big system where you invest lots of time lots of money building it, you want to exploit this for 5-10-15 years hopefully. So when you look at the percentages, you’re building something for one year and then you are using it for nine more years, let’s say. It’s a 10 year life cycle. So when you think about it, you spend 10 times… er, 10 percent of your time building something and 90 percent of the lifecycle of the application or the system responding to change requests, building new things, changing, adapting, and maintaining. So essentially, our job is not to build out. Our job is to sustain all these request changes. I think that’s the first point we need to clear.

Welcome friends to The Modern .NET Show; the premier .NET podcast, focusing entirely on the knowledge, tools, and frameworks that all .NET developers should have in their toolbox. We are the go-to podcast for .NET developers worldwide, and I am your host: Jamie “GaProgMan” Taylor.

In this episode, Dejan Miličić joined us to talk about the CQRS pattern, how it came from CQS, what CQS is and how it’s related to Alan Kay’s original ideas for object-oriented programming, being pragmatic as developers and engineers, the importance of system design and system’s thinking, and how we all need to realise that our software lives on for years after we’ve pushed our changes to the repo.

So I, indeed, agree with you that people should pay more attention to system design. Start looking at the whole picture. And the extreme of this thinking would be: okay you will go into job interview, they will ask you about, I don’t know, quick sort you, will implement it on the whiteboard, and then six months later on you will go and purchase or maybe download a NuGet package with with a quick sort implementation, because you know it makes sense; you don’t want to implement something that have been implemented this many times before.

Whilst this episode doesn’t focus on .NET per se, I think that the subjects that Dejan and I covered are incredibly important for anyone who wants to seriously level up their game. In fact, Dejan proved this point when he said:

We [are] all, like, containerised into our own small silos and working on our own without being aware of the whole system. When you look at what people go through when they prepare for job interviews at, I don’t you know, Google, Facebook, nowadays Meta, Microsoft, you have all these books on the system design. And then they go, instead of, you know, going to the job interview after 20 years of experience and talking about what you do and what you know, people with 20 years of experience still sit down and learn and prepare for the system design interview. I’m not saying they shouldn’t be preparing, but some of my colleagues told me that that was the first time in their careers that they started looking at the system design as a discipline.

Anyway, without further ado, let’s sit back, open up a terminal, type in dotnet new podcast and we’ll dive into the core of Modern .NET.

Jamie : [0:00] So Dejan, thank you ever so much for joining us today. I know we’re going to talk about something that’s more of a, sort of architecture pattern, design pattern, theory sort of stuff; but I really, regardless of whoever comes on the show, I’m really really appreciative of people who who spend some time with us and help us to learn something new. Because it’s quite important to to at least be exposed to new ideas and new thoughts right?

Dejan : [0:25] Thank you for having me today.

Jamie : [0:27] No worries, no worries.

Jamie : [0:29] So I wonder could you perhaps give the listeners a bit of a feel for who you are perhaps an elevator pitch, you know, a short description of, “hey I’m Dejan and I work on these things. And this is what, you know, this is my history of software development.” You totally don’t have to turn it into a resume but just like, “hey you know this is me, this is what I do. This is you know my experience with things,” that kind of thing?

Dejan : [0:53] Sure.

Dejan : [0:54] I’m still learning, that would be the first thing. Currently working at RavenDB, [the] NoSQL database producer, as the head of DevRel, which basically means that I’m still writing code but without any deadlines, which is a pretty comfortable position. And I’m also helping people onboard this database. So I’m doing educations, workshops, code walkthroughs, code reviews, best practices, recommendations, and similar things.

Dejan : [1:23] Other than that, I’ve been working as a software consultant for something like 25 years so far, so I’ve seen everything. I enrolled in university before Java existed, so I’m like a dinosaur of computer science. I studied what we call here in my country “Informatics,” which would be computer science in the United States. So heavy mathematical background, theoretical programming foundations. I’m occasionally contributing to the open source. I’m a big believer in open source, both as a means of collaboration and whole philosophy.

Dejan : [2:04] I’m enthusiastic about functional programming. I returned to it in the past couple of years. And I’m also, in my free time, from time to time, I’m mentoring startups and helping develop further local community of software developers so sometimes days are short.

Jamie : [2:27] No, I get that. I get that. People often ask me how I find the time to do all the things that I do, and I’m only doing maybe a quarter of what you do

Dejan : [2:38] Well, it’s all matter of a good PR, you know.

Jamie : [2:44] Absolutely, absolutely.

Jamie : [2:45] So let’s get talking about CQRS then. I know that we got connected and we had this idea of, “let’s talk CQRS, a little bit of its background, and maybe how it links to event sourcing, and how one requires the other but not the other way aroundWhich is really quite an interesting position to be in. So I wonder, could you perhaps give us maybe a 10,000-foot view of what CQRS is? Other than just four letters.

Dejan : [3:18] Sure. So I’m pretty much sure more or less everyone heard about CQRS. It’s short for “Command [and] Query Responsibility Segregation. " And 10,000-feet overview would be you split your system into two verticals: one vertical is in charge of commands and the other one is in charge of queries. And it may seem a bit like strange choice of system separation, but I hope today we can cover [the] motivation behind this, what problem we are actually solving here, and and how it can benefit people who are both doing event sourcing, but also those who are not doing event sourcing.

Jamie : [4:04] Sure, sure

Jamie : [4:06] Yeah I remember, so my experience with CQRS as a whole is maybe one or two projects; and for those listening, I work as an independent contractor. So when I say, " one or two projects," I’m in for three months to six months, and then I’m gone, right. Which is, you know, not a huge amount of experience. But you know having built a couple of successful apps and services with it some people would say, “that was a great amount of experience.” It’s really not.

Jamie : [4:35] I remember a few years back—maybe six or seven years, yeah six or seven years ago—I was at a conference and someone was talking about how great CQRS was, and quite literally used the line, “you have to rewrite all of your apps to use CQRS.” And whenever someone says to me, “you need to rewrite everything to use this brand new pattern,” I immediately switch off. And so my introduction to it was this very excited person who was, you know, really trying to evangelize it and tell me how great it was, but started with, “you need to rewrite everything from the ground up.” And at that point, even as an independent contractor, I sort of sit there and go, “no. Because working code is infinitely more valuable than non-working code.”

Jamie : [5:26] So what I’m getting at is my knowledge of CQRS is install a whole bunch of these packages; s eparate everything into commands; write 10,000 lines of code to be able to read something from the database; and then profit. That’s not quite CQRS, is it?

Dejan : [5:47] Well look, I would first go back to something you mentioned and you basically said, if i remember correctly, you said, “I’m a consultant. I land at the company or at the contract, I do my job and I leave.” And I’ve been a contractor myself, I’ve been consultant, I’ve been working in outsourcing industry. And I think one of the biggest problems with this approach is you are not living the pains or joys of your choices, especially when it comes to architecture.

Dejan : [6:18] Essentially, when you look at it, what we are doing, we are not building software. We are changing it, we are maintaining it. Because when you look at the typical lifecycle of the application, enterprise one I would say, or any big system where you invest lots of time lots of money building it, you want to exploit this for 5-10-15 years hopefully. So when you look at the percentages, you’re building something for one year and then you are using it for nine more years, let’s say. It’s a 10 year life cycle. So when you think about it, you spend 10 times… er, 10 percent of your time building something and 90 percent of the lifecycle of the application or the system responding to change requests, building new things, changing, adapting, and maintaining. So essentially, our job is not to build out. Our job is to sustain all these request changes. I think that’s the first point we need to clear.

Dejan : [7:18] So from that perspective, “let’s rewrite everything,” represents kind of defeat. Defeat in the sense that you throw away everything you’ve built and you start from scratch. In my experience, this is the desperate measure when you cannot help it anymore, when the application you’re working on is beyond repair.

Dejan : [7:44] But essentially, if we are building high-quality systems, then this total rewrite moment or decision will hopefully never come. So I would say this is first criteria for determining are you doing a good job or not. Unfortunately, as we all know, rewrites are not so great. And I’m pretty much sure that engineers working in other branches of engineering practice are looking a bit with a strange eye towards us. We are building something, this is functioning properly and then at one point in time we say, “okay let’s throw everything away. Let’s build it from the scratch”. And when you look at this theory that actually building software is a process of gaining knowledge, this also makes this decision to throw everything away and build it from scratch very dangerous, because all this accumulated knowledge and understanding of real-life systems that we are modeling as software is being thrown away. So it’s huge waste, huge waste.

Dejan : [8:57] So now when you look at these 90%, this is a response to a change. We are building systems that have two types of complexity: one of them is essential complexity, and the second one is so-called ‘“accidental complexity.” So essential complexity is the complexity of the domain that we are modeling itself. We cannot avoid this, we cannot change it. This is being mandated by customers or our users who are going to use it. So they will say to you, “okay I need this accounting system,” or, “I need this orders processing system.” And not too much of a choice there.

Dejan : [9:46] Where we do have a choice is this other type of complexity called accidental complexity, and this is complexity of the implementation. This is complexity that we engineers, developers are creating; it is accumulating over the years and after five or six years, or ten years when someone says, “let’s rewrite it,” this is because this accidental complexity outgrow[s] all, you know, manageable ranges. And you actually have a system which you cannot easily maintain it.

Dejan : [10:23] So, over the years, various techniques have been developed to manage this accidental complexity, in other words, to build simple systems. And I always like to go back to mathematics, because mathematics is this type of modeling that had been proven over the centuries. where mathematicians are building whole systems which are rather easy to maintain. And on the wings of that, back in 1988 or 1987, Bertrand Meyer, author of various books and also author of Eiffel Programming Language, he emerged with CQS. Now, this sounds similar to CQRS, but this is something that represents a concept that came before CQRS.

Dejan : [11:25] So CQS is short for Command Query Separation. And it was presented in the book Object-Oriented Software Construction. It’s a really heavy book, 1,370 pages. So around half of that book, in chapter 23, there’s a “principles of class design” chapter, and essentially Bertrand Meyer there claims that, “interface design of your application boils down to the class design.”

Dejan : [11:59] So the Command Query Separation principle comes down to two concepts: commands and queries. So every single interaction that you have with your system can either be command or query. Commands are changing state of the system without returning any results. So you create a command, you send it to the system, system will accept it, validate it, execute it and change the state. The other type of the interaction is query. You will send query, it will return back to you state of the system without altering the state itself. In other words, these two principles, these two patterns can be summed up in a sentence or principle that asking a question should not change the answer.

Dejan : [12:56] Now, you may ask yourself, “where this principle comes from?” Well, what Bertrand says in the bool, and what some other people are saying is, that simply over the years, people who were maintaining a large complex system came to [the] empirical conclusion that structuring your system in this way is helping keep complexity low and keeping it more easily maintained. Now, what’s important to understand here is that the difference between mathematics and programming comes down to a couple of aspects: one of them is so-called side effects. So in mathematics, when you apply a function to arguments, this function will always return the same results. And mathematical functions are so-called pure functions. When [a] function is not pure, when it’s either a reading from the environment or changing the environment, this is called side-effect. Now, if we go and apply this to CQS, you can see that commands that we are sending to system will produce some side-effects, which could be writing to a database or sending email. While queries will not produce any side-effects.

Dejan : [14:21] So side-effects are good when they are intended. As I said, you want to write something to a database or you want to send email, you want to write something on the screen. But at the same time, side-effects, when not intended, are very bad because they will produce unexpected changes in environment, which will now affect other pieces of your code. So by splitting your system, your interactions with the system, into queries and commands, you are limiting the scope of these side-effects, which now can happen only in commands. And all of this is applicable to apply to the surface of your application, which would be the entities.

Jamie : [15:10] Right. Okay.

Jamie : [15:11] So if I have, say, a web API that is accepting a request, like an HTTP GET to retrieve some, I don’t know, we’re building an e-commerce website, right? I got an HTTP GET that says, “for product 123, get me the product details,” right? My request is coming in. I want to then separate the, I guess, the flow through the app into a command and a query. Everything that’s going to the database, or to some kind of data store can be my query; and that’s going to have, doesn’t matter how many times I ask for that, as long as the information exists in the database I can… it should be idempotent right. I can ask it a million times and always get the same response and the same pathway through the system every single time; whereas from what I’m taking from what you’re saying, there’s a command side of things which is perhaps I’m logging the request, perhaps I’m doing some analytics, or perhaps I’m ensuring that the user is authenticated. That’s separate from my query, and that’s where the segregation part of it comes in. Am I thinking along the right lines?

Dejan : [16:24] Yes. And when you look at the HTTP verbs, GET is or should be read-only operations. So you are reading state of the system, and observing the system, when you apply some common sense, should not change it. This is what we, I would dare to say, like expect from the system. And verbs like PUT, PATCH, [and] POST are essentially commands in this patternm and they are altering the state of the system. So you’re either creating something [or]you’re updating it. Maybe I did not mention DELETE verb as well. So this kind of like interface level separation is helping you comprehend system better.

Dejan : [17:13] So over the years, this proved to be working well. And then CQS, command query separation, was then propagated further down. What you mentioned when you just described REST API, was what happens under the surface when your either command or query is being processed. Well, this is now [the] realm of CQRS. So CQS is operating at the class level or meta level, in other words, on the interface level, interface towards developer who is calling these commands or queries. CQRS is now bringing this separation, or as CQRS stands for responsibility segregation, down to the architectural level. So, as you are separating things on the surface level, now you will start splitting your system into two halves. Each one of them is responsible for handling either commands or queries. And this is the moment when you heard about the idea of rewriting everything to split it.

Dejan : [18:32] So, you will end up essentially looking at the architecture of your application. You will end up with two, let’s call it two separate sub-applications within your big system. One of them will be handling commands, verifying them, altering your domain objects, storing them into the database and returning responses to you in the sense that, “okay, your command have been accepted and processed, but it will not reflect any state of the system.” Your queries will be handled by a different vertical in your application, which should be much, much simpler. CQRS architecture essentially brings all complexity to command handlers, and queries are now becoming very simple. Because queries are accessing just storage, which can be specialised, it is reading the state of the system and returning back to you. And this also means that now you’re splitting all the way down to the database.

Dejan : [19:44] So, queries can be reading view models that you prepared ahead of time. And this also means that when you look at this separation. Its consequences are that you can scale your system in a better way, or more appropriate way, because now reads and writes, or commands and queries which are representing reads and writes, can be scaled independently. They are handled by separate components in your system. And this is giving you flexibility because you can go and optimize your read models for performance without affecting [the] write side of your application.

Dejan : [20:26] So let’s say you are creating new orders, you want to optimize for accessing orders on Black Friday. And then if you’re reading shopping carts, you can use specialized data structures, specialized tools to produce view models, which will be read-only and picked up by the queries. So you have this freedom, this flexibility of changing read-side, putting Redis or something like [in place] that if you want. And it turns out, again, through [the] practice of applying this architectural pattern, that you have better complexity management because complex business logic for writes can be isolated from the read side of your application; and [the] read side is usually much simpler than the write side.

Dejan : [21:20] Also, one of the consequences is security. Your commands and queries can have different security policies and you can handle that with more granularity.

Jamie : [21:33] I like the idea that we can then optimise the separate, sort of halves of our application. We were talking, we both used the idea of like an e-commerce system, and you talked about like that time-based optimisation. We know, for instance, on Black Friday we’re going to have a lot of people making orders, so we’ll give that side of our system more processing time. We’ll maybe, what is it, vertically scale, horizontally, whichever. We’ll scale that side of the business because we know we’re going to see more people making orders, creating orders on Black Friday than there are going to be people perhaps, say, altering their accounts or checking on the state of current orders or whatever. People are going to be hitting the website, buy, buy, buy, do this thing. Whereas perhaps for the rest of the year, we can scale back the other way because we know that perhaps fewer people are going to be making purchases, say, in the middle of summer or in January or whatever. That makes a whole bunch of sense.

Dejan : [22:44] Correct. Usually, when you look at the structure of your application and the usage patterns, you see disparity between the number of the writes and number of the reads. And for me personally, as a consultant, I’m very careful not to use something that I internally call, “driving on autopilot.” And, “driving aon autopilot” would be having these favourite architectural patterns and applying them always, using them as a foundation of every system I’m building. So what CQRS is also implicitly guiding you [to] is this blank slate. You start thinking first about, “okay, this is [the] magnitude of reads that I can expect. This is magnitude of writes that I can expect. What are the requirements for each of these verticals?” And then I can go and adjust each one of them separately.

Dejan : [23:45] However, when you mention CQRS, most people associate this immediately with event sourcing. And this is because even though CQRS is a technique that emerged from practice, when you start looking at these conference talks, you will probably heard CQRS first from Greg Young in 2010. Greg started popularising event sourcing, and CQRS is one of these techniques that’s tightly coupled with event sourcing.

Dejan : [24:23] Just to revisit what event sourcing is: you are not recording the state in your database you are recording now events which are changing the state, and you are computing state on the fly. So by recording events you are not throwing away any business information, and later on you are creating projections given the meaning of the events in the context of your businesses. So when you think about it, if you are saving events, you will need a mechanism which are folding these events and computing current state.

Dejan : [25:08] And CQRS is applicable specifically here because [the] command side will be saving events and [the] query side needs to return state to you. So you can see already that different persistence mechanisms are being used: one for events, the other one for state. And you have this concept of eventual consistency where there’s this kind of trigger or loop within your system where event storage, accepting events will trigger computation of the state which will be placed in the state persistence, and queries will access this state persistence and return state to you.

Dejan : [25:58] What is important to underline, however, is that event sourcing without CQRS is almost impossible. I wouldn’t say impossible, but it’s hard and it doesn’t make sense. However you can use CQRS without using event sourcing. CQRS is a simple principle of separating your application into two verticals, and then all these benefits that I mentioned a moment ago are applied to you, you have more flexibility, you can fine-tune both of these verticals.

Dejan : [26:32] And when you think about it, we should be using [the] best possible technology for every project we are making. So this means that you should use, for example, doing the same way if you’re doing artificial intelligence these days, you will probably go with Python. Well, [the] same logic should be applied to bits and pieces of the system you’re building. So for example, if you’re building a product catalog, you will have to use a specialised tool for providing full-text search. In the same sense, if you have lots of view models, if you create a dedicated pipeline for creating these view models, you will be better off. The whole point here is to use the most applicable tools for specific problems you have.

Dejan : [27:29] So we come down to the fact that some people are using CQRS even though they are not aware of it. Because let’s imagine you have a simple system with a single database and you have problems with performance, with latency, and then you go and introduce to Redis as a write-through cache. So expensive queries you might run against relational database their results will be cached inside of the Redis, and then subsequent queries will be reading directly from Redis. You could say that this would be [a] flavour of [the] CQRS architectural pattern, because you are writing into a relational database, your commands are hitting relational database, and your queries are being directed towards Redis to read pre-computed, upfront prepared, read models. So even though you did not start consciously with CQRS, you introduce this mechanism of customized read and write side along the way to optimize your system.

Jamie : [28:33] That makes sense.

Jamie : [28:34] So even though I’m, yeah, because it’s one of these design patterns that kind of, it makes sense regardless of even if you’re not doing event sourcing, right? You can actually separate your code base in logical ways, right? There’s a lot of different ways that you can separate your code base. You can go domain driven, you can go vertical slice, you can do, you know, design pattern du jour. But I guess, depending on what you want to optimise for, you know, if you’re optimising for scale perhaps, if you already have scale—so just real quick folks, if you don’t have scale, don’t optimise for scale because, you know, in the Pragmatic Programmer, I think it was, i think it’s… I’ve got it in front of me, I think it’s Dave who says that, “if you want to optimise for Netflix scale, go get Netflix scale first.” But if you’re optimising for scale because you have a requirement to optimise for scale, and to use something like event sourcing, then CQRS is definitely the way to go. But you definitely need to have that business requirement, right?

Dejan : [29:46] Absolutely. Absolutely. And this is the part of the autopilot I mentioned in the beginning. Also, what could be characterized as a curse of our profession is that you go to these conferences, you hear about exciting technologies, microservices, scaling, you know, we all read what Google did, you know, people using Hadoop, whatnot. And then the next project comes in, and then you have this temptation of almost childlike urge to play around with technologies. But as you correctly noted, CQRS is applicable in various situations. So for example, when you think about it, the essence of high performance system is using asynchronous operations. If you’re using synchronous programming, especially in the distributed system this is my experience, is this is very bad; lots of blocking, lots of waiting, so by splitting into commands and queries and by not accepting that command will mutate state of the system and right away return this new state for you, you are naturally guided towards asynchronous programming. So if you’re applying this pattern, you will get some things for free, so to say.

Dejan : [31:12] If you’re using microservices, this will be a natural match for you. If you’re using something like actor model, Microsoft Orleans or Akka, this will also play nice for you because actors are essentially accepting commands, processing them on their own in an encapsulated environment. And then you need to read state from actual persistence where actors are acting. So you could say that you are writing in one place, physical place with your commands, and you are reading state from different persistence.

Dejan : [31:50] As I mentioned, essentially, whenever you have disparity between reads and writes, and this is the case with most applications that we are building. And as you correctly noted, it could even go down to the fact that same application in different days or weeks of the year can have different patterns. This will give you freedom up front to be able to optimise. And premature optimisation, we all, I think we all agree, is maybe not [the] root of all evil but something that proves to be a bad choice up front.

Jamie : [32:28] Yeah, I think I love that quote. But, I also think that it was hyperbolic when it was written. And then it’s still hyperbolic now, you know, because like you were saying at the very beginning, you’re still, if you start optimising at the very beginning of your project, you don’t have the knowledge of what the project should be yet. And I can always tell when I’m talking to someone who is perhaps nearer the beginning of their career than maybe the centre of it. Because when I say that to people who are in the beginning of their career, they say to me, “wait, wait, wait. We know what the system is. We know that it’s going to, you know, allow us to sell widgets or whatever.” And I say, “fantastic. Come and see me in six months and tell me that the system is still just selling widgets,” right? Because that’s not how that works.

Dejan : [33:26] Absolutely.

Jamie : [33:26] You know, like you were saying, you’re still uncovering information about the system.

Jamie : [33:30] And I wonder, and I feel like I’m dropping this question on you now without any kind of prep work. And I don’t know whether you have any experience or knowledge to answer this question. But I wonder whether software developers should be taught systems thinking rather than just jumping straight into, “this is how we write a for loop.” Like being able to take the way that, like the canonical example of what systems thinkingis, “solving a problem by looking at the systems that exist around it.” Right.

Jamie : [34:01] And I never understood it at the time when I was at university. But one of my lecturers used to spend forever, a long part of the semester, talking about this one system that he designed for the fuel injection system for an airplane; and I never understood it at the time. I was like, “this is computer science why are we talking about airplanes? Why are we talking about fuel injection?” And looking back on it now, I realise that he was trying to teach us systems thinking; but I just wasn’t paying attention to that part of the course. I wasn’t… there were other parts of the course where we were dealing with graphics and sound, and doing interesting things with computers and robots. But he was saying that what we need to do is we need to think about the solution as a whole and figure out how it works from there rather than saying, “cool, I’ll build an API endpoint to let you get some information from the database. Boom, we’re done. Right, move on. What’s the next API we need?”

Jamie : [34:58] I don’t know how you feel about that and whether you have any thoughts on that.

Dejan : [35:02] I absolutely agree with you.

Dejan : [35:03] So when you look at the systems, a system is a whole that consists of parts. And all these parts are affecting behavior of the system, or properties of the system. And all these parts are interdependent for producing effect. But what is important to remember is that not a single part has an independent effect on the system. And also when you look at the definition of the system itself, it’s not just sum of the parts, it is producing something more. So when you do, you have this term reductionism. Reductionism is an attempt to analyse [the] system by splitting it into individual parts. But if we go back to the definition of the system, that system is a whole that is consisting system of the parts producing some effect that not a single part can produce, you will understand that reductionism is a failed attempt in its very roots. So this means that you cannot improve whole system by splitting it into parts, analyzing, taking one part and improve it.

Dejan : [36:22] So system can only be improved as a whole, and this means that all these parts which are interacting should be improved on their own to produce a joint effect on the system. And I’m afraid that we are forgetting about this. We [are] all, like, containerised into our own small silos and working on our own without being aware of the whole system. When you look at what people go through when they prepare for job interviews at, I don’t you know, Google, Facebook, nowadays Meta, Microsoft, you have all these books on the system design. And then they go, instead of, you know, going to the job interview after 20 years of experience and talking about what you do and what you know, people with 20 years of experience still sit down and learn and prepare for the system design interview. I’m not saying they shouldn’t be preparing, but some of my colleagues told me that that was the first time in their careers that they started looking at the system design as a discipline. And the fact that when people speak about microservices, you will hear this mantra that, “microservices are for scaling.” But essentially, if you start reading about it, it is indeed for scaling, but for scaling of the teams.

Dejan : [37:51] When you look at all this literature, all these books and talks about system design, you will see that actual scaling, especially the horizontal one, comes down to very simple principles like having a queue, having an elastic set of workers which can consume queues, and similar simple techniques; all of them invented in the 60s or 70s. But for the microservices, microservices are actually for managing teams of, I don’t know, you have 150 people. Now the challenge is how to organize them to all work together, collaborate. And this is primarily the idea behind microservices, that teams are owning one or more microservices.

Dejan : [38:36] So I, indeed, agree with you that people should pay more attention to system design. Start looking at the whole picture. And the extreme of this thinking would be: okay you will go into job interview, they will ask you about, I don’t know, quick sort you, will implement it on the whiteboard, and then six months later on you will go and purchase or maybe download a NuGet package with with a quick sort implementation, because you know it makes sense; you don’t want to implement something that have been implemented this many times before. So I think the best developer, the best consultant would be something who is ready to run his or her own company.

Jamie : [39:23] Yeah, I agree. I agree. And I think the core thing that I hope we can, we can get through to people with this conversation: it goes back to something you keep saying all the way through, and that is, “we don’t know what we’re writing until we’ve written it.” And so with systems design, with, not going into autopilot and automatically trying to optimise or trying to shoehorn a specific design pattern into an application, or shoehorn a specific technology in, we will then learn more about the the system we’re designing. But then, like you were saying, you know, we can do… we can apply CQRS in one form or another, or indeed event sourcing in one form or another, other without having to rely on a specific design pattern to get us there, right.

Jamie : [40:26] And I think, so I’m taking from what you were saying about like regardless of whether a person is an independent contractor consultant or whether they are a software developer working for a single company for, you know, their entire career, or part of their career whatever; knowing that these things exist, I think, is the hardest part. Because if you don’t know that something… it’s one of those, what is it? Was it Donald Rumsfeld who said, “unknown unknowns”? If you don’t know that you don’t know about it, you’re never going to go out and learn about it, right?

Dejan : [41:01] Absolutely. And for me, that was the biggest value I got when talking to people more experienced than I am. Because, for example, most people are not aware that the actor model does exist. And this is something like 50 years old, very big systems [are] built on top of it. It offers parallelisation. It offers this original flavour of object-oriented programming. If you go and read what was the original vision of Alan Kay, it was a set of the objects which are messaging each other. And in subsequent interviews, he goes on saying that for his original idea, messaging is more important than the objects themselves. And now when you think about what is the mainstream object-oriented programming flavour today, this is C++/Java style, where you claim to have, for example, you can do inheritance and you can override implementation. But then you go and read about the Liskov Substitution Principle, which essentially is saying that, “subclassing is not subtyping.” So as a consequence if you override the implementation, this inherited class when plugged in where the original objects were the system, will not work properly anymore.

Dejan : [42:29] So for me it’s important to go back, and this is what I’m doing all the time: go back, rediscover something that exists I don’t know, 40 years I wasn’t aware of it. Feeling ashamed and then stand in with the principles. Once you learn the principles, you will see that everything else can be classified and understood more properly. So for example, if you want to learn React, you could as well go, and learn about Elm and then see that essentially React is just implementation of the MVU pattern that was introduced with Elm.

Jamie : [43:12] Yeah, 100%. There’s almost nothing new under the sun, right.

Jamie : [43:16] And my own personal opinion, so this is… what I’m about to say, it’s not going to get me any friends—I’ve been saying that phrase a lot today, not in the recordings but to lots of people. I wonder with a lot of, and this is me just talking out loud, we totally don’t need to talk about this, this is just a thing that I’m thinking, I wonder whether a lot of the the libraries that are coming out now where they’re like, “we will eschew the lessons of the past; and just don’t worry about it we’ll create this library, and it will be the best thing ever.” I wonder whether some of those are going to end up in a situation where they hit upon the wall that perhaps our luminaries, the giants whose shoulders we’re standing on, came up against and went, “right we’ll pivot and do it slightly differently.” Except that back then these, what they were building, were not production systems; but now what people are building is a production system from day zero.

Jamie : [44:17] So like I wonder how many libraries out there are doing things like, you know, “we’ll drop the idea of even having a database. We’ll just have something that’s ephemeral, that represents our entire state, that you won’t switch off because you’re not going to switch off. Because if you switch it off, you lose your entire state.” But also, in six years’ time, [they’ll] realised, “hey, that might not be the best idea because now if you stop paying your bill or, you know, there’s no synchronisation back to disk or anything like that. So when the data center goes down…” That’s a really bad example of what I’m getting at. But I feel like there’s a lot of libraries and frameworks that are coming out in the last couple of years where people are saying, “no, don’t worry about it. Let’s just ignore the lessons of the past and we can innovate faster.” And you probably can innovate faster, but you’re going to hit the same wall that the people 20 years ago hit, which is the same wall that the people 40 years ago hit, except that their apps weren’t production and yours are now production. You know?

Dejan : [45:19] I agree with you. So when I heard that [the] Swift programming language [was] coming out, and when I inspected a bit, Iwas kind of, “oh no. Why do we need one more programming language?” When we are speaking about the cloud, I saw some examples of companies getting grants, especially startups getting grants; let’s say you you get a grant of hundred thousand dollars for one of three major public cloud providers; you start building systems on top of it, and if you’re not careful you run into this potentially very unpleasant vendor lock-in. So instead of using some kind of, I don’t know Kafka or RabbitMQ which are like, kind of, let’s call it kind of generic open source project, you use [the] proprietary queue of the cloud provider. And now you can imagine what will happen after two or three years, after you use five or ten of these proprietary technologies. As a startup, you will get another grant by another cloud provider and you will simply be unable to use it for anything because you have vendor lock-in, you’ve been locked there. And it’s very hard to, you know, you cannot stop business operations for, I don’t know, three weeks and let’s say, “let’s now migrate to another cloud provider.” It’s simply something that will not happen.

Jamie : [46:44] Sure, sure. And I think, how do I put it? I don’t feel as though enough, maybe I’m just stuck in my ways, but I feel as though there aren’t enough devs, engineers, coders, whatever word you want to use, who are doing what you’re saying [that] you’re doing. Looking to the past and seeing where we came from and what is like, you know. I wouldn’t have made the connection between React, MVU and Elm, but you just did because you can see the the evolution, right. And so that means that if someone has come from an MVU background, picking up React is just a case of learning the syntax: you’re still doing the same thing, right. And I wonder whether along with systems thinking maybe we should be teaching the software devs a little bit more about, “okay, so here is, and I don’t think this would work in academia, “but like here is the history of some of the most popular things that you’ll be using, you know. So like you know React came from MVU. And .NET MAUI came from Xamarin, which came from, you know other experiments at getting things working cross-platform,” just to pull some ideas out of the air there.

Jamie : [48:00] I wonder if more developers should be perhaps spending a little time actively looking into the history, or indeed speaking to those people whose shoulders we’re standing on, and just saying, “hey, so right, okay. I’m using this particular programming language, you know, what were you using? How did you build it?” Because, you know, there’s a wealth of experience out there that is not just, you know, the people who are attending the conferences. Because let’s be honest, the people who are attending the conferences are those who can afford to take the time to attend a conference, right? Either during a work day or at the weekend, perhaps they don’t have as many responsibilities as others. So, but that means that if you’re on a team of 20 people, and only one person can attend the conference, there’s 19 other people who have information that could be useful to what you’re building. And if you’re in a company where there are 25, you know, supremely senior, excellent engineers and you’re not talking to them, maybe you should be. I don’t know. How do you feel about that?

Dejan : [49:08] I agree with you we should be learning principles; and I’m also personally missing this kind of information from my university days. And then also, when you think about it, somehow we look at the era of the internet as the beginning of things; and people were making big complex system systems before era of the internet. So when you think about [the] simplest possible use case: credit card processing, you can think about the magnitude of the operations that are happening there. All these things that we are trying to achieve like horizontal scalability, maybe they’re going with vertical scalability, resiliency. When was the last time you were not able to use your card? I cannot recall. So what would be good is to go back and study how systems were built back then.

Dejan : [50:02] I also watched some Usenix talks, there was this guy, even older than me, he was talking about how he was building this system in India where [the] complete population in India had to register, and it essentially came down to the queue and the workers, believe it or not. So, nothing fancy. I don’t know if you’re aware that queue was invented, if I’m not wrong, if I still remember correctly, by Alan Turing in 1947, I think. So when you think about it, one of the essential components of every scalable system, which is able to, you know, buffer the spikes in traffic and requests coming or probably comments coming would be something data structure, which is over 80 years old.

Jamie : [51:01] That’s mad. I’ve seen i’ve seen reports of apps that are—you see this is the thing where I feel like perhaps this is a conversation for people who are way smarter than I—but like i’ve seen a lot of developers rush towards complexity, and I feel like it’s it’s perhaps a discussion for another day, but I just want to hint at it: i’ve seen people, engineers, drive as quickly as they can towards complexity. “We need to build it with a million different parts, and have all of these these constituent things that are rolling around, and things,” you know. When I remember when docker and kubernetes were, like, really brand new things and people were spinning up, “my website, which is a blog is running on docker and kubernetes, and is scaled infinitely, and i get three readers a day.” And I’m like, “that’s brilliant, you know. You’ve learned how to do that thing. And that is amazing. You’ve got real-life proof that you can do the thing with kubernetes, but also all of that complexity for what is essentially WordPress, right? It’s not really needed.”

Jamie : [52:10] And I’ve read, I think it was two years ago now, I read a story about someone who’d built a really, I can’t remember what the problem was they were solving, but it was a website that got millions of people a day and it was a single spreadsheet on the back end; and it worked and it kept working, and I’m like, “that’s the kind of…” We don’t need to push towards that kind of minimalism but, we need somewhere that’s in between this minimalistic a single spreadsheet on firebase or whatever. It was and scalable microservices-based, I know you said earlier when you don’t need to do microservices to be scalable, but they tend to be… people, like you said, they tend to be talked about in the same breath, right?

Jamie : [52:56] I feel like we need somewhere, there’s a midpoint somewhere between minimalistic, single maybe like a php website with a with a firebase powered spreadsheet as a database, and something that is kubernetes-docker, scaled microservices, you know, a million different buzzwords required, that requires two and a half thousand developers to actually get running, because people only know a fraction of the complexity involved. There needs to be somewhere in the middle of that, where most apps sit, right. We need some of one end of that spectrum and we need some of the other end of the spectrum, but we don’t need to be like all in on either end. I don’t know, i’m just the rantings of a crazy person.

Dejan : [53:43] I absolutely agree with you. I absolutely agree with you. And when you think about what I mentioned is that sometimes, or most of the the times, we are like kids: we like to play. There’s this badge of, “I built [a] complex thing, And I go and brag around it,” for some reason we all do it. I do it as well. And when you think about it also the nature of our profession, we are engineers but not in a physical world; there’s not a single civil engineer that would propose, “let’s build the bridge between London and New York.” But software engineers are building bridges from [the] Moon to Earth because we don’t have physical limitations.

Dejan : [54:25] What we also don’t have, because we are not business owners in most of the cases, we are not paying for that. We don’t have, you could call it, “acomplexity budget.” And there was this website that was proposing, “let’s go with boring technologies.” Saying, “okay, let’s imagine you have 10 points. For every piece of technology you pick up, if it’s mature, old, and boring technology, It costs one point, but if you use something new, let’s say 10 years ago, I’m using NoSQL Database. This is, I don’t know, six or seven points.”

Dejan : [54:59] So we are playing. We are kids who are not paying. Most of the times we are not paying for it, especially the thing we started today’s conversation with: a situation where I land on the project, I make some decisions, I enjoy developing, I don’t know what, microservices, and then I leave. I do not have to live with my choices. I’m not going to be the one who will be awakened by a call at 3 a.m. in the morning because things are not working. So part of the solution hopefully could be not only that we build, but also that we maintain, that we respond to changes and then we can observe consequences of our choices.

Dejan : [55:41] I have an acquaintance who was happily participating in three microservices projects. But when I talked to him, it was, “I sat down, I built it, I launched it in production. And the very same day, I switched to the next project.” So I could assume that he’s not fully aware of all the aspects of running microservices, kubernetes, the way you do the testing. Is this just distributed monolith or not? What will happen if one of the microservices fail? Will the rest of the system continue working properly, etc., etc.?

Jamie : [56:22] It’s been good to have that little bit of a rant, and to have you kind of agree with me there. I have to agree, so thank you for that, Dejan.

Dejan : [56:32] You’re welcome. Now we can go down from our balcony in the Muppet Show to all guys bragging.

Jamie : [56:38] Absolutely. absolutely. Send send us a comment folks: which of us was Waldorf and which was a which of us was Statler? That’s what they were called.

Jamie : [56:52] Excellent. Yeah so, I guess, Dejan what what are some top tips you can give out to people? Maybe resources they can go to learn a little bit more about what we’ve been talking about today? Maybe CQRS and event sourcing? Or like maybe folks want to go and check out some of the content that you create. Because you say you’re in DevRel, so that to me—I may be misunderstanding part of the responsibilities of DevRel, but that means there’s loads of content out there that you’ve created to help people get started, right?

Dejan : [57:26] So when it comes to me specifically, I’m trying to educate people about NoSQL. So it’s not only about the concrete product that I’m maintaining, it’s also about the way of thinking. And, you know, we all start with relational databases, but we learn about data modeling, third normal form, third and a half, fourth. This and now NoSQL represents, not only a type of database, it represents whole movement which is quite pragmatic. So starting with the name NoSQL, “No” is not negation it’s short for “Not only.”

Dejan : [58:07] So what I’m trying to spread is this awareness that you should be pragmatic. You should be using the best tools for your job. And I recently gave a talk entitled “CQRS: Do you really need two databases?” where I’m arguing, sometimes you might, or sometimes if you pick up the right database for your project, one could be enough. Or sometimes you might need three databases. So let’s say you might need relational, you might need key value store for computing and serving view models, and you might add something like Elasticsearch or RavenDB to provide full tech search capabilities. So the whole point is you should be pragmatic, and you should be using the best tools for your project, and you should also not drive on autopilot. You should make conscious decisions.

Dejan : [59:01] Whenever I was leading teams in my career, I was never making decisions and delivering these decisions to my teams. I was always kind of presenting my proposal and defending it with the arguments. So if I receive better arguments from you that we should use something else, I’m all for it.

Jamie : [59:24] Brilliant. So what about resources then? Are there, like, do you have, is it a blog? Is it a YouTube? Where do you want to send people to, to learn a little bit more?

Dejan : [59:38] I have a workshop. So I would like to invite people to the workshop. So I will send you a link, and you can distribute it. So I’m trying to educate people about the whole philosophy of NoSQL databases. I’m also active on Twitter, and I’m now trying to be active on LinkedIn; so this could be people or let’s say Twitter would be the best place for people to exchange resources and to maybe spark some interesting discussions.

Jamie : [1:00:12] Amazing. Amazing. So what I’ll do for the folks who are listening in, I’ll put all of any links that I get sent over or any things that I find, like I’ll go find Dejan on the artist formerly known as Twitter and on LinkedIn; I’ll put those in the show notes. So if you’re interested in getting in touch or finding out about what he’s up to, definitely check those.

Jamie : [1:00:36] Amazing. Well, Dejan, thank you ever so much for our conversation today. I have both appreciated the fact that I could get these couple of rants off of my chest, but also having a a wonderful detailed conversation with you where I have learned a whole bunch more things. Like I said, going into this, you know, I had some experience with the CQRS, and I had fallen into the trap of, you know, being told we need to rewrite everything from the ground up to use things like CQRS; whereas actually you don’t.

Jamie : [1:01:07] And you know, one of the things that we.. I don’t think either of us actually explicitly said, I think maybe you did but I didn’t is: That you’re likely doing CQRS but you don’t realise it. You maybe don’t need to rewrite parts of your app and use specific NuGet packages, you’re probably already doing it anyway; who knows, right? But I really appreciate you being on the show Dejan.

Dejan : [1:01:31] Essentially what happens to me is that that that I frequently discover something new on my own. Then I think to myself, “okay, it’s impossible that I’ve discovered something new.” I go dig, do my research, and then I discover I just discovered something that exists like for 40 years.

Dejan : [1:01:49] So thank you for the opportunity to talk to you today. And it’s also thinking out loud, and a couple of interesting ideas sparked in my head during this conversation. So thank you for that as well.

Jamie : [1:02:00] Hey, no worries. No worries, Dejan. No worries. Thank you ever so much.

Wrapping Up

Thank you for listening to this episode of The Modern .NET Show with me, Jamie Taylor. I’d like to thank this episode’s guest for graciously sharing their time, expertise, and knowledge.

Be sure to check out the show notes for a bunch of links to some of the stuff that we covered, and full transcription of the interview. The show notes, as always, can be found at the podcast's website, and there will be a link directly to them in your podcatcher.

And don’t forget to spread the word, leave a rating or review on your podcatcher of choice—head over to dotnetcore.show/review for ways to do that—reach out via our contact page, or join our discord server at dotnetcore.show/discord—all of which are linked in the show notes.

But above all, I hope you have a fantastic rest of your day, and I hope that I’ll see you again, next time for more .NET goodness.

I will see you again real soon. See you later folks.

Useful Links

• Dejan on Twitter: x.com/dejanmilicic
• Dejjan on LinkedIn: linkedin.com/in/dejanmilicic
• Supporting the show:
  • Leave a rating or review
  • Buy the show a coffee
  • Become a patron
• Getting in touch:
  • via the contact page
  • joining the Discord
• Music created by Mono Memory Music, licensed to RJJ Software for use in The Modern .NET Show
• Editing and post-production services for this episode were provided (in part) by MB Podcast Services