Episode 81 - Gremlinq With Daniel Weber
Embedded Player
The .NET Core Podcast
Episode 81 - Gremlinq With Daniel Weber
Supporting The Show
If this episode was interesting or useful to you, please consider supporting the show with one of the above options.
Episode Transcription
Hello everyone and welcome to THE .NET Core Podcast. A podcast where we reach into the core of the .NET technology stack and, with the help of the .NET community, present you with the information that you need in order to grok the many moving parts of one of the biggest cross-platform, multi-application frameworks on the planet.
I am your host, Jamie “GaProgMan” Taylor. In this episode I talked with Daniel Weber about Gremlinq, a Linq provider for graph databases which support Gremlin APIs. David also gives us a desription of graph databases and where you might use them. We also talk about how trying out newer technologies is getting easier and easier, due to the proliferation and the drop in price of cloud storage.
We also had a short chat about the many different ways to support open source technologies: with contributions, really well researched bug reports, and with direct donations or purchasing licenses. It’s a topic which is for much smarter people than me, and was prompted by a discussion around Deuende and Identity Server 5.
So let’s sit back, open up a terminal, type in dotnet new podcast and let the show begin.
The following is a machine transcription, as such there may be subtle errors. If you would like to help to fix this transcription, please see this GitHub repository
Jamie
Hello, Daniel, thank you very much for being on the show. It’s a pleasure to talk to you today. I’m interested to find out about GermLinq, and how it all fits together and how I can use it in applications. But Welcome to the show.
Daniel
Thanks. Thank you. Hi, Jamie. Thanks for having me. Great to be here. Thank you very much.
Yeah, my name is Daniel. I’m from Germany. I’m a software engineer, mainly .NET. From the very beginning in 2002. Like I’ve seen, kind of seen it all, where version 1, 1.1, 2, generics coming up and all that compact framework. And I’ve been doing it ever since.
I haven’t really done anything, anything besides that, from a language perspective, to be honest, um, yeah. And for like, I think it’s four years now, I’ve been developing this .NET object graph mapper, which is called Gremlinq.
Jamie
Excellent, excellent. Well, I’m looking forward to talking about that in a moment. But one of the things, I’ve just kind of made this connection and the majority of the folks who’ve been on the show so far have been in .NET. Since relatively the beginning.
And I think that’s a good thing. I think that shows that there’s a maturity to the language support and the actual API space. And I think that’s pretty good. Because I mean, if everyone who was coming on the show are like, “yeah, I’ve been doing .NET for five minutes. You know, I’m going to talk about this one thing that I’ve done,” and that would also be fine. But I think just the fact that it’s been around for, whatever, 21/22 years now; 21 years, I guess.
Daniel
Yeah, I think the first previews could be dated back to 2001. I guess it was 20 years. Maybe that’s a celebration coming up at some point? I don’t know.
Jamie
Yeah, well, maybe I’ll be we’ll have to, obviously, I have no affiliation with Microsoft at all. But, you know, maybe there’s something coming up at MSBuild, or Ignite, or something, “hey, 20 years! Woohoo!”, or maybe they’ll just go, “you know, whatever.”
Daniel
It would be cool to have some kind of retrospective on that. by Microsoft. I don’t know, if they want to be associated to the old stuff anymore.
Jamie
I think
I do know that Richard Campbell of .NET Rocks! is working on a series of books chronicling the history of it all. And I believe he goes back to the early 90s with it, because .NET has that much of a lineage right. So maybe that’ll get it? I don’t know, I know that; I’ve chatted with him before about it, both on the podcast and in person at conferences. And he says, “Yeah, it’s coming. It’s just it’s gonna take a long time.”
Daniel
That would be super awesome. I would definitely buy that.
Jamie
Yeah, when I first heard about it, it was going to be a single book. And now it’s going to be in volumes.
Daniel
One book for version 1.0, 1.1.
Jamie
Yeah, apparently he’s making it out of interviews with the people who are behind the whole thing. So presumably, the three Scotts, presumably Anders Hejlsberg, people like that. You know, everyone who’s involved.
But anyway, I’ve digressed way too much. Okay, so I’m interested to find out about Gremlinq, I genuinely know almost nothing about it, so I haven’t used it yet. But I know that it’s its used; so let me get this right. So there’s a thing called TinkerPop, I believe, which is used by the cloud providers who provide that sort of graphQL thing? Is that correct? Am I getting that? Right? I’m not sure.
Daniel
It kind of is. So what tinkerpop is TinkerPop is a - if I got that, right, because I’m not too much in the Apache or TinkerPop community is, um, I’m just a kind of a user as well. So forgive me if I don’t have the actual accurate corporate address on that one.
So, let me try to explain. So, as far as I got it, TinkerPop is a graph computing framework. So, given a graph, which is somewhere on represented on whatever storage or memory or whatever TinkerPop is what can leverage this graph. And the graph I should say is a set of vertices connected by edges, with properties on the vertices and properties on the edges. And what TinkerPop does and TinkerPop code is computes on the graph. This is why it’s called the graph computing framework.
So given a graph, and given some things you would like to know about this graph - this is your query, right? You can leverage this graph and do all kinds of stuff with it and come up with an answer about the graph. So this is what TinkerPop does.
It’s, to the best of my knowledge, not so much concerned with the storage, because the storage is pluggable. You can, at least in theory, theoretically, plug pretty much any kind of storage there - you can plug in an SQL database in there any kind of no SQL databases.
Yeah, and what TinkerPop provides - the Apache TinkerPop project - is a graph query language, because it wouldn’t be nothing without a matching language to query the graph. So once you’ve got your graph, and you want to know something about it, you need a formalism to to express your intent. What you want to know about the graph what you want to query, and this is called Gremlin. It’s a graph traversal language.
It’s a procedural language in contrast to other languages, most notably Cypher, and the difference is, in the declarative approach, you would kind of describe your results sets. And on the other hand, what Gremlin does is you will describe how to get there. Gremlin allows you to take a look at your graph, take a look at all the vertices and describe where to start on which vertices to start, where to go, and where to filter out edges and vertices. What remains in the end is your result set.
So this is, I like this approach better. I think it’s more powerful; though this might be up to any language theorists to determine whether probably they are equally powerful. Because in computer science, as soon as you have kind of Turing complete, everything is equally powerful, right. But I like the approach that Gremlin takes much better. And
yeah, this is this is TinkerPop and Gremlin in a nutshell.
Jamie
Okay. I do find that I, when I’m discussing different languages and frameworks and different ways of doing things,
I, I always say, Well, you know, this particular thing, this technique, this language, whatever, is great at this problem, but there will be problems that he’s not so good at, like, you could, one of the things that I did at university was we studied AI, but we use the language prologue, which sounds like it’s a little bit declarative, you declare a data set. And then you say, you know, you might say, you might declare a family tree, and you will say, if john and john have a child called Jim, and you say, who is Jim’s mother, if john if Jim’s father is getting really confusing with my example. But with you that that’s more traversal of data and figuring out data, you can write a calculator in prologue, but it won’t be as, as efficient as something in almost any other language. Right. And so there are several problems. That’s I think that the work that worked really well with the muscle work really well with graph with graph, Dana struggling to find my words this evening. There must be problems that work really well with graph datasets and graph languages and, and that kind of those kinds of techniques that, that obviously other, like, I don’t want to say SQL, but you know, other kind of languages and techniques and data storage methodology is probably don’t work so well in. So you know, I think having not used any graph stuff myself other than talking to Cosmos dB. I would say that yeah, I think I think perhaps graph graph qL graph languages tinkerpop will obviously have a very specific set of problems that it solves really well. And, and the way I see it, as well as if, if one language or framework sort of mirrors the way that you approach problems, and totally use that don’t, don’t try and shoehorn your own process you’re on problem solving process into another language or framework, because you’ll end up fighting with it and producing something that is a halfway house between how you solve the problem and how the language solves the problem. And so yeah, if graph if graph qL and graph, data storage works better, then do that. That’s what I say.
Daniel
I think it really comes down to your domain, whatever domain you want to model. And whether it lends itself more to a graph structure or more to a, I don’t know, some problems are definitely there’s problems that are completely sufficiently described in a key value store, right? for other things, you might need documents because you have an unbounded complexity of things to store. And there’s query languages for that. So it really comes down to your domain, you should take a good look at it and figure out whether it lends itself to that. And not only that, but also the queries, the things you want to know about your data. Yeah, it’s not. You can’t just throw a gremlin edit, and it’ll solve all your problems. Yeah, just right. I agree. the right job, the right tool for the right job, so to speak. Yeah.
Jamie
Yeah, absolutely. And I think this is where getting experienced with at least knowing that these other technologies exist, right? You don’t have to get your hands dirty, get elbow deep in the actual, how does it all fit together, but knowing that these other ways of dealing with data exist, are kind of there, I’d say that they’re very important, because there’s that phrase, when all you have is a hammer, everything looks like a nail, right? And if you’re only ever doing key value stores, then you just gonna do key value stores. And you’re going to shoehorn something in the build, like you say, it doesn’t fit the domain, or you said there about maybe using documents because you got an unbranded context and all kinds of stuff. So maybe maybe documents is better for, you know, whatever you’re working on, or maybe just normalising all your data and shoving it into SQL tables works better, whatever it is, that fits that domain. And then, but like, you know, like, like you said, that whatever fits that domain, but then, you know, you need to have the knowledge of these other tools and techniques exist only because I remember, early in my career, everything, everything is SQL, everything is normalised data. So it’s got to be, and then you have you have a primary key and a foreign key, and a lookup table, and a join table, and another table to join these things. And you end up with a with an object graph, that is 5000 different tables. And whereas if you wanted to, you could store as one document in a document database, right?
Daniel
Yeah, well, everything had to be SQL, because there weren’t really any alternatives. were they? I don’t I don’t know. Maybe some obscure, but I’m not. I don’t, I don’t know how long no SQL approaches have been around now. I think it’s probably already 20 years. 20 years at least. Yeah, they, they kind of offered a completely new view on these things. And it’s good to have them around. Um, but there’s caveats. SQL isn’t too bad to begin with, because it’s, it’s got all these ACID properties, right? And you got to give them up mostly in any no SQL approach. So this might or might not fit your domain or your requirements on your software eventually. But yeah, that’s definitely a thing to be aware of.
Jamie
Yeah, like you say, there’s, there’s pros and cons to everything, right? If you do need to make sure that your trends that you have transactions and that those transactions are acid, and that they, they will always happen in the order that you send them then Okay, you know, SQL is probably the way to go. But if you are just, you know, loading a lumps of data may be a big blob of JSON, you just want to store it somewhere, then maybe a document or maybe a graph might be better. I don’t know, I’m not a database expert. So you know, I just look at what’s available, and try to figure out what’s what fits like you said, What is the domain
Daniel
we figured that graph is the way to go when we started developing how you project We started actually, we actually we started out using SQL. We started out using SQL Server, then we kind of migrated to Maria DB I think it was. What’s Maria DB is a fork of
Jamie
is it my SQL?
Daniel
Right? Right? Right, right. So we use that first then went to Maria dB, we use Entity Framework to do the mapping, then we use an hibernate to do the mapping, because we Entity Framework kind of fell short. We couldn’t use Entity Framework for the classic Entity Framework, which I don’t know what happened version five or six back then then we tried Entity Framework core, which was really in its infancy right back then, that wouldn’t support any inheritance, we kind of back and back and forth, and nothing would really fit our requirements, because we are very heavy in structure, our domain is very structurally heavy, lots of relationships to between all kinds of domain entities. Sometimes you don’t even don’t even know the type of the entities, you might have an entity and go a certain relationship, and you don’t even know what’s there, or you might be unbounded. And in the relationship, like we have this owner ship concept in our, in our domain was, well something can be owned, an entity can be owned by another entity. And this entity in turn can be owned by another entity. So let’s say you want to get the eventual owner like the the top owner of these things, the owner to settle it all right? Pretty much a we get stuck a lot using just SQL and an hibernate or Entity Framework. Because we had these recursive, potentially unbounded relationships, we couldn’t really model the entities in a way that would allow us to have an entity being owned by let’s say, a user, and a company as well, like a phone. Phone can be owned by us and the company. So are you going to model that in an SQL or in essentially an hibernate or Entity Framework? You got it use inheritance. But this got some caveats as well. So we got really stuck. And then we discovered graph databases. And, of course, the the fallacy here is, as you pointed out a couple of minutes ago, well, if you only got a hammer, everything looks like a nail. And this is also kind of what happened to us. We figured, alright, this is this is the solution to all of our problems, right? This is we got to believe graph databases, and everything’s gonna be fine. Nope, clouds in the sky, everything’s going to be awesome. Which it mostly is. But then there’s the ACID properties that are not really there for let’s say, Cosmos dB. There’s other databases, like Neptune, that suit does support acid. So um, yeah, it’s kind of kind of the same. Back and forth. Let’s see what they they can do. Let’s see what they can do. Let’s try this approach. It’s this approach. It’s like the same we had back then only this time. It’s it’s a graph databases. So um, yeah. But it’s one one thing is still true. graphs are the appropriate model for our data. And I don’t know, I would encourage everybody to, to at least take a look when modelling a Greenfield project. Could this be a graph? In fact, I would argue that tables in a SQL database, form them by themselves a graph as well, right? You could view a table as an entity, whenever you got a foreign key, somewhere, that’s an edge, right? So every every row in your in your table is a vertex, and any foreign key is an edge to another table to another. So this is kind of already a graph. But you might not need graph Technology is graph database. In the end, it all depends. So, but I would encourage anybody who’s free to in their, their choice of technologies to have a look,
Jamie
whether it makes sense I mean, I, I often say to folks, you know, just build on build a hobby project, it doesn’t have to be super complex, you don’t have to, you don’t necessarily have to do with, you know, tests in mind and great architecture in mind. Just get a project, build it in, you know, if you’re, if you’re trying to learn, say, let’s say, graphing databases, right, where I want to build something where I get an idea of how I’m going to use a graph database, well, I can sit here and I can read a book, and I can think about the theory of how it works. Or I can get my hands dirty, do dotnet new, create something that talks to a graph database and go, Oh, that’s how it works. That’s how all of these entities hang together. This is how this specific domain works. You know, totally, though, that I feel like, I feel like we kind of, we have almost no excuse, right? We’ve got all of these tutorials online, we’ve got the documentation for all of the libraries and show this lots of really good documentation regram language i’ll come on to in a minute. And there’s there’s all sorts that you can just, you got to see a watch, go viewer talk in watch it on YouTube, plural site, whatever. There are other training providers, of course. And it’s just, it’s almost, it’s, like I said, you have no excuse now, right? Because everybody, you just go graph qL in dotnet, boom, there you go. Right. Okay. I’ve got 5 million results. Brilliant. And that will get me started. Right. And so yeah, I’m not saying spend hours and hours and hours building something. But maybe two or three hours on a on a rainy weekend, when you can’t go anywhere, build something real quick and go, right. Okay, I am getting how this fits together now, right? Because for a lot of us SQL databases, like I said, I really my in my early in my career, SQL was the way to go. And the only way that we learned that was we built our SQL database, right. And I feel like perhaps sometimes developers get a bit complacent with their technology, or I’ll use dotnet and jQuery or dotnet, and react and a SQL database. And as the application dawn, I can move on, Well, look, we’ve already discussed, right, you’ve said there in your, your particular problem domain, there’s a, it doesn’t fit that nicely with standard SQL tables. And graph databases just seems to fit a little better. Unless someone unless you or someone on your team had spent the time to look into graph databases. You’ll never have known that right?
Daniel
Yeah, it caught us by surprise, a bit. I think we were late to the party and graph database had already been around for a couple of years when we discovered them. So we were kind of living under a rock, I will happily admit that we only got aware of the graph database, I think in 2015, or 16. But to my defence, I should say that cloud providers offering a gremlin API or Graph API whatsoever, have only been around not a lot longer. I think the the cosmos DB offering has been introduced in 2017. I think Neptune is around kind of like, for the same time. And there’s others. But yeah, the the theory has been around I think, like 2012, for the tinkerpop project, I think I think the protests are 2009. I don’t mean. But it’s only gaining momentum, I think with the availability of cloud database providers, because I’m sure you might want to, you might choose to set them up themselves and deploy those servers yourself. There’s offerings for that as well. But I think the cloud providers being available as a really big factor in growth in that regard, when it comes to graph databases and their acceptance.
Jamie
Now, absolutely, if if I as as a developer, right, I put my developer hat on not my podcasts, I have them as a developer, if I’ve got to go and instal loads of software on a server, and set it all up and you know, provide passwords and all that kind of stuff, just to be able to connect my web app to it for storing data. And it’s going to take two or three hours. Or if I can go to Azure or AWS or Google Compute or link Whoever push three buttons, and boom, there it is, we’re going to choose, I’m going to choose the easy path. And of course, like you say, because because I know of Cosmos dB, and I remember going to see a. So this is all I know about graph graph. databases, right? I remember going to an MS Ignite. And it was a number of years ago, two, three years ago, there were they showed off, hey, we’ve got this view of a graph database that has every single character in the Marvel Comics universe in it. And we can just sort of move around it without having to, you know, without having to do lots of long running queries. And I could find this character appeared in this comic with this other character. And they appeared in this series and all this kind of stuff. And I’m like, Wow, that’s awesome. That must be a huge, and they were saying it was a huge amount of data. And I’m like, brilliant. And that’s it. That’s all I know about it. That’s where my experience ends. But that was, but they were showing off, hey, you can do this in cosmos. Right? And I’m not sure that there, I could do that with an on prem database server that I have to set up myself. And so I agree completely that it is it’s the cloud provided databases, because there’s, there is a limit to it, but technically is unlimited compute, right? And unlimited space, as long as you’re willing to pay. So and I don’t have to set it up. So let’s do it that way.
Daniel
Definitely. Yeah, I didn’t see that particular example. Sounds very, very interesting. Sounds pretty awesome. I should say that Gremlin on Cosmos DB is just one of the API’s they offer. Because at the core, Cosmos DB is a document database. You can store arbitrary documents, Jason have documents. They actually Jason but formatted as Jason when you query them. And so at the, at the core, it’s a it’s a document database. And there’s various API’s, there’s a kind of stripped down SQL dialect, to query your data in Cosmos dB. Think they got a Mongo DB API, they got a Cassandra API. And eventually, at the at the end, everybody’s going to be modelled as documents, whatever you throw at it, whether it’s through the MongoDB API, or the Cassandra API, whatever, eventually, they’re gonna model what you throw at it as documents, which is pretty awesome. Right? So it’s, it’s, it follows the multi model approach. And Gremlin is one of the API’s that you can use to query Cosmos dB. And this is in my in my book, also pretty easy, which is your vertices and your edges in the end are just plain documents in the coastal city, in the documents store. And usually, where you would have isolated documents sitting around in the in the store, once you kind of view
those documents as vertices and edges, and interpret the data on those vertices and edges as links to other vertices and edges. Kind of all makes sense, comes together. So yeah, I think this might be theoretical, but I think with a document database, at the core, you can model pretty much any data model you like. They can do key value as well, they can do rows. So yeah, this is this is really awesome. But it’s not a it’s not just a graph database, Cosmos dB. I should say. I should look up that example with the with the Marvel characters.
Jamie
totally sure. If I can find out put it in the show notes for anyone listening, but yeah, it was. It seemed like a really, because I’m not a comic. I’m not a big comic book fan. Right. So you know. Yeah, right. And I, you know, I’ve seen some of the movies, I get that it’s a big universe. But like just thinking about like this, that there’s like 5060 years of shared stories between all of these characters. And they’re always be crossing over and appearing in different. So you know, you said earlier when you have that thing of the ownership, right? Well imagine a story that contains a number of characters. But that exists in perhaps, I don’t know if that’s the term that they use but a universe right when they reboot that universe to retell the same stories, but with a slightly different character that is, otherwise the same stories will they similar enough, it’s the same character but a different version of the character. So like, just trying to map some of that, in your mind, you start to realise, yeah, this wouldn’t work with, with tables, right? Because how are you going to manage all of that? So I totally, totally see why they did that. Yeah, I do know that. I’ve used the the cosmos DB API using, you know, Entity Framework. And SQL they like so as I know that the cosmos DB can do you throw some data at it, and it does some magic and stores it and then you throw a query at it, it does some magic and you get your data back. It’s brilliant. I have to say, Okay, so we’ve talked about, we talked a little bit about graph databases, we talked a little bit about Gremlin, let’s talk about grammar link. Right? So top of my head, I’m thinking grim link just from the name. Hopefully, you’ve picked a name that would do this, right? uses Gremlin, and link. Is that right? Or are we completely completely out? miss the point whatsoever?
Daniel
No, that’s, that’s the point. That’s kind of the intention guide that too. In the if you’re being pedantic, you could say there’s not really link. Because for link, which is language integrated query, you should be able to write this special syntax in C sharp, where you go, like, I don’t know, from this from x in whatsoever, whatever. And from this, in this, select this and that. This is this is not what Grambling provides. But the suffix link is mainly due to the fact that you can write on the operators in the link section where you could also write select and brackets and dot where and dot select again. And that’s like many. because it enables you to write queries, just that that look like link queries, like your link to SQL queries or whatever. Look, they look like it in a fluent fashion. But yeah, the link part is a bit of a stretch, but it works. It works. And this is what you do in the end right here. You’d have you have some, some queries, you get them in your language, which is, which is C sharp. So it’s kind of language integrated query. I hope Microsoft doesn’t mind that I call it link. Web Link, that falls, it falls together pretty nicely. It comes together less Gremlin and there’s links, those are pretty obvious is that.
Jamie
And to be fair, I think that makes sense to do a lot of people when they think link, they’re thinking extension methods with lambdas embedded in them. That’s, like you say, they’re thinking collection dot where x goes to x.id equals seven, right? That’s what they think when they think link, they don’t think, from collection, where ID equals right. They’re not they’re not thinking because like the, the, the the link syntax is very SQL like, isn’t it? Like the actual when you I mean, it’s terrible. We’re talking, we’re talking on an audio podcast, and I’m speaking syntax, but you know, when when you have like, I have a collection of intz. And I do from int collection, select x equals three, you know, that’s, that’s the link. But, but for a lot of people is collection dot where x equals you know, and I totally get that I totally get why you’ve chosen that because like you say, A, it uses that fluent like syntax with the extension methods that take in the predicates and the functions and things and be it just rolls off the tongue really nicely. Yeah. Yeah, naming things is hard. I think you want. Excellent. So So from the from the name of it, then I’m thinking grim link allows me to connect to some Gremlin API, and use the link syntax to actually get my collections and filter them and sort them and select and projects and things like that. Is that is that what’s going on?
Daniel
This is pretty much what’s going on. So as with any query language, or any language that is a different language as your house language, be it SQL, you could already you could also write Gremlin you could also write raw Gremlin, right? that’s a that’s a library that Grambling relies upon as well. Gremlin dotnet Which allows you to do just that. Right? Your raw Gremlin code, you could even write and fluently but you would still have all these strings, right? You would still have to write dot has label and then a string for your vertex label. And, but you could also write a raw Gremlin script and just execute that on the server. But it’s, it’s just a string, right? And for Cosmos dB, use, you’re still stuck with that you you can’t do at least when you’re using Gremlin dotnet, you can still do better on for Cosmos dB, then write a raw string script, which raw Gremlin inside. So after all, it’s just a string, you can’t really it doesn’t it doesn’t refactor. magically, when you refactor other code, right? It doesn’t, it doesn’t change names when you hit f2, and change your class names and so on. So maintaining a string and coming up with a string and maintaining it and composing strings, composed composing string queries, is not something you would like to do in this, this might be a bold statement, because there might be reasons to do that. There’s definitely reasons to write raw SQL. If it’s handcrafted, and it’s optimised and optimised, and it’s kind of stable. It’s definitely a good approach. Especially if you got people who are really good at what they’re doing in SQL, and they really know their craft, that is perfectly fine. But for a rapid development approach, or when you’re in development, and things are changing rapidly. This might not be what you want to do. You might not, especially when you’re new to Gremlin databases, you might not want to write all these strings, and we write them and then there’s a missing bracket. But it doesn’t show you the compiler can’t help you with that, right? You don’t at least I did not want to write raw Gremlin strings. So I should say to the defence of revenue dotnet, which is the awesome library that is underneath Grambling it, it does allow you to have a kind of fluid syntax. As, as I said, there’s still the native elements of a graph databases, labels, property names, which are still raw strings. What Grambling now does is, not only does it provide you with a fluent syntax, to to, to come up with with a query, but it would like a good object relational mapper, like Entity Framework and stuff. It allows you to reason about the graph in your, in your language C sharp, using your pocos your domain objects, you’re playing out see objects, right. So instead of writing something like V, capital V, dot has label, let’s say person as a string, you can go and put the person which is now your, your poco your class into into square brackets and be strongly typed. This is this is the magic that remining does. So it allows you to reason about the objects that are on the on the graph server or in the graph database. But using your domain objects in C sharp to reason about them and to express queries. So
then it did allow you what you just said, it’ll allow you to write dot where x, right arrow, x dot name equals Jamie, for example, it allows you to do that, formulate this and you get complete compiler support here, right? If there’s no property called name on your personal vertex, it’s going to start to compile. And what Grambling does is it translate your query which is now a strongly typed checkable query in C sharp it compiles down to valid Gremlin which is executable on the server. It can compile it to a remnant script. We Just let you the plain text string. But it can also compile it to bytecode, which is then handled again by Gremlin dotnet, the underlying library to execute it on the server to query the server to return results to Grambling, which does, then all the heavy lifting required to map it again to your domain objects. So let’s say you’re querying a range of persons in your database, you get some raw data back by Gremlin dotnet, you get some raw data back in. And graphs on this is the JSON dialect that comes back. Graham link will now do the mapping that’s required to map to get the name property from the raw data to get all the other properties from the raw data and map them to your pocos and return those pocos to you in the end. So yeah, I think I, when I started out doing this, we already had a lot of code in Entity Framework already present. So we had all those pokers around. And we figured, Alright, we’re not gonna throw these away, we’re not gonna throw the logic away. And we’re not going to throw these entities, entity classes away. What are we going to do is that something like Entity Framework for the graph world, there wasn’t. So we figured we might as well just write it, right?
Jamie
I love it. I love when Necessity is the mother of all invention, right? You’re saying, hey, we’ve got this graph database. We don’t want to use magic strings. Because although you can make a lot of money in magic strings, because then you’re the only person who can fix them when they break. You know, like you said, we’ve got all that we’ve got this entire domain mapped out and C sharp code. How do we marry that up? How do we join that up to the graph database? Or? Well, there isn’t one. So let’s break our own. I absolutely love that. I love to hear stories like that, where it’s like, Hey, we needed this thing. And they didn’t exist. But now it exists is brilliant.
Daniel
It does. And some I get some feedback sometimes on GitHub. And it seems that because I want to, I want to bring this this topic forward, right? I want to get graph databases. I want to do what I can I want to help getting them getting accepted and considered more. So I like it when people say thank you for the library. Sorry about the Thank you, but about the I wouldn’t have used a graph databases had it not been for Grambling, which is kind of you got to be a dotnet developer to use Grambling. Right. So this is only this only covers a small or maybe not small, but just a percentage of all people using graph databases. So I’m not speaking for all developers use a graph database and everybody as much as be fine with that tooling. But yeah, I like it when people say, I would not have considered I’m a dotnet developer and I would not have considered using a graph database has not had it not been for Grambling. This is this is not a nice thing to hear.
Jamie
That, yeah, I can imagine this. One of these days, I’ll make something that everybody uses. feel the same?
Daniel
Yeah, that would put a big burden on me, if we kind of everybody would start using that I would have to quit my job and rely completely on GitHub sponsors.
Jamie
I think I think that’s a potential downfall in the in the open source community at the moment. Nobody really is. I mean, open source has been around for what? Since the 60s technically, and, you know, nobody’s really fully come up with a nice way to monetize a human being for support, or whatever. But yeah, cuz like, there’s this story of, you know, the the guys who the folks who make identity server, or the next version of is going to be non free. So that’s the only reason that because they would they were saying the same thing. We don’t get any money from doing this, and everybody uses it. How are we going to make the next version?
Daniel
I think that puts I didn’t get that. I think it was Jimmy Bogart earlier today who posted a comment on GitHub. You might have seen that on Twitter. I I didn’t get the context from that. There was kind of a user. Yeah, I guess he expressed this dislike with something that went up. I don’t know. To be bothered, kind of commented on that. Yeah, that might have been related to to that. So I didn’t know that. So they they’re gonna they’re gonna go commercial. Alright, didn’t know that.
Jamie
Yeah, it’s I believe they’re called do Wendy world do is for your benefit. David, I’ll put it into the chat system that we use for that there. And I’ll put the link into the into the show notes. But yeah, they’re hidden the same problem. They’re like, Hey, we make you this is this is now our thing that we do and everybody uses it, and we’ve got no support. So instead of judging people, so and that, that makes perfect sense to me, right? If like, if you think about everyone who’s used identity server for all of their authorization stuff, and they’ve used it off free and never paid any money to help develop, it makes perfect sense. And you’d end up in a situation similar if everyone in the dotnet world started using graph QL. And then sorry, graph databases and went right, well use grim link. That would be great, but also terrifying.
Daniel
Yes, definitely. Not so sure. So I guess, I’m not too familiar with entity, identity server, but it’s kind of a standalone project. Right? So that makes sense to me to have a standalone application. Go commercial and being paid? I don’t, I don’t see that for libraries, actually, how are you going to go and charge for something that is a new get? Or are we going to go back to I don’t know, put to putting some licencing code in there and check, maybe check the system clock and kind of stop working after after your licence runs up? I don’t know, this is gonna, I don’t know, this is gonna work for libraries. So I don’t know, do you?
Jamie
Like I say open source is hard, like figuring out how to fund open sources I’ve heard of, there’s a licence called the A GPL? I think it is, yeah. Where you can say, Hey, you know, you can use it for free. But I want you to pay me, you know, so much money for every production version or whatever, which would make sense, right? But like I say, open funding open source is out. But yes, I feel we’ve gotten off the topic of grim like a little bit.
Daniel
It’s definitely a hot topic. I don’t think I could charge people to use it, because people would just I don’t know, it’s licenced. gramling is licenced under MIT terms? Some people would just probably just fork it. And yeah. leave it at that. Probably? If no, if there’s nobody else to maintain it and develop it further. But they would, I guess people would always use the free version. That’s around. I don’t think I could, because I don’t want to go for the hassle. And I don’t know, did any licencing system into it to make people stop using it? So it’ll probably always be free to use. I wouldn’t know how to know, I wouldn’t know how to make it non free to use or I wouldn’t, I wouldn’t know how to make it stop working as you’re not being paid for right. But, um, yeah, I could do some I can offer consulting on that, which I have done a couple of times in the past. So these are services that I can definitely offer some amount of money.
Jamie
And I think that’s, that’s the the true support and consulting, I think is the, the agile way to, in my opinion, the agile way to to make money as an open source creator, made maybe not the best way to support an open source creator. But I think one of the ways that is the most accepted because I do I love using open source software, right, we’re gonna go on a bit of a tangent towards open source software, but I love using open source software because if there’s something that I want to change, I could just literally you know, get clone, change it, build it and away we go. But I also I sometimes dislike using open source software because I know that someone somewhere is sitting in their house at four o’clock in the morning, trying to figure out why isn’t working. And then they get up at eight o’clock or whatever and go to work. Right? And they come home and they work through the evening on this as passion project. Which is great. But also it means that you know, this person is is not sleeping. That stuff, we can’t do that to people. And so you know that, I often worry, I have to wonder about how I can give back and maybe, maybe submitting a patch is the way to get back, maybe just shout it again, hey, that project is amazing. Thanks for making it, I’m having a lot of fun it is for much, much smarter people than I to figure out I am not a smart person, we’ll let the smart people figure this out.
Daniel
Yet contributing back is definitely a way to go. Whenever I contribute to some other project, I’m usually make sure that I put in some effort. So this is my way of giving back not not just even if it’s a bug report, I try to make it as concise as possibly as possible. Because I consider this a contribution as well, right? If you do the research on your part and really examine the buck, or whatever it is, or the unexpected behaviour. doing proper research is already contributing. I assume so. And yeah, I specifically asked this from people who raise issues on my project to give me something I can run easily quickly. So but I can see there is there is some effort being taken on the other, the other side, and to save my time, right a bit. But it’s it’s an interesting, it’s definitely an interesting thing to see how people interact on GitHub, as people who would put more work into an issue and less people put less work into an issue. Yeah, definitely. But yeah, the issue about being paid or getting paid. I thought about this is some just something that came to my mind. Some point, maybe we should we should offer things to be fixed of features as a Kickstarter project like micro Kickstarter. Well, you don’t have those, the view on Kickstarter projects, which are big and require millions of funds to be to be collected. But you should you could go, I implement this feature for, let’s say, $2,000. Now everybody can just throw money at it once once you reach the goal, you go ahead and implement it. That will be some interesting thing to
know. Yeah, that makes sense. Because then that’s covering that developer or designer or whoever is covering their time. Right? That makes perfect sense to me, because people are asking for documentation. And you mentioned documentation early on you. You assume that there is extensive extensive documentation on Grambling spoiler that there isn’t. This is one of the questions I get all the time, like, are you planning to provide some documentation? And sure I am kind of, but I would like to be able to go ahead and say, Look, this is my Kickstarter campaign, I’m going to write some extensive documentation. During my free time, during the night shifts, I’m not going to sleep at all for six weeks for X amount of money, right? Because documentation is at least for me, the not so fun part about doing this, and as a non native speaker coming up with stuff that look reformed texts, that is kind of I’m struggling right now, you should try to find
text, rather rather formal and more documentary style. It’s rather hard for me. We I can banter back and forth, and I’m struggling all the time. But writing documentation that doesn’t look really bad is hard for me. So yeah, this is the not so fun part about that. show people could could sponsor me. I don’t know, fund me to write my documentation, which is actually don’t because if you would fund me, I would have to write documentation. I would rather rather give it to somebody else. But
Jamie
yeah, I think you’re right. I think you’ve come up with a great idea for the future of open source because You know, if if, let’s say some new feature, right, and there’s, you put up a poll and 400 people say, yes, we’d like this feature. And he said to them, I’ll build the feature if every single person says me $5, because that will cover the time that it will take for me to build that feature. It will cover the time that I would have to take out of work out of life. To build that feature. I think that I feel like that would make more sense.
Daniel
Definitely. Yeah, I think there is a platform for that there is something like bounty and GitHub, let
Jamie
me just Alright, yeah. So you can say, hey, yeah, I like that. And it makes sense, because I think that’s what the DND folks are doing. Because it’s like, Hey, I think they test something like that, they’d only received 1000, or $10,000, in direct financial support in the 10 years that they’ve been doing it or something like that. And that, to me, personal opinion, is kind of criminal, right. With, with, with, with companies making billions of dollars, essentially, using this free software. I think it’s unfair of them to have not contributed back in some way. But anyway, we’ve digressed way too much. Let’s say I want to I want to get using graph a graph database. And I want to use Gremlin is the the equivalent of a Hello World application that you can think of off the top of your head that would make use of a graph database, or is it just a just get some entities, hang them together, throw them into a graph database and use Grambling to get your head around it.
Daniel
Yeah, the the Hello World equivalent would, I don’t know, if you’re starting off with a with an empty database, you’d probably just add some some words that this is into the database to begin with. And yet, your easiest HelloWorld application would be just I don’t know, writing advi, putting thrown some pocos, edit, write your personal vertex or whatever. Fill up the person object with some properties like name and birth, date of birth, and so on. And in the end, you just write to array a sink, or after the bare minimum, first a sink, just like like link, right? It’s just like a basic async. Link and execute it and get something back. So this would be the there is a sample project on GitHub, which is just the main repo with the postfix dot samples. And there’s some, some code to start with. The second there’s a console application, and an ASP application that you can run right off your ID just enter some credentials in the app settings that Jason, just enter where your graph database or your Cosmos DB is, or your Gremlin server, whatever. And the password, the auth key, and you’re set to go. So what the exempt what the samples project does is create some person that is in the database, put some connection between them, like the nose edge, right? Like Jamie knows Daniel. And there are some pets around there, the cats, cats and dogs. This is just to show off the inheritance support in Grambling because they inherit from a common animal vertex. So you can query it as a pet, I don’t know. So you can query pet and you get all the cats and all the dogs. This is a bit to show off the inheritance support. Yeah, this this would be a good starting point. There is no really not really a Hello word in this domain, I guess. Okay.
Jamie
Okay. I mean, as long as you can show that there are there are relationships between entities. You can map it in a graph database, right? As long as you with those with those relationships being those vertices, you can, you can map anything, right. So you can say, I have a bookcase and I have 12 books, three of them are by this author, two of them are by that author. Maybe in those three books that I have the common author, there are 15 common characters, two common situations, and they’re all in the same genre of story. Right? You could probably map that, I guess.
Daniel
Definitely, definitely, definitely. And as I said earlier, I think already when you model your traditional SQL database You’re already you most of the hour, I would argue a lot of the time, you’ve got a graph in mind. So the, the ultimate test that I would ask everybody to do is when modelling your domain, somehow, considering a graph database, or considering a traditional SQL database or whatever, explain your domain to somebody without any prior knowledge to your non technical friends, or whoever, and take a piece of paper and a pen and try to explain it to them. You might at some point, start drawing circles, and putting names into the circles and connecting those circles by by edges and unknowingly, maybe unwillingly. But if I don’t know I have discovered this quite a lot. That if you’re going to explain something to somebody, you draw, you draw edges a lot of the times, right. And if you draw enough edges, and enough circuits, enough arrows and enough circles, then it’s a good indicator that you might have a graph here. As a general test, as a general indicator.
Jamie
Sure, so take whatever context you’re working in, draw it as a diagram. And instead of using, say, squares, to represent an entity, who does a circle, draw the lines between them end up with some kind of spider ish graph of all these nodes and circles? Give everything names, and then guess what you’ve just gratified. So even what you’ve just turned your model, your domain into a graph, go make it a graph.
Daniel
Of course, you would have to think about your queries, but you want to know from the graph, because if all your queries are kind of trivial, trivial. You don’t really need a powerful graph. traversal language, but if you want to know, I don’t know, it’s just the example you came up with a minute ago about the the books and the authors like this, I don’t know given given a book. What other books has this author written? It’s, this is already a trivia question. But it’s really nicely expressed in a graph language, right? And the graph traversal language, I wouldn’t have to make any joints, arbitrary tables, I wouldn’t really know how to make any I wouldn’t really even know any more how to write sequel.
I wouldn’t know what’s what’s required to make make this like we have a book table and an author table and given a book. And now you’re going to join this with the author and join it back with the books. This is already two joints, right? I wouldn’t, I wouldn’t know how to do this anymore, I will freely admit that. And in a graph query language, it’s easy. You just traverse from the book to the author by whatever edge connects them, I don’t know. You might, you might call the edge has written because I like to name. I like to give vertices a label that is a noun. And I like to give edges a label that is a verb. So you can actually tell stories, author has written a book. So you would just go back this has written edge, and then traverse back all the has written edges out and get out of the books. And this is pretty easy, easily done in a graph, query language. But there’s this is still a very easy use case. There is there is harder problems than that.
Jamie
Should I guess I feel like it was an unfair question. Really? Give me the killer app for graph databases? Well, there isn’t one. It’s what it like we said earlier was whatever your domain fits into, right? If your domain fits into a graph database, give it a try with a graph database. If your domain fits into a SQL database with tables and joins and foreign keys, then go to that way. You know, I’m a big, very, very big proponent of whatever fits do it the way that you use the technology that fits. But with that being said, Daniel just real quick. With that being said then where, where can folks go to find out more about Grimm link and to find out more about yourself because you said earlier on You know, your everything’s on GitHub? Is that the way that you that you let people know about new releases and things like that? Is it GitHub? Is it Twitter? What’s the? what’s the what’s the way?
Daniel
Sure you can follow me on Twitter. It’s at Daniel C. Viva business at Daniel, c, w e, b, e. r. Follow me on GitHub doesn’t really matter. You can follow me on GitHub, but I have never quite figured out but this is good. on GitHub, the x just Google X from Grambling. This is E x r a m dot gramme link, the addresses github.com slash x from slash x dot REM link. This is the go to page and there is this prefixed version with the samples. And this is the separate, but it’s also there’s also a link on the on the main page. There is, unfortunately no dedicated website for Grambling yet. And I wouldn’t really know whether it makes a lot of sense, maybe for documentation, if there is ever any to come up. Yeah, there will be documentation in the end. But I don’t know. There’s those samples, like, I guess people can go from there, right? Just clone the repo and have a look. Press the dot button on your keyboard, make the ad come up with all the suggestions, have a look at the signatures, look, have a look of what’s going inside those methods, right, see how it fits together, hover over the the methods to see what it’s returning what the signatures and you can go from there. But I agree, of course, that proper documentation is
just a professional way to go. shopping in the end. But we’re getting there.
Jamie
I mean, maybe people can submit professional documentation as a pull request.
Daniel
Definitely they can. I would like everybody to figure out kremmling as as thoroughly as possible, and to be able to come up with documentation. But
Jamie
yeah,
Daniel
just like this open source, it’s sometimes thankless profession.
Jamie
It is, unfortunately, sometimes, okay, what I’ll do is I’ll put all of those links into the show notes. If you’re listening along and want to check it out, which I suggest you do. Anyway, pressure to the show notes, they’ll be there’ll be a section just with links on there. Definitely go check that out. or indeed, you know, go find Daniel on Twitter. And I’m sure you’ve got a link on your on your Twitter. If not, then they can find it from there. That’s wonderful. Excellent.
All it really remains to say, Daniel is thank you so much for being on the show. I’ve, I now feel like I’ve learnt. So a huge amount about graph databases. And I think what I’m going to do is I’m going to this weekend, I’m going to take an application of mine, draw it out on paper, or the the basically hand drawn an entity, relational diagram, just to show how everything’s linked together and see if I can convert it into a graph database. Why not write the graph database and see what happens?
Daniel
Yeah, good. Well, thanks for having me. Thanks for inviting me on the show. It’s been a pleasure. Thank you very much.
Jamie
No worries. Thank you so much.
The above is a machine transcription, as such there may be subtle errors. If you would like to help to fix this transcription, please see this GitHub repository
Wrapping Up
That was my interview with David Weber about Gremlinq. Be sure to check out the show notes for a bunch of links to some of the stuff that we covered, and full transcription of the interview. The show notes, as always, can be found at dotnetcore.show, and there will be a link directly to them in your podcatcher.
And don’t forget to spread the word, leave a rating or review on your podcatcher of choice - head over to dotnetcore.show/subscribe for ways to do that - and to come back next time for more .NET Core goodness.
I will see you again real soon. See you later folks.