Macrocosm of Microapis

Video size:

0:09 Miko Pawlikowski

Hello and welcome to Conf42Cast, another exciting episode from the podcast from the other planet. My name is Miko Pawlikowski and today I have a guest from the Microapis.io. He's the author of "Microservice APIs with Python" by Manning, Jose Haro Peralta. Jose, it's great to have you here today.

0:32 Jose Haro Peralta

Hello, it's really great to be here.

0:35 Miko Pawlikowski

Thanks for finding the time to do this. So, we typically just confuse our guests as the first way to kind of get them into their zone. Your question for today is, what's your least favorite planet?

0:45 Jose Haro Peralta

Right. I guess I don't really have a least the earth planet. I guess, if I was studying astronomy or something like that, I would maybe have one. I don't have any reason to like one planet more than any other. If anything, I would say I appreciate perhaps Jupiter to be where it is, preventing asteroids coming to Earth. That's what I would say.

1:08 Miko Pawlikowski

Got that. Okay, equal opportunity employer. That's pretty cool.

1:14 Jose Haro Peralta

Exactly.

1:15 Miko Pawlikowski

Alright. So, I'm hoping to talk about the book that you wrote and the Microapis.io, but one of the things that I notice, and you know, if anyone has looked at any of your presentation, is that you tend to talk about APIs a lot. So, maybe you can get started by telling me a little bit about where that love for API's is coming from?

1:39 Jose Haro Peralta

Right. That's an interesting question. I guess it comes from the fact that one of the first things that I started to do as I got into IT was build APIs. And before working with APIs, I worked for a while with traditional websites, with dynamic content generated in the backend. So with frameworks like Django and stuff like that. Then I discovered this world of APIs and I got introduced to the REST APIs. And the great thing about REST is it gives you a lot of guidelines about how to structure your resources and how to model your endpoints. And there's a lot of technical guidelines about how you form the responses and how you create meaningful communication between the client and the server. And I found that very useful, because one of the things you struggled the most when you're starting to do software is how you structure your code, how you design your application, what goes where, and things like that, I find it useful to follow the guidelines of REST APIs to start thinking about how I structure my projects. And everything suddenly was looking a little bit cleaner and more organized. I guess that's how I started to like them so much.

2:44 Miko Pawlikowski

That makes sense. You mentioned REST API. What makes an API restful as opposed to, you know, a lot of people will just look at an API, they'll see some JSON schema and different HTTP methods. What's the defining characteristic that makes an API restful in your experience?

3:07 Jose Haro Peralta

I would say it's primarily following the resource-oriented principle of rest. So you have resources organized, you have resources in your server, you're managing resources, it is up to you what is a resource, you'll define it. But once you've defined the resources, you create endpoints around those resources. You have an endpoint to these resources and, within the same URL path, you have an endpoint that allows you to add a resource to the list. You don't have an endpoint to, if it is a website for furniture let's say, you don't have an endpoint, which is least resources, and another URL path, which is create furniture, or something like that. There is a structure to your URLs. And it has to make sense, in a way, it has to be predictable. This is within the principles of rest design. If you follow those principles, everything should be kind of predictable. And I think that's the essence of a REST API.

3:57 Miko Pawlikowski

That speaks a lot to humans. I'm wondering, now with the kind of other options that you have when you go and set out to design an API, you can go, obviously, the open API route, and you can design your JSON based schema, you can go GRPC and define your models in code first. Or you can go something like GraphQL. And there's probably a lot of other options that I should mention here too. So, do you focus particularly on the REST API for any particular reason? Or do you see this as kind of separate things that coexist in the similar space?

4:41 Jose Haro Peralta

Yeah, I see them as, in terms of open API, open API/REST vs GraphQL vs GRPC. I see them as different options within the same space. You use one of them to build your API. And my first focus on REST was just the first type of API I got to work with. I think it's also the mainstream type of API that we use nowadays. For many people, when you talk about APIs, is kind of synonymous with REST. People don't immediately realize that GraphQL is also just another type of API. And yeah, I guess that's the reason.

5:16 Miko Pawlikowski

Okay. You've kind of touched upon the fact that you like that the different resources were logically separated and it was making sense from a human perspective of understanding how to use an API. But is that really the best prediction of what makes a good API? What would be the criteria of the work that you do with your clients? If you can talk about that. Or maybe some open source APIs that you like? What makes for a good API in general?

5:50 Jose Haro Peralta

It's more than that actually. I would say, the most defining feature of a good API is it complies with the technicalities of the chosen protocol. So if it is REST, you follow the designed principles of REST. If you follow GraphQL that's the same principles of REST. And the same for GRPC. I think something that is fundamental to create a good API is to have a designed stage before you're going to implement it. I'm a big proponent of the documentation-driven design approach to APIs were designed first or API first, you can call it under different names. Just the basic idea is to design the API first, and then you're going to build it. And I think it makes a crucial difference, because there's also this approach of CodeFest, right? You use one of these frameworks that allow you to build the API first, and then you generate the documentation from it and stuff like that. The problem with this approach is, without having a design first stage, that's how you end up usually with these random endpoints. And it doesn't make a lot of sense. And this happens all the time, when we are not thinking about what we are creating, we just add one more endpoint and another endpoint. And there is no logic relationship between them. It's just one after the other. And it also makes things more difficult to develop the client and the backend side of the API. Because, if we don't have the documentation first for the backend developer service, fine, you're creating the API. But for the frontend developer, for the consuming side, or for another microservice, which is consuming the API, it's difficult to build integration, because you don't know what you're building against. And very often, in the workflow, there is sometimes this this approach of, "Okay, instead of documentation, we're going to use JSON examples. I'm just going to give you a sample payload of what the API is going to look like. And you just deal with that". It kind of works, but not really, because that JSON example is just a limited representation of what the API is going to be. It's not giving you the types of the fields. It's not telling you, if I field this option or not. And it's not telling you also exactly how it is coming. Because sometimes we are returning a raise, right? A list of items from an endpoint. But do we have a key which is pointing to the array or is it just the array? How is that coming from the internet? At the end of the day, it's all about this small details. The small details may grow or do or undo the API integration. So, by having the design and the specification first, development is so much easier afterwards. And we can validate that the server and the backing are talking the same language. And then, the second most important aspect is that, you're following the design principles of each protocol. So with raise, like I said, it is resource-oriented, your URL pattern, your endpoints have to represent that, raise also leverages HTTP. You need to be using the correct status codes for your responses and stuff like that. And in GraphQL, you're leveraging nodes and connections between different objects in your API. People want to be able to traverse your graph of entities and the connections between them. That makes for a good GraphQL API.

8:49 Miko Pawlikowski

Sure. I think it's hard to disagree that knowing what you want to build and building it at the beginning in a way that makes sense is great. But there's also the aspect of life that just keeps getting in the way. And then as the software evolves over the years, the requirements change, and you end up finding things that you would probably have decided differently if you knew in advance of all the things that were going to happen. Do you have any, from the experience you've gathered so far, do you have any pointers or advice of how to make it as forward-compatible as possible in general?

9:29 Jose Haro Peralta

Yes. I'm a big proponent of documentation-driven development, but at the same time, I think it's more important to be practical. In reality, it's very unlikely that, when you start working with an API, you know absolutely everything of what you should be doing. So when I say this should be design stage, doesn't mean that you should just be writing a YAML file or something like that. It means to do some experimentation stages well and you're gonna write some code that is already creating some inputs, you may generate some documentation from that. On the client side, you need to be trialing as well, what do you need from the API. So you need to be building something with expectations of what the API should be doing. And out of these experimentations, then you will put something together and you agree on something, and it doesn't need to be at the beginning. It doesn't need to be a formalized specification. Be pragmatic, you're discovering your features, what you need. I think the most common approach is probably: you build something fast, that just works. It's simple at the beginning, so it just works. You're not having many duration issues. And then you consolidate the documentation or the specification on top of that. And then what happens also is API specifications are not meant to be static. They're supposed to be changing and evolving all the time. But you should be making those changes in a backwards-compatible way. So, perhaps it depends a little bit on the type of changes that you're making. If you can make soft changes, that don't break the previous integrations, that's great. Otherwise, maybe you want to use versioning for different versions of the API.

10:57 Miko Pawlikowski

Sure, diversion definitely helps with most things in life. I'm curious about the documentation first design. I've experienced multiple schools of thoughts. To give you an obvious example, if you've already designed an open API specification, you can probably just go and generate at least a bootstrap of your API from that in the language of your choice. There is also the other side of the spectrum where you have self-documenting APIs, where you have some structure and some descriptions inside and then you generate the docks on the other way, in the other direction. Do you have a preference of which way you typically go? Did you find that one of the ways ended up being better long-term for any particular reason? Curious if you have any experience to share on the forum.

11:57 Jose Haro Peralta

Just from experience, just working with with my clients, what usually is the case, we start working without documentation first. Like I said, we need to go through a discovery phase. And usually it's a small and close team working on the API integration. So you're closely working with the people who are building the client code and the people who are working on the server. On a first stage, probably there is no documentation, you're using one of these self-documenting frameworks. When it comes to Python, I mean any language has a framework that can generate documentation from your code. So you use one of these. And that gives you the basic template, something to start with. But moving forward, what happens is we need to start making changes to the server into the client. And from that point, it becomes very important to have a point of reference in the documentation that tells you what's going to happen. At some point, we switch towards documentation first. And we make the changes to the specification first and then we cooperate them to the client and in the backend. That's otherwise, what I see often is we need this new feature, we need to change maybe the format of a field in the spec, the backend makes it, the frontend breaks. Or the frontend makes it and the backend hasn't done it yet any it breaks, because you're not working with the same type. And that's just one silly example. But there are many more, like additional fields to the API. Or we changed endpoints, suddenly certain URLs do not exist anymore. So if we can agree on the changes first and we can document them, it makes life so much easier.

13:27 Miko Pawlikowski

Definitely. So is there any particular API that you would consider the best example that when people want to design, they should design like that one?

13:39 Jose Haro Peralta

I think when you're spending so much time working with APIs, you are more, I think, if you're not complaining more about them than picking up the good examples. I could tell you some terrible APIs. I wouldn't want to mention names anyways. But I guess something like, again, nothing is completely perfect, but something like stripe and GitHub, I've enjoyed using them, the APS of these companies.

14:04 Miko Pawlikowski

Okay, so GitHub gets thumbs up, everybody listening. All right, so let's shift gears and talk a little bit about your book. So, at this stage, it's kind of making sense to me why you would go and write a book about APIs. Apart from just being deeply into that, what was the primary driver for writing an actual book about that? And also, weren't you worried that this might age quickly? I know, from the quick scan, it looked like you've focused primarily on Python in there.

14:38 Jose Haro Peralta

Yeah, it's in the title. It's about microservice APIs with Python.

14:41 Miko Pawlikowski

Exactly.

14:42 Jose Haro Peralta

So yeah. The primary driver for this was after working with various companies, building APIs, I learned a lot about the workflows and the process of building an API, like the methods and approaches that are more likely to lead you to a successful API integration and the approaches that are more likely to fail, the tooling, all these things. The thing that stands out when, not always, but in many cases is just how difficult it is for people to build APIs correctly. And I just found in my case, sometimes set in the situation where I'm always teaching the same thing over and over when I join a new team. And sometimes it's not just teaching, but also having a hard time convincing people of what are the good practices. Anyways, what I thought is, I just want to collect my experiences, everything I've learned about building good APIs and I wanted to put it to one place where people can read it. And perhaps within some time, when I join a team, I don't have to repeat the same things again, over and over again. Perhaps there are some basic stuff that we all know already. I think there's a number of basic things that we should know when we're working with APIs. I'm making a big effort in the book to explain these things in detail, in an easy way. Yeah, it's mostly on the basic stuff. There's some basic we need to know. I think that's the primary driver for writing this book.

16:00 Miko Pawlikowski

I like that. So, basically got bored of explaining the same things over and over again. Next time I'll be like, "Just read this". Okay. It sounds like you do find that there are some main takeaways and like the right ways of doing things, at least for the basics, and some low-hanging fruit to avoid problems, which makes sense. I'm also curious about, you know, I went through the process of writing a Manning book myself last year. How was the experience with meap so far? Did you get valuable feedback from people?

16:40 Jose Haro Peralta

So far it's difficult to get feedback. I get some people who reach out to me, not through the, cause meap has a forum where people can write comments and stuff. But that's not very active at the moment. I would like people to write comments and to say, because obviously, there are going to be things that are missing or not well-explained or wrong, or things people disagree with. And I'm particularly interested where people disagree with, because that allows me to include additional views on the topic, or especially also what is wrong. I don't want to release a book that has something wrong. And people always know a lot about something, and they would be able to pick it up if they wanted to. So yeah, it's not being useful in that way. People reach out to me throw all the channels, through LinkedIn, or some other mediums, and they give me some feedback. I think people can use the meap platform, you buy a book, you have access to it. And you purchase the book, you have for every right to demand from the author to improve it, because it's still in development. You can put your demands forward and make your comments and request things.

17:47 Miko Pawlikowski

Yeah, so for non-Manning people, meap stands for, I think, "manning early access program". Which is pretty neat, if you ask me about that. Currently seven chapters available, which is roughly around halfway through it mark.

18:08 Jose Haro Peralta

Yeah, halfway through the release. We have a few of the chapters written, we will be releasing in the coming months. We just released chapter seven and it's actually one of my favorite chapters. That one is about APIs, it's about service implementation patterns for micro services. So it teaches us things like the repository pattern, the unit of work pattern, hexagonal architecture, how you use these things to actually build more maintainable and decoupled code.

18:36 Miko Pawlikowski

Okay, so I wanted to ask you a bit more about the choice of Python. It makes sense on one hand in my head, because depending on how you count, it's definitely in the top three of most popular languages. On the other hand, it's pretty terrible at some things, like concurrency. There are languages now that do it better or make it easier to do it better, like Golang that seems to be more and more popular every day. Why did you pick Python? And why did you decide to focus on that almost exclusively in the book?

19:11 Jose Haro Peralta

Primarily, it's because I'm mostly Python developer, it's my main area of expertise. Perhaps, contrary to what many people think, it's a very common choice for building web services. And sometimes when I talk to people who are from outside of the Python community, they seem to think that Python is not very commonly used, but my experience tells me otherwise. Obviously, you're in your area, you know your community and your appliance and stuff like that. In my experience, Python is extensively used for building web services. I would say you never choose Python for performance, for concurrency speed or anything like that. You choose it because it has a very simple interface and it makes it easy to prototype very quickly and to iterate very quickly in your designs and your code changes. So if you want to build something quickly, Python makes it easy. It doesn't have any kind of overhead, doesn't enforce types or anything like that, it's not compiled. In that sense, it's a great choice. And I think appeals a lot to startups who will have to build very quickly and release very quickly. And it also has a huge ecosystem for really anything you want to do, especially when it comes to data analysis and modern websites. It's difficult to think they wouldn't include some sort of data analysis component. So with Python, you have an extremely rich ecosystem of tooling for doing those things.

20:29 Miko Pawlikowski

That makes sense. Definitely the popularity always helps. But I'm wondering, can you mention a few of the dark secrets the duo, the async.io, the kind of bad side of using Python?

20:46 Jose Haro Peralta

Yeah, so if you're unaware of, like you mentioned, Python doesn't make it easy to write concurrent applications, right? If you're not aware of that, and why it is. So you need to understand Python always executes in a single thread, because of this thing we call the gil, the "global interpreted lock", you need to be aware of that, because then sometimes you see people, they think they can do some multi-threading to speed up the code, and it doesn't happen, it won't happen. There's no way, because Python doesn't work that way. And it actually can make things worse. You wouldn't take that approach in Python. You can do asynchronous programming with async.io, the thing about it is one of the most complicated modules in Python, and it has one of the most complicated programming models in Python. So, in practice, is not used that often. Your best shot at that is using frameworks that already leverage async.io. So, if you're going to build web applications and APIs, there is this very good framework. It's not super popular in Python, it's called fast API, it's built on top of a starlet. And they're asynchronous, they are built on top of async.io. And when it comes to web service, as asynchronous programming is kind of the key to everything, because you're going to have incoming requests all the time, some requests, doing a job that may take longer, or may finish sooner or later. But in the meantime, some of the requests have to be served. If you're pro this asynchronous, you can address this situation. If you're trying to do something else, your requests are going to be waiting, and they're going to timeout. Just be aware of these things when it comes to Python. Many performance issues, you're gonna have to address them with more infrastructure. And you have to realize, if your application is all about speed and performance, maybe Python is not the right choice. It is the right choice if you have to build something quickly and if you can address performance without infrastructure.

22:42 Miko Pawlikowski

I know quite a few places that use Python to basically do the first implementation and then, when the need arises to make it higher performance, they would just be re-implemented in something else. And that's probably not going to change because, like you said, Python is hard to beat at prototyping and the speed you can go and the amount of stuff that you can get away with doing.

23:07 Jose Haro Peralta

And for some, unless you end up with a component for which performance is critical, you don't need to re-implemented in something else really. I've seen some, you know, when you're working with a language, there is always love and hate. And there are things about any language that we would like to complain about. And it definitely happens with Python. Especially the more popular the language becomes, the more complaints we're going to have about it. And there have been some companies that, I read about this a while ago, some component like cases where people decide to switch to a different language, from Python to Node JS, for example, this is fantastic for web. But then the switch doesn't really work. I guess it depends a little bit on like switching. When you're going to switch languages, you need to consider your capability to do it, you need to have the skills and the knowledge to transfer to a different ecosystem. And you need to know the ecosystem as well, you need to know the libraries and the best choices to build your stuff. If you're not very strong, if you're not in that position, the transitions is likely going to fail.

24:10 Miko Pawlikowski

Definitely. So the microapis.io looks like some kind of a side project. Is that you basically helping people with their Python APIs?

24:22 Jose Haro Peralta

Not at the moment, but I would like to expand it into something like that as well. At the moment, really the main point of it is, the main thing it does is, it allows you to spin up a mock server in the cloud. But the main point of it is you can do it without having to sign up or log in. There are lots of cloud services that can do this thing for you. And you can do it in your local machine as well, too. You know, this is especially used for developers who are building API client. Frontend developers, mobile developers, or microservices developers who are integrating with all the services and you need to simulate the integration with the backend that you're talking to. There are multiple ways of running a mock server. You can do it locally, but then you have to install packages, you need to have the right run time, you may have to resolve some dependency conflicts. And you need to have one process running locally, which isn't always possible. Running it in the cloud with most providers, you need to have an account, which again, sometimes for for a quick test is just not what you want to do. So the point of this is, you need to do a quick test, perhaps in a CI process. So you can do just a simple API call, you get a mock server, and you can run the integration against it and throw it away.

25:35 Miko Pawlikowski

Awesome. So microapis.io to check it out. One last question. And this is one that I really like, because I get to steal good things from people. If you were to pick a single highest return on investment thing that you did for your career. And it could be anything, from changing your mindset to going on a trip in Himalayas, what would it be? If you could pick only one?

26:04 Jose Haro Peralta

Tough question. My professional history's perhaps a little bit different from other people. I don't come from a formality background and it wasn't easy for me to get into this path. But the way I did it was, I was working on a PhD on an unrelated topic, I decided that I could use software, I could write my own software to analyze data. And it's something that I just had the idea and nobody really took it seriously. But I believed in this idea and I pushed forward for it. Basically, I think that the takeaway is, if you believe you can do something and you really want to do it, then go ahead and do it. Just believe in yourself and hang in there, because the journey can be tough and things may not come easily, but they will eventually happen. The most important thing is you work hard and you stick to your goal.

26:53 Miko Pawlikowski

Work hard and hope for the best.

26:57 Jose Haro Peralta

Yeah. And usually it's fine. Usually things will be fine.

27:01 Miko Pawlikowski

Okay, I like that. Usually things work out.

27:04 Jose Haro Peralta

In the end. Just be patient.

27:07 Miko Pawlikowski

Alright, Jose, thank you so much for your time. That was a really interesting conversation. For everybody who wants to check out Jose's book, it's called "Microservice APIs with Python", it's available at manning.com. In the early access meap program. If you need a little match, Jose and Manning were kind enough to provide two codes. One is for a free book. So whoever uses that first gets it. You need to use a code and the code is confapirf-D954. So confapirf-D954. And if you missed this one, you can get a 35% discount code. I think that's available for all the books. We're using the code podconf4221, podconf4221.

28:00 Jose Haro Peralta

Yeah, I'm not sure that one is available for all of them. Definitely it is for mine.

28:04 Miko Pawlikowski

Definitely.

28:05 Jose Haro Peralta

I just can't promise all the others.

28:07 Miko Pawlikowski

Go grab a copy and learn from the mistakes that Jose made when he was designing for his previous APIs. So that you can learn from his mistakes and not make your own. Thank you so much Jose, that was really cool. And I'll see you next time.

28:25 Jose Haro Peralta

Thank you so much. It was a pleasure talking to you.