Hey everyone, welcome to Conf 42. Thank you for tuning in to listen my talk. Today I'll be presenting on optimizing performance and security and building a robust API in go. So we'll look into different ways how we can make our API much more performant, fast and secure. So let's do a quick introduction about myself yeah, so I'm currently working as a senior SRE at Capsule. I'm also a docker captain and apart from that my work revolves around backend container systems and security. So let's get started. There are one building an API. There are a couple of things that we wanted our API to be. One is we want it to be secure, we want it to be fast, and we want it to be robust. So in this talk we'll look into how we can, how we can address each of those using different techniques. So first let's see how we can make our API secure. There are a couple of things that need to be done when you're working with security and authentication and authorization. First of all, like if you are using conventional username password, it's best to use token based authentication as it is more secure and also know when like what type of token to use. In what kind of scenario like access token or id token, you could use something like assigned JWT with strong algorithm and strong secret. However, we all know that JWT comes with its own set of issues, mostly regarding misconfiguration. This is where the newer newer way newer newer tech like pistol comes in, where it is more opinionated and leaves less room for errors. If you are using SSO, using Oauth, it's good. If you like, it's good to choose like Oauth 2.0 with proper authentication. Sorry, the overflows like when you're working with microservices authentication and authorization, you need to use like you it's good to use access token and refresh token. So it depends on the company and the team. The other thing that we can make our API more secure is use a proper course configuration. A lot of time we see people using leaving course configuration as default, which leaves API open to different, different problems. And, and most of the time when we store cookie s or so store token, we store it as a cookie. And in order to make our cookie robust or secure, there are a couple of things that we could do. One is enabling secure and HTTP only flag. Basically what this says is our cookies can be transmitted only if the connection is using TL's and means SDP. Only flag tells that JavaScript cannot interact with the cookie. We also need to store our cookie. It's good to encrypt it. There are a couple of different ways, but one of the widely used is HMAC with salt. People use salt and pepper, but it's HMAC with strong secrets as a salt should we should be good enough when we're working or whenever. When we are developing locally. We use SEM site policy as a bit relax or lax, but people forget to change the configuration in production to be more streamlined. It's good to use principle of list privilege when you're working with authentication and authorization. And the other thing that I like to do when working with API is I like to re authenticate user before any kind of critical change, whether it's the email change or password or update profile, profile update basically, even if they are logged in. So any kind of change they are trying to do should be, should be checked against. Is the user authenticated to the do the operation and also is the user are authorized to do it? Again, this is kind of extra guardrail we can add in when working with API. The other thing or the next thing that I wanted I'd like to do is validate inputs. It's good to validate and sanitize all the inputs coming in so that we protect ourselves from injection attacks like SQL, NoSQL or XSS. You could roll your own utility functions to do this stuff. But there are other better test libraries which you can leverage like validated and blue Monday. Here's an example which uses validators. Let's go to the other one. I like to create an audit trail of user activity. This is like, this is interesting because how it works is first we define set of activity that user is allowed to do and create. Then, then for users activity we create audit or log trail. And if user tries to perform any kind of activity which is outside its defined baseline, we alert, or, or basically alert the user. Like administrator example would be user who is a customer should not be trying to log into the portal. Like, so basically this, this like having a trail would help us in the retrospect of the things that you as a user was trying to access also, and also helps us proactively protect us from malicious actors. The next thing that I, it's good to do is use appropriate response as part of the, as part of the API response. Because let's see an example here. When like if in the above example we can see if the user doesn't exist, we say user doesn't exist, right? However, with this error message, the attacker can enumerate whether the user actually exists on the system or not based on the response. However, in order. Like instead of explicitly saying whether user exists or user doesn't exist, it's good to have a generic response for everything so that people cannot run like malicious actor cannot run enumeration attacks. So password management is also one of the key thing that you need to be careful when working with API security. It's good to use one way hash functions so that while storing the password and also while comparing them, one of the most widely used is bcrypt. The other thing that I like to do when working with passwords is I don't, I don't allow certain consecutive characters from user email or any of their field in their password. So what this does is basically eliminates attackers trying to do brute force based on previous known knowledge of their like emails or something. And the other thing, interesting thing that we, we can do is check user's password against a list of like one hundred k a million probably common most common password. And if it falls on one of those range, we just deny it. It's good to do that. So here is an example I'm checking a password to in like checking the request password in if it falls on the range of common most common use password and if it's, if it false, then we say password to common and send a status bad request. Other than that, what we could do is we could check for complexity. So they're making sure that user password has consist of certain healthy amount of alpha numeric characters, a string or and, and numbers and different like uppercase and lowercase list. And we make sure like certain length is also fulfilled. We don't want the password to be too short so that it could, it is easily breakable using brute force. And the next thing that we could do to secure IPA is like using passing secret to config file instead of environment variables. Personally I don't like using environment variables, like are using secrets using on like passing secrets through environment variable. The main reason being they can be accessed from any part of the program. Like people can just go like to osgod like get amp and then fetch the secrets. Instead of that you could just pass the secret from environment where like config file. Because now when you load the config file and pass it to each functions which require it to like direct argument or dependency injection, like you are basically the single like single instance of that configuration is being passed around throughout the program. And also you can lock the like lock the file with file system permissions as well. So it's like, I find it much much more like what you call gated compared to environment variables. So here's an example of server struct which, which has configuration as a dependency and you can pass it using like pointer receiver. The next thing that we can do to make our API more secured is secure is masking sensitive data. Basically we want, we want to mask any kind of sensitive data which might reveal user information while either it's in transit or address. So there are different techniques like just using the masking data masking technique where you could just like encrypt the data as well or use to organization depending on the circumstances. So next thing that we could do to implement is implement rate limiting. People get confused like how limiting actually helps in security, but, but it helps us to prevent abuse and protection using DDoS attack. So basically we don't want people like malicious actors slamming our API endpoint and, and like dosing it, right? So it's good to implement rate limiting algorithm. There are different algorithm like fixed window account, leaky bucket and token bucket. However, implementing rate limiting, we need to consider a couple of things as well. One is we need to plan for burst. There might be some situation where certain source of traffic might come, like example salesbitting problem. Or we need to make sure like we need to make sure like each endpoint are different and each endpoint like traffic on each endpoint come at different rate. For example, password reset API endpoint might not receive same amount of requests, same number of requests per like per minute or per second compared to get orders or list items, right. So we need to make sure like each of them have like specific rate limiting policies applied to it. And what I like to do is implement like respond the rate limit as part of the response metadata. For example, here I am responding the total limit of the request and the remaining amount of rate limit. And also like when the next time the like rate limit window would be re initiated. Basically this is, this is the next time the rate limit would be reset. Next thing that we could do to secure our API is having us like HTTPs connection between client and the server or API. And we could also implement API to support TL's natively. This is like kind of good practice because not always we will have some kind of proxy sitting in front of our API to terminate the TL's. It's good to have the functionality where you can natively support TL's if need be. And other thing that we need to be very careful is we can't build everything on our own, right? There are instances where we need to use external libraries. This is where staying on top of your patch management is a must. And I use tools like a sneak GitHub dependable and even if it's too noisy, this, it gives, it gives you at least like helps you stay on top of your patch management. And, and using tools like CSM, like CSM, tools like Wiz is like added benefit. So we talked about how we can make our API secure. Let's see how we can make it fast. I had to put it here like there are countless instances I've seen developers using select asterisk to get in the data and then perform the conditional queries. However, the problem with this type of query or this query is it loads everything into the memory and then performs the logical operation instead of that. If you know what are the fields that you are looking after or looking for, you can explicitly define okay, I need this column from this database where, where the details is this. And, and basically what this does is it reduces the total amount or optimizes your query. And, and basically the, the load or like resource load on the database would be much much smaller. And if you find like multiple column, like same column being requested a lot of time we could, what we could do is implement indexing so that it is more efficient and optimized. The other thing that we could do is minimize redundant queries within our API as well. So next thing that we can do to make our API fast is implement caching. I think this is a no brainer. If the same thing is being repeated again and again without being changed, it's good to cache it, right? Because it has a lot of benefit. One is like it reduces the latency. The next thing is it maybe like it puts lesser load on the database as well because now API doesn't need to actually like talk to the database, database doesn't need to do the processing and return the value. There are different ways you can cast one is in memory, or you can use external key value stories like Redis and I like to respond, respond with like cache cache control headers like etag or cache control so that client can cache certain data for like on if they have the capacity to do that. And leveraging CDN for static contents like images binaries is also a good idea. The next thing that we can do to improve our API performance is pagination. Basically, instead of returning all the data and trying to organize into group, what we say is I only need this much like this first ten data and, and then you request the next ten data. What this does is it reduces the data transfer size and which, which ultimately makes our query much faster and, and if it like effectively makes our like reduces the latency as well. So pagination, like using pagination, we can make our API like pagination is not just for what you call you. You don't have to use pagination just for like making it look nice, but it also actually helps in making your API much fast. You can use different techniques like cursor based pagination or offset based. Here I'm using offset based pagination where I specify the page number and the total page size I want. And I think everybody seen this coming. If your application is too large, it's good to break it into smaller services. Breaking your application into microservices helps you make it like make the application independently scalable, maintainable, update able. Like it will be resilient to failure if you make it right and future proof as well. So microservice is good. However, microservice is also a double edged sword like sword. A lot of time we see people implementing microservice just because things like they think it is good. However, like do not just go about converting your app into micro service just because you can, right? For personally, I think what I think is microservice is only required if each business unit or each domain of your app is handled by separate teams. An example would be like fulfillment, order processing or payment. So yeah, now we talked about how we can make our API secure. We talked about how we can make it fast. Let's, let's see how we can make our API robust. The one thing that I follow is fail early, fail fast approach. Basically what it says like what this means is we fail immediately if something, some condition are not satisfied. Like we don't wait until, until, until the actual like resource is, is required. This helps to reduce cascading failures and also helps us quickly identify different issues issues before they become a major concern. And in general it makes the API more robust and stable. So here's an example. Since our server is dependent on configuration from past configuration like protocol or TL's passed from the config file, we, we first check whether config file is present. If it's not, we just feel like immediately return the error and fail. So this is a good example of fail fast and fail early. Let's look at the next slide. Yeah. The other thing which we can do to make our API more robust is use appropriate status code based on the situation. So like, let's see an example here. Here we have to get responses to API admin. One returns 401 unauthorized and the other one responds with 403 forbidden. Now suppose, suppose user logs in the credential are correct, but the user is not allowed to access the page. Now which, which response code do you think is the correct one? This is the like correct and more more readable one, right? Because you 401 unauthorized means like credential was correct but user is not authorized to do something. So basically what? Like basically using appropriate status code, it helps us like add more clarity and understandability to your API as well. And also it helps us to easily detect the error. However, we don't want to, as discussed previously as well, we don't want to reveal too much information which would be, which can be used against us to compromise our security. So we need to find that sweet spot between security and robustness. And yeah, I think every, like all the groupers know this, don't just check errors, handle them gracefully because just returning nil. Okay, looks like I did a typo over here. It shouldn't be capital like, please forgive me. However, like don't just return error like this. They call it naked, naked error or something like that. It doesn't have any information. It doesn't give you more idea what actually happened and caused the error. So adding context to error is a good idea. So what I, what I personally try to do is I only panic on the main function and any function outside the main or any of the modules, I return the error back to the main function and return them up the call stack. So basically, suppose if something, some error occurred, I would return, create an error and then return it, return it up the call stack so that it can be printed with a nice, nice context around it. So the next thing that we could do to make our API robust is passing context. Context as in here, I mean, request id require, like passing request id on each request helps in tracing and debugging here. Let's look an example over here. Right? So somebody did a post request and the request id is this. And let's, let's assume the request came in and the request was successful, but the client, like we sent a response, client was not able to get the response for some reason, and it tried to retry again with the same request id. Now if we pass the request id, we can check whether this id is already seen or not. And if it's already seen, we can actually ignore or don't don't take any action on this request. So it's really good idea to pass id request id when you are dealing with APIs and there are other benefits as well, it helps you to correlate between different logs. Because when we aggregate log all the log into single location, we know like not all logs comes at the same time. And it helps like using request id helps us correlate. And also you can pass it with the tools like JQ, right? It's your integration with tools like Sentry, Jaeger and like in general maintain idem potency. Let's go next one. Yeah, so the other thing that we can do to make our API more robust is make it acid compliant. What? Acid compliant? I'm like, what I mean by that is our database transaction from our API should be either all or nothing. So suppose if you are using, if you are using a banking, like creating a banking application where user was charged, the user bought something and when user buys something, some amount needs to be deducted from their account, right? But for some reason the account got like amount got deducted but the checkout was not successful. There should be some mechanism where you set up so that if the order fails the transaction for reducing like deducting the amount should also be rolled back. So this is where like using context and transaction makes the operation more asset compliant. And it is more like making API asset compliance is much really important when you are working with multicultural collection updates because updating to one collection might work, whereas might not work for the other one. So it's good idea. Let's make it next. Yeah, the next thing that we could do is use structured log. Like as said in request id are passing context. Context request id using structured log helps you maintain and it helps you to like debug and search search your logs easier as well. Using structured log you can directly integrate with tools like low key or other different observability tools. The next thing that we could do to make our application more robust is implement health check. Here's an example of readiness and liveliness. Uh, health check response. So basically health checks has a lot of benefit, mainly be being like, you know, whether your application is running or not and also whether the application is ready to receive the workload or not. So this is like, this concept is like it is already used in containers or docker containers as well. However, there's no reason for us to not not use the same in our APIs as well. It like you can use it for service discovery, you can use it for auto scaling and load balancing. So here this is the example of readiness, sorry, liveliness probe. So liveliness response or health check where we like see, check whether the server is running or not with with like commit id, branch and tag. And here we are checking whether the services that the API depend on are okay and the connection is successful or not. This is where like we, we verify whether the application is ready to receive the workload. The other thing that we could do to make our API more robust is handle retries really gracefully. So let's imagine like some client got booked and it kept on retrying the same request again and again. Like we call it retry storm. Other than like having a bad user experience, it adds loads to the load to the server because it like we are effectively trying dosing our server, our API. The, the way we could resolve this is using a technique called exponential backup where client like progressively awaits longer time. Like longer time is like longer each time like backup limit is reached. So this makes API more fault tolerant and also more reliable. And it helps in resource utilization as well because now API doesn't need to handle all those like bogus retries. Why is it going to, yeah, so yeah, as I mentioned before, while handling retries as well, you could use appropriate status code. I like to use 429 and 503 for scenarios like this. And also I like to respond with retry after header. So basically now client knows after like how long to wait before trying again. The other thing that we could do to make our API more, more robust is expose metrics. Basically exposing time series metrics of your application. It helps you like it helps you, exposing your metrics helps you to set up a baseline and then compare, compare that baseline with when the application starts to receive the metrics. And now that comparison can be used for a lot of different benefits such as like detecting anomalies. Like if there anything suspicious going on on the application with the application we can use it for planning, like capacity planning. We could use it to set up SLA's with other team or our customers. And also, yeah, in the identify like use that as part of the root cause analysis. And the next thing that we could do to make our API more robust is version them. I think we all are aware of it. Making versioning our API makes our like makes it more maintainable, stable and flexible. And like it makes the API adaptable as well. Like we have. We want to make our API to be backward compatible as well because some provider or some client or some customers of our API might not be interested in upgrading to a newer version because they have implemented the solution in such a way that they can't use the newer feature that was released. So version your API is really good idea for making it more robust. In conclusion, building a robust API is trying to like watch a movie in this nineties tv, right? We twist and turn the norms to find the sweet spot and where we are happy with brightness, jitter and sound and picture quality. So it's all about finding that sweet spot. So thank you for watching. Thank you for listening to me. You can scan this QR code to reach out to me directly. Really appreciate you listening to the talk and enjoy the rest of the conference. Thank you.