Hi, hello everyone and welcome to my talk. Today I'm gonna talk about security by testing and how I try to improve the security of my go projects just by writing more tests. Before starting the talk, I will give you a little introduction about myself. So, my name is Alessio Veggie. I'm a software engineer, full time cat food opener for my furry friend. And jokes apart, I'm passionate about reading and taking long walks on the social media. You can find me on GitHub, LinkedIn, Twitter and so on with this unique account and this amazing avatar. So let's start the talk first concept to know is about code coverage code coverage is a metric that can help developers understanding how much of their source code is tested and how much is not. It is mostly used when writing a unit test, but not only related to them, and it's expressed as a percentage. Let's go a bit more in depth with code coverage. With Golang it was first time introduced in version 1.2, April 2013, if I remember well, and this support was specifically for unit tests. Here's the link to the announcement. And the story continued after almost ten years with version 1.20 with a new grid support for the integration test. So this means that combining the unit test with the integration test coverage, the coverage percentage of our projects sensibly increased thanks to the last feature. During this time. Also, the go community introduced some nice tools, like for example the go to cover that gives you the ability to see the to render the source code with its coverage in an HTML page and so on. So that's, that's what happened. Another important concept is about Secomp profile. First of all, Seccomp is a security feature that resides on the Linux kernel and the second profile is basically list of syscalls that you can use to put them into a rule and attach this rule to a program by defining if these syscalls could be executed or not by the program itself. So it works as a firewall for system calls. Additionally, it is extensively used in the kubernetes ecosystem, especially when you enable the second default feature flag. So this means that all the workloads that you are going to run will be, will be attached to this default profile. That is a default second profile that have inside the most dangerous syscalls that you should never use in your run. Let's see now, the main idea that I had during the tasks, so the main thought that I had was to create to generate a second profile as an artifact at the end of a test pipeline. Why specifically the test pipeline because in the test pipeline, we are testing, as much as we can, our test our code through the test that we write. And depending on the percentage that cover our, our source code, we can rely on them for a security profile. So if we are testing the 70% of our code, the syscalls that we are going to extract on the test pipeline will be reliable for the 70%. So that was the main idea. Additionally, this second profile could be used in different ways. The, the most common that came into my mind was to use it through an init container. When we deploy our application, the init container basically injected the second by downloading it from the artifact registry and store it into the Kubernetes node in order to allow the application container to load it and run with it. And there's the example. So the init container basically download it and store it under Barlib Kubelet second, and then the container, in this case the NginX container for example, uses the security context second profile of type localhost by referring to the second profile itself, but with the localhost profile. So now that we have the main concept in our mind, let me explain you what was the step that I followed and how I tried to achieve this goal. So in this image, you can see basically a call graph of an example program. So having that in mind and having all the tests that are rendered can basically understand what is missing and what is not. So we have a general overview about the potential syscall that you are missing if you are not testing the binary. So in order to extract the syscalls from the test with the integration test, that was the easiest part. What we have to do is to build a binary, provide some script to check for the expected result, general scripts for the integration test. An example that I really appreciate is the test script suite. And basically we can run our binary by using strace or pair for whatever you want. So in this case we can extract, we can collect all the executed syscalls from the binary by testing all the possible branch that the executioner has. So this was the first part, but it was not enough in my opinion, because we were missing the unit test that are such a big part of the test pipeline. So how I tried to extract also the syscalls from the unit test, first of all, it's quite simple to say, but it was a bit more complicated. First of all, the gotest command, we have to think that the gotest command compile and run the test binary all at once. So when we run go tests, we are basically compiling the test and then automatically run with the same command. At first glance we couldn't do strace of go test because we were going to hook, we were going to trace calls that were not related to our specific test, but syscalls that were included in the go test command that were out of school. So what we could do at first I thought that we could for example compile only the test binary without running it using go test, but running it separately. So if we basically type gotest c, we could compile the test binary and run it by ourselves. The problem in this case is that even doing strace about of the test binary, we could include some noise that are not related to our specific function that we are testing, because for example we can include some function that load some test data from, from the, from the environment. So as in as an example we could open a file, read the file content, and this syscalls will be collected by strace. But we don't need that, we should try to find a way to avoid this kind of noise. My personal idea, I don't know if it's the good one, but it seems to work, at least in some cases. But the main idea was to create using a BPF to define a trace point that basically start tracing the syscalls. When a probe that is attached to the function that we want to trace inform us that the execution of the function started and we can stop the trace point when the uert probe inform us that the execution of the function is finished. So we have a range of time that we can rely on in order to collect the syscholes that derives from our function. An additional information is that Arpun was based on go BPF at the beginning, but then I moved to Libpfgo and I will explain later why I took this decision. So let's see the juice part. So in order to run harpoon what we have to do, we should basically build the test binary first as we were trying to do before. So just typing gotest c followed by the package pop that host our function. So in this case we are going to build the test binary. So as a result we are going to have a binary that will execute the test where our function is located. Consequently, we have to extract from this binary the symbol name of the function that we want to trace. So in the example I typed the myfunction. So myfunction is the function that we want to trace and we are using objdump since followed by the name of the binary test by graphing for the function name. In this case we are going to find the symbol of the function name. So as you can see in the example below, so we are doing objdump to test binary of mine. It was an independent project from this talk and we are graphing for the function interface exist. So interface exists is a function that I've created in my personal project, a go function that we can find on the binary on the test binary with the following name. So GitHub.com allegra 91 forwardctlash packages iptables dot interface exists. So once we have the symbol, the symbol name of the function, we can use arpun in order to extract the syscalls from the binary. So by typing this command arpun fm to specifying the the symbol name of the function followed by the command for the test binary. So as I explained before, what is going to happen with arpun is that arpun will take the elf binary. So binary test and we'll attach a u probe and a u ref probe to the main dot. Do something in this case, but whatever is the name of the symbol or of the function and will, after it attached these probes to the function, will basically run the binary and the trace point will be informed by the u probe and the u rh probe about the starting and the xd and the end of the function itself. So in this limited range of time we are going to extract the syscalls. And here it is an example. So by typing arfun fn followed by the function, the symbol name of the function, followed by the command of the test binary. So iptables test. So here's the list of the functions that are related only to the function in interface exists. Let's see now the worst part of this of this project, everything was looking nice, but at some point I realized that not all the things were working properly. Main thing that was not working properly was that the u rat probe sometime was not informing the program that the function was returned. What was the problem? Basically the Europe probe overwrite the return address of the probed function with an address of a trampoline. This trampoline is basically pointing to our EPF program. So once the EBBF program is executed and after it sends the instruction pointer should restore tool to point to the next instruction. But this thing doesn't happen all the time, since the stack in the go binaries dynamically changes due to the garbage collector, for example. So this kind of behavior could cause the program corruption. So I had to find a way to solve this issue, and luckily for me I found this workaround on Internet. So the workaround consists in starting from the fact that the U probes could be attached to a specific offset. So we don't need to attach the U probes only on the symbol function names, but we can do the same to a specific offset in the function, in the binary function. So in order to simulate the URET probe, what you what we could do is of adding for each rEt instruction that is inside the function, a uprobe that basically simulates the functioning of a UART probe. So instead of attaching one single u ret probe in the function, we can add the one u probe for each ret instruction within the function that we are tracing, and the rest of the function is the same. So benefits of moving to lib bpf go so at the beginning when I introduced arkun, I mentioned the thing that was based on go bpf, that is a library from the iovisor organization and that is using bc under the hood. So now I decided to move to lib bpfgo, mainly because libpfgo gives you the ability of attaching the u probes to specific offset, and this thing was not supported by go bpf. So we can basically simulate the functioning of our UART probe by attaching the U probes to the RET instructions, thanks to Libbypfgo. And the use of LibfDo makes the project even more distributable, easily distributable because the LibPF Go is a library that is a wrapper of Lib BPF. So Lib BPF gives us the ability to write the program with Cory. Cory means compile once runs everywhere. We don't have to build the binary every time we run the application as we did before with the Go BPF that was based on BCC that needs GCC as a dependency to do this thing. So this time we can simply compile the binary the first time from the pipeline, for example, and then distribute the entire program through the repository. So the talk is almost finished. I just want to point you to some links that helped me a lot understanding this problem and links from which I learned a lot. I also want to thank some people that helped me doing this project. Thank you for your attention and I hope you enjoyed the talk.