Transcript
This transcript was autogenerated. To make changes, submit a PR.
Hey everyone, welcome to our presentation.
Today we will be talking about CI
CD tool that we developed to secure runners using
Berkeley packet filters. And I'm
Mert and I'm joined by my friend Jenk, who's the CEO
and co founder of Kondukto.
For introducing our tool, we will first talk
about the trend landscape and to showcase why we're
developing this tool. And then we will talk about
why we choose Burkett Packet filter and how
we developed our tooling approach, and then
we will showcase a demo of the tooling. So if we
look at the trend landscape today, as you can see,
there are a lot of malicious application in the wild which
dramatically increased throughout the years. And if you just
look at the two years ago and compare
it with the last year, you can see there is a
lot of interest from attackers in
terms of malicious packages. And as
you can see, there are common attack vectors for attackers to achieve
this, the first one being the vulnerable packages. They publish
a vulnerable package using typoscutting attack or publishing
a package that resembles a legitimate package
name that is widely used so that they are
trying to trick the developers to use those vulnerable packages instead.
And similarly, they also took an approach of compromising
pipeline tools so that they can tinker with them
or they can publish again typoscarted pipeline
tool, which might seem benign but
might not be. And lastly, they can write
out compromise the artifact integrity. So this could be a code
or your container image, so they
can target those and compromise them in order to for example,
publish or deploy code of their own choice for
crypto mining purposes or some other purpose to leave backdoor.
So based on these common attack vectors,
we can talk about, for example, one of the recent threats that
is happening in the wild nordic career based
threat group recently published, for example, NPM package
called execution timer sync,
which is a copy
of a packet called execution time which is being used
as a code profile profile provider.
And this package aims to steal the victims login credentials and
passwords from the browsers. But as you can see on the left
side, it also has a python script capability to
further the compromise. Once you have this NPM package,
it collects the credentials and passwords,
and afterwards it tries to download this python script
so that it can remotely control the compromise instance
in order to achieve this. On the right side, you can
see the diff of the recent build that
was discovered by Filio. They're constantly
changing the IPs to stay hidden.
And this has been. If you follow the
blog post, you can see that this has been tied to North Korea.
And this is not the first time that some
threat actor group is trying to masquerade as a
legitimate package. And that's why
when we see similar threats in the wild,
which can affect also ci CD pipelines in form of malicious
packages or compromised pipeline,
we decided to develop our own,
our tooling, which is also open source called
control, and now jenk will deep dive
into the package filter. What is it and why are we using it?
As well as some of the use cases that we
looked at to implement within our tool to monitor
and prevent these kind of attacks.
Yeah. Thank you mark. So welcome everyone.
So in the second call section of this talk,
I will explain why we have chosen EBPF technology
to implement this open source security solution.
So EBPF technology is gaining more and more popularity
in the observability and the security space. And we decided to use
EBPF because we want to monitor
and prevent attacks or anomalies in the
runtime. And since BBPF allows
us to write portable applications because of the
compile ones and run everywhere approach, this was obvious.
And let me explain you what is EBPF?
Let me rephrase once again. So EBPF is
a way to write kernel
level applications without literally
writing or compiling anything in the kernel. And because of
this portability approach,
when you develop such an application, this tool
will basically run any newer Linux
kernels which most of the Ci CD runners
will apply.
Ebpf basically lets us
develop a user space application while modifying
and touching to the
kernel space. And as you can see in this example,
literally writing in small hooks in a specific
kernel section or kernel syscall, we can either
asynchronously see or monitor the props or
even modify for some of the.
So we asked ourselves, so,
how to develop and what to do while developing those
securing solution. And we explored
the options. And when we have seen all the
attacks, and when we have analyzed all the attacks, we have seen a common pattern
which was the network layer. Obviously we
can develop more and more controls and security
tooling using this technology.
We started to develop this open source tool with the
traffic control layer.
And how the system works or system has
been designed is whenever a new event occurs
in the pipeline, we are going to attach appropriate
kernel calls. We create
separate sections or channels, send those probes
into the user space, and user space will act
as a decision making or policy engine which
will allow or prevent or
block the relevant access code. So in the latest
attack that we have seen, we have observed the north korean
one, the Lazarus group. This design was pretty
variable. And if the
vulnerable package has been triggered in the developer's machine,
it was acting differently than the on this other side it was running
in the Linux section there was Android approach. So that's
why we have developed the monitoring capabilities as well as the privileging
capabilities and added those controls
in the tutorial. So what is the playbook?
So playbook, as I explained, first we should decide
which events or which books we are going to attach.
Obviously we wanted to have the TCP and the UDP
connections, so we are attaching the TCP
V connect and the IP four datagram
connect calls. Those calls are
creating dedicated channels into our user space
and our user space or acts as
a policy engine. And whenever an malicious
event has been identified, we were basically telling
our filtering function or filtering
to prevent the access. So this is the overall picture
and our aim, our idea is to develop
a full fledged open source ci CD
security solutions with EBPF power.
So let's jump to a quick demo. Then let's
explain how the system works. Right now we have
developed this as a single binary. So this
means that since the single binary, and again because of the
core architecture, it will work basically in any Linux
kernel. So as you can see here, on the left side we
have run the code, and on the right side we try to get
some HTTP calls from the Google and then
the Amazon. Since those are blocked, we see that those tcp
communication has been blocked already. And we
then created a wallet
call and download the artifact. And as you can see,
those TCP we get were
successful using this approach.
Let's show us how the same approach will
work in the GitHub runner
on a GitHub action. As you can see
here, we have developed a pretty simple and
pretty valid GitHub action
file. So we have two actions here,
or two steps. One step will act
as a malicious event, and the malicious event
or malicious step will simulate and
secretly cooperation. And on the other end we will have a
load action which will download the artifact or which will
upload the artifact. And finally we
will see the report, we will just see
what's going on. So we are triggering the event,
and since it's a single binary,
and we have developed and dedicated Docker as well, so it's
dockerized. You can run in the docker format as well, based on the
CI CD runner. So control binary
has been developed and one action prevent
action occurred. As you can see, the network unreachable,
and in the allowed action it works. And down
below we have a report that works
as expected. What was the idea behind this tool?
Obviously, if we can monitor the network
or what's going on in the pipeline itself,
we can take certain measures.
And we decided to develop this on an open source
because with the help of the community, security community and EBPF community,
we believe we can implement more and more sensors and
the latest version or the new version of the
tool supports UDP protocol as well,
and based on the configuration it can prevent access
or allow access to the GitHub meta IP range and
in progress feature we have
developed an open policy agent based rule engine.
It's been tested and we will publish soon. And afterwards
we will create an prop monitor process monitor
sensor to understand which
files has been accessed or even
touched, and we will get a list and
we will try to find anomalies in the
file system using the proc sensors.
So that was it for today.
Please join or contribute
to this project and if you ever have any questions,
feel free to reach us. We are happy to answer
or even contribute to your questions.
Thank you. Thank you for having the call.