Transcript
This transcript was autogenerated. To make changes, submit a PR.
During the presentation, I will talk about using Go for
automation and ad advertise Goyek. Goyek is
used to create build automation in Go as opposed to many other tools.
Goyek is just a go library with API inspired
by testing Cobra flag HTTP packages.
However, I really think that my presentation will be interesting
even for those not interested in yet another automation
tool. Here is the agenda. First, I will tell a few words about
automation. They will demonstrate free tools that can be used for automating.
In the end, I'm going to share with you my subjective conclusions.
During development, you format code, run unit tests, generate code,
deploy, et cetera. You want to have these things automated for sake of
continuous integration, continuous deployment, basically automating
your daily routine stuff for the sake
of this presentation so that I can demonstrate the automation tools.
By example, we will have a
simple build workflow for a Go repository.
We'll have an all target
which will basically contain the whole build workflow as it
will depend on FMT and test targets.
Then the FMT target will simply run go FMT for
all the packages and this target will be imported
as it can be used by multiple repositories.
Test target will run unit tests and
generate code coverage even if any test fails.
All of the code is hosted on a GitHub repository,
so you can look at it at any time you want.
We will start our journey by looking at make
file is the facto standard for automating builds
in go. Let's look at the
simple build pipeline created in make.
Let's start by running make which
executes the default make target the default
make target prints the usage. Let's run
the old target. As we can see,
it executes the dependencies. It run
FMT and then test which had failed. We see
that the go test had failed, but still the
HTML code coverage report has been
created here.
Let's look at the make file which contains the definition of
the make targets.
The first line is a declaration that
the common make file should be included. We'll look
into that a little bit later. We have the
all target which is the build pipeline which depends on the
FMT and test targets. Here's the definition
of the test target. It's a script,
a bash script, which first declares
a variable exit which is equal to zero in
the beginning. Then it runs go test with code
coverage, and if it fails, it assigns
to the exit its exit
code so that the
HTML coverage report can be generate later and finally
exits with the code assigned here
or with zero if the go test had passed.
Now let's look at the command at
the go mk file.
We are setting that the default shell is bash, so it
is more probable that it will work on other creating systems. For example,
macOS zsh is often the default shell.
We are setting the default goal which is the default target as
help. And help is a magic target
which basically prints
the usage based on the comments which
are added to the targets. The comments needs to
be defined as the
double bash characters. We have a reusable
print target function which says which function
is executed and then we are having some reusable
targets. And here we have the definition of the FMT target which
executes go FMT.
What are the challenges when using make?
Writing complex logic is hard for anything more complex.
I always need to look at docs, I need to look up at some
recipes, look at some make targets which I already was
using, or I need to use stack overflow or basically Google
some stuff. Also,
I do not like the bugging bash nor make which is often like
print into the console.
At last. It's not easy to create automation in bash at make
that works on Linux, macOS and
Windows.
When I was starting to learn Go, I heard that some
sres prefer writing go apps instead of bash scripts for
compatibility, maintainability and reusability reasons.
Most Go programs using the standard library work
perfectly on all supported operating systems.
Moreover, I prefer debugging go programs rather than bash
scripts or make targets in other language. It is
quite popular to use its own ecosystem for
automation. As far as I know, everything started with rabbi's
rake, but now Java has gradle, net has nuke,
etc. Therefore, I wanted to explore the possibility
of creating build pipelines in Go.
The first thing which I found was mage.
Mage is a makerake like tool using go.
You write plain old goat functions and mage is automatically using
them as make file write runnable targets.
It's time to look at mage. Let's run it by
executing go run mage go as
previously for make we see the usage.
Let's run the old target.
As previously, we can see that the test has been executed,
have been executed, and we have the HTML report
generated.
However, we have no information what test had failed and we have no information
what target had failed. Therefore we
need to run mage inverbose mode.
Now the output is similar to make file to make
let's look at mage go which we are
running using go run.
It's just a wrapper script on mage main which
executes mage. There are two ways
of using or executing mage.
One is using this wrapper mage go
file. The benefit of it is that thanks to
the Go modules,
we'll make sure that all developers are using the
same version of mage. The other way is to install
the mage CLA app.
Mage works magically by
finding the targets defined in mage
files which are required to use to
have the mage go build tags.
So these two lines are required. Then we have
regular go code. We have no regular imports, and here we have
the magical mage import which basically includes
all the mage targets defined in demo common
package which we will look at later.
Here is the definition of the all target.
The targets are defined as an export as exported
functions. In Mage we have an information
that we are depending on the
FMT target from the command package and on test
target from this package. This is the definition
of the test target. It returns an error because it can fail.
We are using the sh package
provided by mage here and
it executes go test and go tool cover to
run tests and execute test coverage. We are
using errors joined from the Gostanta library to make sure to return an
error if any of those two execution failed.
Here is the definition of the common build targets.
This is a simple definition
of FMT target. Let me
share with you the issues that I have with mage. For me,
mage is too magical. The target
discovery requires the magical build tags and because
of that you often lose the intellisense in
your id like vs code or goend. It's also recording
the magical import commands which you need to add before importing
the packages which contains the reusable
targets. It also has
some gotchas. Debugging is hard.
Even the author admitted that he is debugging by printing stuff to
the output, like in make concurrency.
Logging is tricky when you're running concurrently many
targets,
the logs are not synchronized, and also without
the verbose mode you do not get a lot of information in
the output and when the
things are failing in no verbose mode you get almost no information.
So as we had seen in the example, we have, for example, no information which
target had failed. At last,
the API surface of the sh package is huge and
I always had trouble which function
I should run. There are functions like run
run, v run with v run with et cetera, et cetera.
And the API is so big that it
makes hard to know which ones should be used.
And even if you know what you should use, then the readability is
a problem. When you read a code
that was written some time ago,
I felt that something was wrong and I could do something easier,
simpler, more idiomatic. It took me one
year to create initial version of Goyek,
so after many long nights I thought that it
should be a library. As libraries tend to be more usable and flexible.
The tasks and targets should look like unit tests.
Registering the targets can be done similar to registering
Cobra commands. Parsing arguments
should be leveraging the flag packages package and extensibility
could be done similar to HTTP middleware's pattern.
Now let me introduce to Goyek, let's run
it by executing Go run build.
As previously, you see the usage of the goyek.
We have the information which tasks are available, but also we have the information
which flags you can use. As usual,
let's run the old target.
When something is failing and you're not running in various mode,
you have the same experience like when using go tests.
So you just got the information that the test task
had failed and you had the output from this one task.
If you want similar experience to the make file, you just
run it in verbose mode like for go test and you have the information
of the whole execution.
Let's look at how everything is defined. The build
pipeline is defined in build folder and the
main function contains an execution of
a convenient boot main function.
The boot package contains, I hope
it's an extension of GOIC main which defines some reusable
flags and configures the flow basically in a convenient way.
So basically it is a set of commonly used middlewares
and flags which are used when you're using
Goyek for a build pipeline.
If we look at all,
it contains the definition of the
all tasks and the definition looks similar
to the definition of a cobra command. So we have a name all,
we have the usage and we have also listed the dependencies of
this task which is FMT and test.
If we look at the test definition, we can
see an action which is similar concept
to a unit test. When you have
the testing t so you here we have the definition of
the task action which executes go
test and go to cover to generate HTML
report. The CMD exec is
a convenient function for running commands
in a shell like way. When you
use CMD exec and test and
the execution will fail, it will mark
the task as failed, but it will continue the execution.
So under the hood,
under the hood it's just logging the execution
and logging and using error
to report a problem.
Reusing reusing task is simple.
We simply have a package demo
task which contains the definition of
an FMT task
which is defined as FMT usage.
Go FMT and is just executing go FMT.
One of the big benefits of using Goyek
is that intellisense is working because
it doesn't use any build tags. Also because it's
a regular go application, you can
easily debug anything. So I can put a breakpoint here,
I can press f five another
breakpoint which I put and we can see that here. Now it
has not failed because it was running fine, but if we
step it's running executing go
test, we need to wait a moment and
now we see that the task is failing. We see that
the value of the failed has changed and
probably if we looked at the output,
I just need to find
it was debug console. Yeah, we received the output here
so we can continue the execution.
So as we can see, we can debug. We have problems and we can use
intellisense during development, which is very convenient.
It was just a teaser. I hope you enjoy it and I encourage you to
check out more because Goyek is more powerful than
just that. You can add middlewares, create reusable,
customizable, build pipelines, customize creating,
integrate with viper and any package, and I encourage you to
check out the repository for documentation and see the
examples. I especially encourage you to look
at how Goyek is dog fooding its
own functionalities for its own build pipeline.
Here there's another quick example of usage.
You can see that in the output you see in
the locks. You can see the file name
and line number and it's very
helpful during debugging. And GOIC is very
useful when you have a little more, little even a little more complex
logics, like if statements for loops,
you can use a helper to mark
some functions as helpers. Then this function
will be not printed when logging and
it's really easy to debug and develop like regular boring
go code. Let me summarize the presentation.
I still use make not
only for Gil repositories, but for non go repositories as
well. For stuff like automating, setup of kubernetes, local clusters,
protobuff validation. Usually for automating simple
things I do not state make is bad, it's just complex,
maybe even some aspects more complex than go.
Moreover, I find doing complex things in make not straightforward.
Regarding mage, the good thing it requires go. It has community,
a lot of stars, and I know a lot of repositories are using it.
But it has gotchas that are annoying me and they're annoying
me so much that personally I was never using it in
production. GoiC also requires go.
It's a library instead of a framework with AK inspired by
popular go packages, and I hope it's more idiomatic.
It's simply yet extensible,
and it works very well with your Ed and I use it in production
and the behavior is similar to go test,
but and if you like it,
make a shout out. Feel free to try it,
and any feedback is more than welcome. Here are
some hyperlinks. Feel me to ask questions.
I'm open to any feedback and thank you very much for your time
and your attention. I hope you enjoyed the talk. Thank you. See you later.