Go! Run! Go!

When I started learning Golang I’ve read several tutorials – especially how to build a go program. When you already worked with go, you know how easy it is to compile a program. Just run go build

The tutorial also mentions, that a golang program can be directly executed with go run main.go

As you may know a golang program is compiled to machine code; go run can be seen as an short-hand for go build && ./main . At this time I just thought “Meh. A little toy.”. And forgot about it.

Some time later I had to do some scripting in the context of gitlab ci. Especially I had to call a rest api with some json and as always I’ve put some curl calls into a bash script. Let’s have look at the excerpt from the script:

function startTask {
	templateId=$1
	taskOutput=$(curl -s -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' --header 'Authorization: Bearer '$token -d "{\"template_id\": $templateId, \"debug\": false, \"dry_run\": false, \"playbook\": \"\", \"environment\": \"\"}" $node/api/project/1/tasks)
	taskId=$(echo $taskOutput | jq ".id" | sed 's/\"//g')
	echo $taskId
}
...
function waitTaskFinished {
  id=$1
	while true; 
	do 
      
	  x=$(curl -s -X GET --header 'Accept: application/json' --header 'Authorization: Bearer '$token $node "/api/project/1/tasks/$id" | jq ".status")
	  if [[ "$x" == "\"success\"" ]]; then 
		  break; 
	  fi; 
    if [[ "$x" == "\"error\"" ]]; then
      exit 1
    fi;
	  sleep 5
	done
}


id=$(startTask 2)
waitTaskFinished $id
...

Puh – maybe there’s an easier way to escape these json strings in Bash – especially the startTask function. Maybe not – don’t care it’s just a script. With bash scripts I often encounter kind of limit where the simplicity of it falls apart and gets really annoying, time consuming to change it or extend it. For me personally this point is often reached when integrating some if else logic or including some variables in longer strings – e.g. json.

So I rewrote it into a little golang program. As obvious for the project I had to add a binary file to the git repository. However, checking in a binary smells “funny”. Also setting up an additional repository for a small scripting task feels not right. Do I have to setup ci for this? How do I keep the binary the repository in sync with the other repository? Then I put some thought about scripting in general.

General thoughts about scripting

Scripts can be seen as an extension of a standalone program and embedded in a bigger context. They are “attached” to these programs or services.

What I mean is that they often are used for maintenance like cleaning up, archiving, integration with other systems or moving and transforming data from a to b – typical “glue code”. Often you can’t or even don’t want to change the underlying programs. In my mind they cannot run by itself. It’s important to note, that this is just a model in my mind. I know, technically you can write a script that can be run without any externally dependency but often they are used for the above mentioned use cases. So therefor it’s a good idea to place them into the repository of the application where they are used for the maintenance task or the integration.

In my case, the goal is to add some CI capabilities to an ansible playbook. What’s really nice about scripts is, they can be checked in with the regular source code of the program and can be executed(with some runtime) easily within the context of the program and can be especially useful for maintenance or (gitlab) ci.

Our current main language for services or programs is currently golang because it compiles to native maschine code and cross compiling for arm devices is really easy – we currently use a lot of raspis in the project. For my pains regarding bash scripts for getting at some point really dirty I remembered go run. Maybe we can use instead of clunky bash scripts golang in combination with go run. With some tricks you can also use it with she-bang(see https://golangcookbook.com/chapters/running/shebang/). For me – that’s interesting, but it’s only a nice addition.

Gitlab CI with go run

As previously stated I’ve used to call a compiled golang program. Now I moved the source code of the program to the repository – it’s easy because it’s just one file 🙂 Previously the relevant section looked like the following:

test-on-templates:
  image: cfmanteiga/alpine-bash-curl-jq
  stage: integration-test
  script: ./update $CI_COMMIT_BRANCH 

Replacing the call to execute the golang program with
go run update.go
will have the same effect without the drawback checking in a binary file and the problem with keeping the binary in sync. Because we’re “compiling” inside the docker container we also have to use another image with golang toolchain included.

```
test-on-templates:
  image: golang:1.14
  stage: integration-test
  script: 
    - go run update.go $CI_COMMIT_BRANCH
```

Commit it and push it to gitlab. But we’re not quite there. This will work for golang programs which use functionalities provided by the std library. But for sake of simplicity with rest apis I’ve used resty(see https://github.com/go-resty/resty). Running it will result in a dependency error. You can fix that with:

test-on-templates:
  image: golang:1.14
  stage: integration-test
  script: 
    - go get -d ./...
    - go run update.go $CI_COMMIT_BRANCH


go get -d
means that it will stop after fetching it’ dependencies. This is especially useful if you have multiple golang programs with multiple main functions. A simple “`go get“` will then fail with the error that multiple main entry points have been found.

Pros and cons

Now let’s evaluate what are the pros and cons using golang instead of a bash script.

During development even in small context I found it at the beginning a little bit odd having to “script” in a typed language. But I would put it on the pro side because I’ve avoided several errors like missing parameters or providing the wrong datatypes. It adds a little bit of more “noise” to the source and compared with a bash script a golang program is more verbose regarding types, conversion, loops etc.

In my opinion error handling is a little bit more easier with golang – even there are no exceptions . Golang relies on the err variable technique. This is similar to bash scripts but the IDE warns you if you do not handle an error returned by a function and after a while coding with golang it becomes a habit handling your errors right.

I’ve used for the developing goland and it has nice features like auto-completion, auto-importing, refactoring tooling or code navigation. Maybe there are also very good ides for bash scripts but til this day haven’t found one. Maybe you know one 🙂

As I’ve written above, working with if conditions are not my favorite part of bash scripts – so let’s see how they compare:

...
  if [[ "$x" == "\"success\"" ]]; then 
    echo "succeeded"
  fi; 
...

For me, I can read the golang version a lot more easily than the bash script implementation. This may also is a kind of habit.

One thing I really miss is the pipe operator. To my knowlede there’s no equivalent. Maybe this is also a good idea because you’re not tempted to use a lot of sed and awk in combination with piping. What both languages miss is map reduce which can be found in languages like javascript. But maybe with golang 1.17 and generics we finally we’re going to get these functions. Personally I think map reduce can be used to build very readable transformation functions.

One important drawback should be mentioned. At the beginning of this article I’ve mentioned that go run can be seen as an alias for go build && ./program. This means every time you’re running this “script”, the program will be compiled first. This can result in longer startup times – although some things may be cached. Keep that in mind and always: measure.

What is also nice about golang in this context external dependencies can be used – but the way to do it feels a little bit hacky that a go.mod is needed – combined with a
go get -d

Ah if you like – batteries for test and debugging are also included!

One topic I ignored intentionally is calling other commands like ls, grep or external programs. This would be a good topic for another blog post 🙂

Closing Thoughts

Unused variables or imports will result in a compile error – when using plain bash scripts not. Linting is maybe a way to mitigate this problem – but it’s already builtin in golang and there’s no need for additional tooling.

Maybe we should also start to treat our script similar to our core source code. Often we treat them as second class citizens and let them rot until nobody really understand them anymore. Even worse, they are an integral part of this application – maybe during an CI/CD run or for regular maintenance.

It could be a good idea to use a language that supports some aspects of clean(er) code than a bash script 🙂