Build your golang project using go mod vs bazel

11 min readDec 13, 2024

Let’s understand the golang build history so that we can appreciate the new ways to build your golang code.

Build your simple golang code

If you have a single file golang code without any dependencies then you can build and run this code as below.

#main.go
package main

import "fmt"

func main() {
  fmt.Println("Hello, Go!")
}

To build and run

go build main.go
./main

Write code to import local package

In Go, you don’t import files directly. Instead, you import packages, which are collections of Go files in the same directory that are compiled together. All .go files in a single directory should belong to the same package.

import path is a string that uniquely identifies a package. It’s like the address you use to find a package. Now let’s code as below.

go-tutorial
├── fortune
│   └── helper.go
└── main.go

// go-tutorial/fortune/helper.go
package fortune

import "math/rand"

var fortunes = []string{
    "Your build will complete quickly.",
    "Your dependencies will be free of bugs.",
    "Your tests will pass.",
}

func Get() string {
    return fortunes[rand.Intn(len(fortunes))]
}

// go-tutorial/main.go
package main

import (
    "fmt"
    "go-tutorial/fortune"
)

func main() {
    fmt.Println(fortune.Get())
}

History to use go modules

Go 1.11 (February 2018): Go modules were introduced as an experimental feature. The GO111MODULE environment variable was used to enable them.
Go 1.13 (September 2019): GO111MODULE=auto became the default. This meant modules were used automatically if a go.mod file was present in the project or any parent directory. Otherwise, GOPATH was used.
Go 1.16 (February 2021): GO111MODULE=on became the default, making modules the standard build system.

Build without using go mod

First thing you need to do is turn off go module as below.

$ go env -w GO111MODULE=off

What is $GOPATH env variable?

It specifies the location of your Go workspace. This is where your Go projects, downloaded packages, and compiled binaries are stored.

The $GOPATH directory typically contains three subdirectories:

src: Contains the source code of your Go projects and external packages.
pkg: Stores compiled package objects (.a files) to speed up builds.
bin: Holds compiled executable binaries.

Now you need to move your code inside $PATH/src/ as below structure

$PATH/src/go-tutorial
├── fortune
│   └── helper.go
└── main.go

Now run following command

$ cd $PATH/src/go-tutorial
$ go install .

This will generate executable file inside $PATH/bin/go-tutorial which you can run it.

Build using go module

With go module, you don’t strictly need to use $PATH variable. But you still need it for local package. Let’s make sure to use following folder structure.

$PATH/src/go-tutorial
├── fortune
│   └── helper.go
└── main.go

Following are steps.

$ export GOPATH=/workspace
$ cd $PATH/src/go-tutorial
$ go mod init
$ go build

This generates executable in same directory.

Do I still need to define GOPATH with go module?

The short answer is: No, you don’t strictly need to set GOPATH when using Go modules. Go modules manage dependencies directly within your project, using the go.mod file and the module cache. This means Go no longer relies on the GOPATH to find packages.

GOPATH has other purposes: GOPATH still serves some purposes, such as:

Storing your Go source code (although you can now store it anywhere)
Holding the module cache ($GOPATH/pkg/mod)
Containing the bin directory for executables ($GOPATH/bin)

Build using go module without $PATH

To use go module without $PATH, we need to come up with unique module name first. For example, we can run following command.

$ cd ~go-tutorial
$ go mod init github.com/your-github-username/go-tutorial
$ go build

This will generate following go.mod file.

module github.com/your-github-username/go-tutorial

go 1.18

Please note; now main.go needs to be updated to use the package as below.

// go-tutorial/main.go
package main

import (
    "fmt"
    "github.com/your-github-username/helloworld/fortune"
)

func main() {
    fmt.Println(fortune.Get())
}

How to add new module as dependency?

Modify your code to use the dependency. Let’s update code as below.

// main.go
package main

import "fmt"

import "rsc.io/quote"

func main() {
    fmt.Println(quote.Go())
}

To update go.mod we need to run following command.

$ go mod tidy

It will download and update go.mod as below.

golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c h1:qgOY6WgZOaTkIIMiVjBQcw93ERBE4m30iBm00nkL0i8=
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
rsc.io/quote v1.5.2 h1:w5fcysjrx7yqtD/aO+QwRjYZOKnaM9Uh2b40tElTs3Y=
rsc.io/quote v1.5.2/go.mod h1:LzX7hefJvL54yjefDEDHNONDjII0t9xZLPXsUe+TKr0=
rsc.io/sampler v1.3.0 h1:7uVkIFmeBqHfdjD+gZwtXXI+RODJ2Wc4O7MPEh/QiW4=
rsc.io/sampler v1.3.0/go.mod h1:T1hPZKmBbMNahiBKFy5HrXp6adAjACjK9JXDnKaTXpA=

You can also run following command to download the dependency first.

$ go get rsc.io/quote

Use of go.sum file

The go.sum file is a crucial part of Go's module system, designed to ensure the integrity of your project's dependencies. You don't typically generate it manually; instead, Go handles its creation and updates automatically. Here's a breakdown of how it works:

Understanding go.sum

Checksums:

The go.sum file contains cryptographic checksums of your project's dependencies. These checksums act as fingerprints, verifying that the downloaded dependencies haven't been tampered with.

Integrity:

By checking these checksums, Go ensures that you’re using the exact same versions of dependencies that were intended, preventing potential security risks.

How go.sum is Generated and Updated

go mod init: When you initialize a new Go module using go mod init <module_path>, Go creates a go.mod file.
go get: When you add or update dependencies using go get <package_path>, Go downloads the necessary packages and automatically updates both go.mod and go.sum.
go mod tidy: Updates the go.mod and go.sum files to reflect the current state of your project's dependencies.
Automatic Updates: Whenever you build, test, or run your Go project, Go automatically checks the checksums in go.sum to ensure their validity. If any discrepancies are found, Go will report an error.

Use of vendor folder

The vendor folder in a Go project is used to store copies of the external packages (dependencies) that your project relies on. It's essentially a local cache of your dependencies.

When you use go get or go build with Go modules enabled, Go typically downloads the necessary dependencies and stores them in the module cache ($GOPATH/pkg/mod).

However, if a vendor folder exists in your project's root directory, Go will prioritize using the packages from that folder instead of the cache.

Managing the vendor folder:

go mod vendor: This command populates the vendor folder with the dependencies listed in your go.mod file.
go mod tidy: This command cleans up your go.mod file and the vendor folder, removing any unused dependencies.

Note: There is more cons to use vendor folder and polluting your code repository compare to benefits.

Bazel : Build a Go Project

First thing we need to do is to install bazel binary as per https://bazel.build/install. Following is quick command for macbook.

$ brew install bazel

Do I need to install go?

You don’t need to install Go to build Go projects with Bazel. The Bazel Go rule set automatically downloads and uses a Go toolchain instead of using the toolchain installed on your machine. This ensures all developers on a project build with same version of Go.

Let’s write following go code.

// hello.go
package main

import "fmt"

func main() {
    fmt.Println("Hello, Bazel! 💚")
}

Now build this file using bazel we need to write following BUILD file.

// BUILD
load("@rules_go//go:def.bzl", "go_binary")

go_binary(
    name = "hello",
    srcs = ["hello.go"],
)

We also need to add MODULE.bazel file as below. Our MODULE.bazel file contains a single dependency on rules_go, the Go rule set. We need this dependency because Bazel doesn't have built-in support for Go.

bazel_dep(
    name = "rules_go",
    version = "0.50.1",
)

Finally, MODULE.bazel.lock is a file generated by Bazel that contains hashes and other metadata about our dependencies. It includes implicit dependencies added by Bazel itself, so it's quite long. Just like go.sum, you should commit your MODULE.bazel.lock file to source control to ensure everyone on your project gets the same version of each dependency. You shouldn't need to edit MODULE.bazel.lock manually.

Then run following command to build the code.

$ cd go-tutorial
$ bazel build //:hello

It will produce binary at bazel-bin/hello_/hello which you can execute it to test it.

You can also run following command to run code directly.

$ bazel run //:hello

Please note; it will generate symlinks folders as well. These symlinks act as shortcuts to important output directories, making it easier to navigate and access build artifacts. Instead of traversing deep into the Bazel output structure, you can simply use these conveniently placed links.

Here’s a breakdown of the common symlinks and what they point to:

bazel-bin: This is probably the most frequently used symlink. It points to the directory where Bazel places executable binaries and libraries built by your project.
bazel-out: This symlink provides access to the general output directory for Bazel. It contains various subdirectories holding different types of build outputs, such as intermediate files, test logs, and more.
bazel-testlogs: As the name suggests, this symlink leads you to the directory where Bazel stores logs generated during test execution. This is helpful for debugging and analyzing test results.
bazel-go-tutorial: This symlink seems specific to your project ("go-tutorial"). Bazel often creates symlinks related to external dependencies or workspaces. This particular one might point to the location where Bazel has fetched and stored the "go-tutorial" project's files.

Bazel : With local package

This program uses a separate Go package as a library that selects a fortune from a predefined list of messages.

go-tutorial
├── BUILD
├── MODULE.bazel
├── MODULE.bazel.lock
├── fortune
│   ├── BUILD
│   └── fortune.go
└── print_fortune.go

Following is code for fortune.go file.

package fortune

import "math/rand"

var fortunes = []string{
    "Your build will complete quickly.",
    "Your dependencies will be free of bugs.",
    "Your tests will pass.",
}

func Get() string {
    return fortunes[rand.Intn(len(fortunes))]
}

The fortune directory has its own BUILD file that tells Bazel how to build this package. We use go_library here instead of go_binary.

# BUILD
load("@rules_go//go:def.bzl", "go_library")

go_library(
    name = "fortune",
    srcs = ["fortune.go"],
    importpath = "github.com/bazelbuild/examples/go-tutorial/stage2/fortune",
    visibility = ["//visibility:public"],
)

Note: We also need to set the importpath attribute to a string with which the library can be imported into other Go source files. This name should be the repository path (or module path) concatenated with the directory within the repository.

We can run following command to build fortune folder.

$ cd go-tutorial
$ bazel build //fortune

Please note; you need to run build command from the root folder i.e. go-tutorial here.

Next, see how print_fortune.go uses this package.

package main

import (
    "fmt"

    "github.com/bazelbuild/examples/go-tutorial/stage2/fortune"
)

func main() {
    fmt.Println(fortune.Get())
}

Note: print_fortune.go imports the package using the same string declared in the importpath attribute of the fortune library.

We also need to declare this dependency to Bazel. Here’s the BUILD file in the same directory.

load("@rules_go//go:def.bzl", "go_binary")

go_binary(
    name = "print_fortune",
    srcs = ["print_fortune.go"],
    deps = ["//fortune"],
)

Run following command.

$ bazel run //:print_fortune

Understanding the bazel labels

A label is a string Bazel uses to identify a target or a file. Labels are used in command line arguments and in BUILD file attributes like deps. We've seen a few already, like //fortune, //:print-fortune, and @rules_go//go:def.bzl.

A label has three parts: a repository name, a package name, and a target (or file) name.

The repository name is written between @ and // and is used to refer to a target from a different Bazel module (for historical reasons, module and repository are sometimes used synonymously). In the label, @rules_go//go:def.bzl, the repository name is rules_go. The repository name can be omitted when referring to targets in the same repository.

The package name is written between // and : and is used to refer to a target in from a different Bazel package. In the label @rules_go//go:def.bzl, the package name is go.

Most Go projects have one BUILD file per directory and one Go package per BUILD file. The package name in a label may be omitted when referring to targets in the same directory.

The target name is written after : and refers to a target within a package. The target name may be omitted if it's the same as the last component of the package name (so //a/b/c:c is the same as //a/b/c; //fortune:fortune is the same as //fortune).

On the command-line, you can use ... as a wildcard to refer to all the targets within a package. This is useful for building or testing all the targets in a repository.

# Build everything
$ bazel build //...

Write test using bazel

Let’s use following diretory structure.

go-tutorial
├── BUILD
├── MODULE.bazel
├── MODULE.bazel.lock
├── fortune
│   ├── BUILD
│   ├── fortune.go
│   └── fortune_test.go
└── print-fortune.go

fortune/fortune_test.go is our new test source file.

package fortune

import (
    "slices"
    "testing"
)

// TestGet checks that Get returns one of the strings from fortunes.
func TestGet(t *testing.T) {
    msg := Get()
    if i := slices.Index(fortunes, msg); i < 0 {
        t.Errorf("Get returned %q, not one the expected messages", msg)
    }
}

This file uses the unexported fortunes variable, so it needs to be compiled into the same Go package as fortune.go. Look at the BUILD file to see how that works:

load("@rules_go//go:def.bzl", "go_library", "go_test")

go_library(
    name = "fortune",
    srcs = ["fortune.go"],
    importpath = "github.com/bazelbuild/examples/go-tutorial/stage3/fortune",
    visibility = ["//visibility:public"],
)

go_test(
    name = "fortune_test",
    srcs = ["fortune_test.go"],
    embed = [":fortune"],
)

Run following command to test.

$ bazel test //fortune:fortune_test

You can use the ... wildcard to run all tests. Bazel will also build targets that aren't tests, so this can catch compile errors even in packages that don't have tests.

$ bazel test //...

Go standard folder structure

While Go doesn’t enforce a strict directory structure, the community has converged on some common patterns that promote organization and maintainability. Here’s a breakdown of a typical Go project layout:

Root Directory

go.mod: This file defines your module's path and its dependencies. It's essential for managing external packages.

Common Top-Level Directories

cmd/: Holds the main applications or executables of your project. Each subdirectory within cmd typically represents a separate application with its own main.go entry point. Example: cmd/my-app/main.go, cmd/my-tool/main.go
internal/: Contains packages intended for internal use only within your project. Go's compiler enforces this, preventing other projects from importing packages within internal/. This helps with encapsulation and avoids creating accidental public APIs. Example: internal/database/, internal/config/
pkg/: Houses reusable packages that can be shared with other projects. These are designed to be imported as external libraries. Example: pkg/my-library/, pkg/utils/
api/: (Optional) Often used to store API definitions, such as Protocol Buffer files (.proto) or OpenAPI specifications.

Other Common Directories

web/: For web application-related code (controllers, templates, static assets).
scripts/: Contains scripts for building, testing, or deploying your project.
configs/: Holds configuration files.
tests/: Can store integration or end-to-end tests.

bazel and monorepo

Bazel and monorepos are a powerful combination! Here’s why Bazel is particularly well-suited for managing monorepos and how it excels in this context:

What is a Monorepo?

A monorepo is a software development strategy where code for many projects or components is stored in a single repository. This contrasts with the more traditional multirepo approach where each project has its own separate repository.

Why Bazel Shines in Monorepos

Built-in Support: Bazel is designed with monorepos in mind. Its core concepts like workspaces and targets naturally lend themselves to managing multiple projects within a single repository.
Scalability: Bazel handles large codebases with ease. Its efficient dependency analysis and incremental builds ensure that only the necessary parts of your monorepo are rebuilt when changes are made. This is crucial for maintaining fast build times in massive projects.
Code Sharing and Reusability: Bazel makes it simple to share code and dependencies across different projects within the monorepo. This promotes consistency and reduces code duplication.
Dependency Management: Bazel’s WORKSPACE file and dependency management rules provide a centralized and consistent way to manage external dependencies for all projects in the monorepo.
Hermetic Builds: Bazel’s focus on hermeticity ensures that builds are reproducible and independent of the environment. This is especially important in monorepos where different projects might have conflicting dependencies.
Advanced Build Features: Bazel offers advanced features like remote caching and execution, which can significantly speed up build times in a monorepo environment, especially for distributed teams.

Key Bazel Concepts for Monorepos

Workspace: The root directory of your monorepo, containing the WORKSPACE file that defines external dependencies and Bazel settings.
Targets: Individual build units within your monorepo, such as libraries, binaries, or tests.
BUILD files: These files, located throughout your monorepo, define targets and their dependencies using Bazel’s build language.

Benefits of Using Bazel with Monorepos

Improved code sharing and reuse
Simplified dependency management
Faster build times
Increased consistency and reliability
Better scalability
Enhanced collaboration among teams

If you’re considering adopting a monorepo approach or looking for a robust build system for your existing monorepo, Bazel is definitely worth exploring. Its features and performance make it a great choice for managing complex projects with multiple interdependencies.

Happy learning golang :-)