Notes on converting a repository to use Go Modules

Updated 5/16/19: Added a note to the Incremental Conversion section about using specific revs of a repository instead of replace directives.

Go modules is the new dependency management system for Go language based projects. I just spent the last three days converting the Groups.io codebase to use Go modules. It’s a large code base, and we had vendored ~75 dependencies over the past five years of development. The code base pre-dates any of the pre-existing dependency management systems, so we were flying a bit blind when it came to which version of each dependency we were using. There are several more comprehensive guides to converting to Go modules; this is just a set of my notes. I may update them as I get more experience with modules.

Incremental Conversion

I wanted to do the conversion incrementally, to give me more opportunity to test things and make sure I wasn’t breaking the build. What I did was create the empty go.mod file, and then within it, I added replace directives for each dependency we had vendored. The replace directive pointed into the vendor directory, like this:

github.com/jackc/pgx => ./vendor/github.com/jackc/pgx

Once I had done that, I started running go build. That would complain about missing go.mod files in each of the dependent directories. I would dutifully go in and create these missing go.mod files one by one. I continued doing this until there were no more errors.

NOTE: It’s been pointed out, here, that instead of adding a bunch of replace directives, I could have instead used the specific revision of the repository and go modules would handle it correctly. I didn’t know about this capability when I did the conversion.

Dependency Review

Now that I had a working build, I started the review process. I went through each dependency and, by seeing when we vendored it, I could review the upstream changes. For dependencies that had no changes I was uncomfortable with, I would then remove the replace directive in the go.mod file, letting Go manage that dependency.

Local Changes And Deleted Repos

We have made changes to several of these dependencies (that are not appropriate for upstream pull-requests), and we need to keep those changes. Also, a few dependences that we had vendored no longer exist, because their creators have deleted the repositories. I created a new top-level directory, internal. I then moved those dependencies from vendor into internal, and updated the relevant replace directive to point into internal.

Keeping A /vendor Directory

After all that work, we had a go.mod file with a large number of require directives, and about 14 replace directives. I removed the existing vendor directory from the repository. A downside quickly became apparent, however. Within Sublime Text, like most editors, I can hover over a type or function call and Sublime will list the various files where it may have been defined. This is very helpful during development. But without a vendor directory, Sublime couldn’t locate any of the moduled dependencies. It can be handy to have all the source code on hand/available to the editor when doing development. To generate a /vender directory from your go.mod file, use the command go mod vendor. To then use the /vendor directory for builds, add the parameter -mod=vendor when building.

Another advantage of keeping a /vendor directory is if you run the godoc tool locally. If you include the query parameter ?m=all when accessing the godoc webpage, it will include the /vendor directory (as well as the /internal directory, if you have one) when generating documentation.

go install gotcha

With go modules, the GOPATH environment variable is no longer needed. If you run go install without a GOPATH, it apparently does nothing, as it doesn’t know where to put the generated binary. To get go install to work again, I had to set the GOBIN environment variable.