- ARM Users - Using Delve on ARM architecture is unstable, consider debugging with another architecture.
- Nakama Version - The code examples set the Nakama version in one place, the
NAKAMA_VERSIONbuild argument. You can use any version of Nakama on your own environment. You can find all Nakama releases from here.
Debugging with Delve
Delve is a debugger for Go. With it you can set breakpoints in your custom server runtime code, stop the server when they are hit, and inspect arguments, local variables and the call stack while Nakama is paused.
This guide walks through building a debug stack and using it, step by step. It covers breaking on your own Go runtime code and, for Hiro projects, breaking inside Hiro itself. We’ll cover working from the Delve CLI as well as your own IDE.
This guide assumes you are already familiar with running Nakama inside a Docker container. For more information see the Installing Nakama with Docker Compose documentation.
Choose how you will attach #
The debug stack runs Delve as a headless server inside the container, listening on port 2345. You can connect to Delve through Delve CLI or your own IDE.
- Delve CLI Nothing to configure, and useful when you want to inspect something quickly or work over SSH.
- An IDE, covered in Attaching from an IDE. Breakpoints in the gutter, variables in a panel, and stepping with the keyboard.
Creating the debug stack #
This guide builds a debug stack that runs alongside your normal one. It is a second Dockerfile and a second Compose file, so your usual docker compose up is untouched and you switch to debugging only when you want to.
Dockerfile.dsymbuilds everything without optimizations and puts thedlvbinary in the image.docker-compose-dsym.ymlruns that image with Delve as the entrypoint, listening on port 2345.
The Dockerfile #
The main difference from your normal Dockerfile is that nothing in it is optimized.
- Your runtime plugin is built with
--gcflags "all=-N -l", which disables optimization and inlining. - Nakama comes from the
nakama-dsymimage rather than the standardnakamaimage. It is the same release, built unoptimized with debug symbols retained.
The other difference is that dlv is built into the image, so the debugger runs inside the container alongside the process it is debugging.
Create a file called Dockerfile.dsym:
| |
A few things to note.
dlv is installed before your source is copied in, so editing your code does not rebuild Delve on every image build.
The dlv binary is copied into the final image so Delve runs inside the container, next to the process it is debugging.
The all= in --gcflags "all=-N -l" applies to every package in the build, including the standard library. The next section explains why.
The Compose file #
Create docker-compose-dsym.yml:
| |
The entrypoint runs the database migration and then hands the Nakama binary to Delve, which starts it under the debugger. Note the -- between the Nakama binary and the Nakama config flags. Without it, Delve consumes those flags instead of passing them through.
--accept-multiclient lets a client detach without terminating Nakama.
security_opt, stdin_open and tty are all required for Delve to run under Docker.
"127.0.0.1:2345:2345" rather than "2345:2345" keeps it reachable from your machine only.If you are using Hiro #
Every Hiro release ships debug variants of the binaries, named hiro-dsym*.bin. Copy them into your lib/ directory alongside the standard ones.
Nakama selects the Hiro binary by a fixed filename, so the debug builds have to be placed under the names it looks for. Add these lines to the end of Dockerfile.dsym:
| |
Breakpoints then work inside Hiro as well as in your own code, with one limitation. See What you can see inside Hiro.
Starting the debug stack #
| |
Check that Delve is listening and waiting:
| |
A line reading Adding target followed by the Nakama command confirms Delve has loaded the process but not started it.
Installing Delve locally #
The Delve running in the container is the debug server. To drive it from the CLI you need a dlv client on your own machine:
| |
This installs to $(go env GOPATH)/bin. If dlv is not found afterwards, that directory is not on your PATH:
| |
IDE users need this too. The IDE launches a local dlv as its adapter and talks to the container through it.
A debugging session from the Delve CLI #
The example below uses a runtime plugin that registers a single test RPC:
| |
Connecting #
Run this from your project root. Delve records source paths relative to the build directory, so connecting from the root is what lets it find your files:
| |
You will get the Delve prompt:
(dlv)
Breaking on startup code #
Nakama has not started yet, so your plugin is not loaded and its functions do not exist in the process. Setting a breakpoint on one anyway is the right move:
(dlv) break main.InitModule
Command failed: location "main.InitModule" not found
Set a suspended breakpoint (Delve will try to set this breakpoint when a plugin is loaded) [Y/n]? Y
Breakpoint 1 set at 0x0 for :0
Answer Y. Delve holds the breakpoint and applies it the moment your plugin loads. Now start Nakama:
(dlv) continue
Breakpoint 1 materialized at ./main.go:18 (heroiclabs.com/nakama-server-sandbox.InitModule)
Execution stops inside your own InitModule, and Delve prints the surrounding source with an arrow on the current line.
To find a function, search for its bare name. Delve returns the full name, qualified by your module path:
(dlv) funcs RpcTest
heroiclabs.com/nakama-server-sandbox.RpcTest
Breaking on an RPC #
Set a breakpoint on the RPC and let Nakama finish starting:
(dlv) break heroiclabs.com/nakama-server-sandbox.RpcTest
(dlv) continue
The prompt will not return. That is correct, Delve is running the process and waiting for the next stop event.
From another terminal, authenticate and call the RPC:
| |
The request will hang. That is the signal that you are stopped at the breakpoint, since the server cannot respond while it is paused.
Inspecting #
Back in Delve, args shows what the RPC was called with, including the payload:
(dlv) args
ctx = context.Context(*context.valueCtx) 0xbeef000000000008
logger = github.com/heroiclabs/nakama-common/runtime.Logger(...) 0xbeef000000000108
db = ("*database/sql.DB")(0x4154d9886340)
nk = github.com/heroiclabs/nakama-common/runtime.NakamaModule(...) 0xbeef0000000308
payload = "{\"hello\":\"world\"}"
~r0 = "trace_id"
~r1 = (unreadable could not resolve interface type)
~r0 and ~r1 are the function’s return values. The function has not returned yet, so they hold whatever was in those slots beforehand. Ignore them until you step to the end.
You are stopped on the function’s opening line, before any local has been declared, so locals has nothing to show yet:
(dlv) locals
(no locals)
Step forward with next and the locals appear:
(dlv) next
(dlv) next
(dlv) locals
payloadExists = false
(dlv) next
(dlv) locals
payloadExists = true
stack shows the full call chain, from Nakama’s request handling into your code.
Let the RPC return:
(dlv) continue
The curl in your other terminal completes:
| |
What you can see inside Hiro #
Hiro’s implementation is private, but it is fully described to the debugger. You can set breakpoints on Hiro functions, including unexported ones, and inspect arguments, locals and the call stack:
(dlv) break github.com/heroiclabs/hiro-gdk/server.(*AchievementsSystem).GetAchievements
Breakpoint 3 set at 0xffff3e24e120 for ...GetAchievements() github.com/heroiclabs/hiro-gdk/server/achievements.go:1156
(dlv) continue
> [Breakpoint 3] ...GetAchievements() ...achievements.go:1156 (hits goroutine(853):1 total:1)
(dlv) args
a = ("*github.com/heroiclabs/hiro-gdk/server.AchievementsSystem")(0x4154d96c1280)
ctx = context.Context(*context.valueCtx) 0xbeef000000000108
logger = github.com/heroiclabs/nakama-common/runtime.Logger(...) 0xbeef000000000208
nk = github.com/heroiclabs/nakama-common/runtime.NakamaModule(...) 0xbeef000000000308
userID = "a363f68b-cf32-490b-98b2-1164ae8b606e"
What you cannot do is read the source so list fails.
(dlv) list
Command failed: open github.com/heroiclabs/hiro-gdk/server/achievements.go: no such file or directory
To break on a Hiro function you need the RPC id that reaches it. Hiro registers each RPC under its enum name, so the call that fetches achievements is RPC_ID_ACHIEVEMENTS_GET. The full list is in hiro.proto in the hiro module.
Leaving a session #
Press Ctrl-C at any time to halt the running process and get the prompt back.
quit ends your session. To stop the stack itself:
| |
Attaching from an IDE #
Any IDE with remote Go debugging can attach to the same Delve server on 127.0.0.1:2345. The stack does not change, and you still need a local dlv, which the IDE launches as its adapter.
VS Code #
Create .vscode/launch.json in your project:
| |
Set to to your module name as it appears in go.mod. The build uses --trimpath, which replaces absolute paths with module paths, so without this rule VS Code cannot find your source.
Add a rule for each dependency you want to step into, most specific first, with the version taken from go.mod:
| |
Set your breakpoints before you attach. VS Code sends them on attach, and Delve holds them until your plugin loads, at which point they bind and hit.
InitModule. If you want to set a breakpoint in startup code, set it before attaching VS Code to the debugger. RPC breakpoints can be added at any time, because the code runs again on the next request.GoLand #
Create a Go Remote run configuration pointing at 127.0.0.1:2345.
Open your project at its module root. GoLand resolves the trimpath module paths from there on its own, so no path mapping configuration is needed.
Set a breakpoint in your RPC, then attach and call the RPC. It stops with the payload and locals in the Variables panel.
InitModule never binds even when you set it before attaching. Use the Delve CLI if you need to debug startup code.Note on Nakama startup #
The dlv exec command in the nakama entrypoint in docker-compose-dsym.yml has no --continue flag, which means Delve loads Nakama and then holds it, unstarted, until a debugger attaches and resumes it.
This is deliberate. It is the only way to break on code that runs during startup, including your own InitModule.
The side effect is that port 7350 stays silent until you attach. A request before then returns a connection reset, because Docker accepts the connection and nothing inside the container answers:
curl: (56) Recv failure: Connection reset by peer
That is the expected state.
Further debugging #
The above is a basic walkthrough. Delve is a powerful debugger, and the full command set is covered in the official Delve CLI documentation.
