ARM users
Using Delve on ARM architecture is unstable, consider debugging with another architecture.

Debugging with Delve #

This guide assumes you are already familiar with running Nakama inside a Docker container. For more information see the Installing Nakama with Docker Compose documentation.

The main.go file #

For this guide we’ll be using an example main.go file that registers a single test RPC function that we can debug using Delve.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

package main

import (
	"context"
	"database/sql"
	"fmt"

	"github.com/heroiclabs/nakama-common/runtime"
)
func InitModule(ctx context.Context, logger runtime.Logger, db *sql.DB, nk runtime.NakamaModule, initializer runtime.Initializer) error {
	logger.Debug("GO SERVER RUNTIME CODE LOADED")
	initializer.RegisterRpc("RpcTest", RpcTest)
	return nil
}

func RpcTest(ctx context.Context, logger runtime.Logger, db *sql.DB, nk runtime.NakamaModule, payload string) (string, error) {
	logger.Debug("RpcTest RPC called")

    payloadExists := false
    
    if payload != "" {
        payloadExists = true
    }

    return fmt.Sprintf("{ \"payloadExists\": %v }", payloadExists), nil
}

Creating the Dockerfile #

In order to build our server runtime code in a way that can be debugged with dlv we need to make a Dockerfile that ensures both the runtime code and the Nakama binary are built without optimizations.

Create a new Dockerfile and call it Dockerfile:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

FROM registry.heroiclabs.com/heroiclabs/nakama-pluginbuilder:3.22.0 AS builder
ENV GO111MODULE on
ENV CGO_ENABLED 1
WORKDIR /backend
COPY go.mod .
COPY vendor/ vendor/
COPY *.go .

RUN apt-get update && \
    apt-get -y upgrade && \
    apt-get install -y --no-install-recommends gcc libc6-dev

RUN go build --trimpath --gcflags "all=-N -l" --mod=vendor --buildmode=plugin -o ./backend.so
RUN go install github.com/go-delve/delve/cmd/dlv@latest

FROM registry.heroiclabs.com/heroiclabs/nakama-dsym:3.22.0

COPY --from=builder /go/bin/dlv /nakama
COPY --from=builder /backend/backend.so /nakama/data/modules/
COPY local.yml /nakama/data/

ENTRYPOINT [ "/bin/bash" ]

Some things to note here. In our builder step we install some packages that will then allow us to build our Go runtime plugin using the --gcflags "all=-N -l" flag which effectively disables optimizations in the resulting plugin file.

We also install dlv using RUN go install github.com/go-delve/delve/cmd/dlv@latest. We’ll copy this over to the final docker image in the next step.

In our final Docker step, we are using the nakama-dsym image rather than the standard nakama image. This is an image that provides us with a Nakama binary with optimizations disabled, suitable for running with dlv just like our plugin.

We then copy over the dlv binary as well as our server runtime plugin and local.yml configuration file as normal.

The other modification here is that we override the default ENTRYPOINT so that Nakama does not attempt to automatically start up. This gives us the opportunity to docker exec into the container and run dlv ourselves.

Adding the Docker Compose file #

The docker-compose.yml file we will use for debugging is similar to the standard one provided in the Installing Nakama with Docker Compose documentation, however we have made a few adjustments. Here we make sure we’re building our Dockerfile image and removing the entrypoint property on the nakama server to ensure that Nakama does not launch when running the container. We’re also setting a few values that are needed to successfully run dlv inside the container. These are the security_opt, stdin_open and tty properties.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

version: '3'
services:
  postgres:
    command: postgres -c shared_preload_libraries=pg_stat_statements -c pg_stat_statements.track=all
    environment:
      - POSTGRES_DB=nakama
      - POSTGRES_PASSWORD=localdb
    expose:
      - "8080"
      - "5432"
    image: postgres:12.2-alpine
    ports:
      - "5432:5432"
      - "8080:8080"
    volumes:
      - data:/var/lib/postgresql/data
  nakama:
    build:
        context: .
        dockerfile: Dockerfile
    depends_on:
      - postgres
    expose:
      - "7349"
      - "7350"
      - "7351"
      - "2345"
    healthcheck:
      test: ["CMD", "/nakama/nakama", "healthcheck"]
      interval: 10s
      timeout: 5s
      retries: 5
    links:
      - "postgres:db"
    ports:
      - "7349:7349"
      - "7350:7350"
      - "7351:7351"
      - "2345:2345"
    restart: unless-stopped
    security_opt:
      - "seccomp:unconfined"
    stdin_open: true
    tty: true
volumes:
  data:

Exec’ing into Docker #

With our Docker configuration files done, we will now launch our Nakama and Postgres containers:

1

docker compose up

With our containers running, open a new terminal window and exec into the Nakama docker container (where server-nakama-1 is the name of your Nakama container).

1

docker exec -it server-nakama-1 /bin/bash

This will drop you into a bash shell inside the Nakama container.

Manually migrating the Nakama database #

From here, we will first make sure we migrate the Postgres database for use with Nakama.

1

/nakama/nakama migrate up --database.address postgres:localdb@postgres:5432/nakama

Running Nakama with Delve #

With the database migrated we can now run Nakama via Delve. Note the extra -- between the path to the Nakama binary and the Nakama config flags. It is important you include this or dlv will not correctly pass those flags to the Nakama binary.

1

./dlv --log --log-output=debugger exec /nakama/nakama -- --config /nakama/data/local.yml --database.address postgres:localdb@postgres:5432/nakama

Here we’re using the dlv exec command to execute our precompiled Nakama binary and begin the debug session. You’ll be presented with the dlv command line interface:

1

(dlv)

Debugging custom server runtime code with Delve #

Delve is now waiting for user input to tell it how to proceed. At this point, if we typed continue, Delve would continue execution of Nakama and we would see the standard Nakama server output log. However, because Nakama goes into a state of waiting for a termination signal after startup is complete, we would not be able to debug our runtime code if we did this as Delve would be unable to interject.

Instead, what we must do is set a breakpoint to some point before Nakama goes into this waiting phase, but after which our custom server runtime plugin has been loaded.

To do this, let’s set a breakpoint on line 181 of main.go.

1

(dlv) break main.go:181

You should see an output similar to the following, confirming the breakpoint:

1
2

2022-06-30T12:29:48Z info layer=debugger created breakpoint: &api.Breakpoint{ID:1, Name:"", Addr:0x1f266cd, Addrs:[]uint64{0x1f266cd}, File:"github.com/heroiclabs/nakama/v3/main.go", Line:181, FunctionName:"main.main", Cond:"", HitCond:"", Tracepoint:false, TraceReturn:false, Goroutine:false, Stacktrace:0, Variables:[]string(nil), LoadArgs:(*api.LoadConfig)(nil), LoadLocals:(*api.LoadConfig)(nil), WatchExpr:"", WatchType:0x0, VerboseDescr:[]string(nil), HitCount:map[string]uint64{}, TotalHitCount:0x0, Disabled:false, UserData:interface {}(nil)}
Breakpoint 1 set at 0x1f266cd for main.main() github.com/heroiclabs./v3/main.go:181

Now, let’s continue execution of Nakama until we hit that breakpoint.

1
2

(dlv) continue
> main.main() github.com/heroiclabs./v3/main.go:181 (hits goroutine(1):1 total:1) (PC: 0x1f266cd)

At this point, our custom server runtime code should be loaded. We can confirm this by using the libraries command and looking for our plugin .so file.

1
2
3
4
5
6
7

(dlv) libraries
0. 0x7f8099127000 /lib/x86_64-linux-gnu/libdl.so.2
1. 0x7f8099106000 /lib/x86_64-linux-gnu/libpthread.so.0
2. 0x7f8098f45000 /lib/x86_64-linux-gnu/libc.so.6
3. 0x7f8099131000 /lib64/ld-linux-x86-64.so.2
4. 0x7f8062758000 /lib/x86_64-linux-gnu/libnss_files.so.2
5. 0x7f806177c000 /nakama/data/modules/backend.so

Above you can see our custom server runtime code listed as /nakama/data/modules/backend.so.

For this example, the custom server runtime has an RpcTest function which simply logs a message out to the Nakama server console and returns a JSON response with a value indicating whether a payload was given or not.

Let’s verify that the function exists:

1
2

(dlv) funcs RpcTest
heroiclabs.com/nakama-server-sandbox.RpcTest

We can set a breakpoint on this function:

1
2
3

(dlv) break RpcTest
2022-06-30T12:37:16Z info layer=debugger created breakpoint: &api.Breakpoint{ID:2, Name:"", Addr:0x7f395d54970a, Addrs:[]uint64{0x7f395d54970a}, File:"heroiclabs.com/nakama-server-sandbox/main.go", Line:14, FunctionName:"heroiclabs.com/nakama-server-sandbox.RpcTest", Cond:"", HitCond:"", Tracepoint:false, TraceReturn:false, Goroutine:false, Stacktrace:0, Variables:[]string(nil), LoadArgs:(*api.LoadConfig)(nil), LoadLocals:(*api.LoadConfig)(nil), WatchExpr:"", WatchType:0x0, VerboseDescr:[]string(nil), HitCount:map[string]uint64{}, TotalHitCount:0x0, Disabled:false, UserData:interface {}(nil)}
Breakpoint 2 set at 0x7f395d54970a for heroiclabs.com/nakama-server-sandbox.RpcTest() heroiclabs.com.-server-sandbox/main.go:14

and then continue execution:

1

(dlv) continue

Going to the Nakama console now and triggering this Rpc from a user will allow us to debug this function.

As soon as we hit the breakpoint, Delve will give us back control and allow us to inspect various things:

1
2

> heroiclabs.com/nakama-server-sandbox.RpcTest() heroiclabs.com.-server-sandbox/main.go:14 (hits goroutine(276):1 total:1) (PC: 0x7f395d54970a)
(dlv)

The first thing we can do here is inspect the arguments that have been passed to the RPC by using the args command.

1
2
3
4
5
6
7
8

(dlv) args
ctx = context.Context(*context.valueCtx) 0xbeef000000000008
logger = github.com/heroiclabs/nakama-common/runtime.Logger(*github.com/heroiclabs/nakama/v3/server.RuntimeGoLogger) 0xbeef000000000108
db = ("*database/sql.DB")(0xc0004781a0)
nk = github.com/heroiclabs/nakama-common/runtime.NakamaModule(*github.com/heroiclabs/nakama/v3/server.RuntimeGoNakamaModule) 0xbeef000000000308
payload = "{\"hello\":\"world\"}"
~r0 = ""
~r1 = error nil

We can see from above that as well as the typical RPC arguments (context, logger, database etc) we also see the value of the JSON payload passed to the RPC.

Next we can step into the executing function, see the call to the logger execute and then inspect the locals. We can do this using the next and locals commands.

1
2
3
4

(dlv) next
2022-06-30T13:18:38Z debug layer=debugger nexting
{"level":"debug","ts":"2022-06-30T13:18:38.878Z","caller":"nakama-server-sandbox/main.go:17","msg":"RpcTest RPC called","runtime":"go","rpc_id":"RpcTest"}
> heroiclabs.com/nakama-server-sandbox.RpcTest() heroiclabs.com.-server-sandbox/main.go:19 (PC: 0x7f819f58d830)

Continue execution by calling next until you hit line 21 and then inspect the local variables by using locals.

1
2
3
4
5

(dlv) next
2022-06-30T13:18:46Z debug layer=debugger nexting
> heroiclabs.com/nakama-server-sandbox.RpcTest() heroiclabs.com.-server-sandbox/main.go:21 (PC: 0x7f819f58d835)
(dlv) locals
payloadExists = false

Let’s continue execution using next and observe the value of payloadExists changing.

1
2
3
4
5

(dlv) next
2022-06-30T13:22:57Z debug layer=debugger nexting
> heroiclabs.com/nakama-server-sandbox.RpcTest() heroiclabs.com.-server-sandbox/main.go:25 (PC: 0x7f819f58d84d)
(dlv) locals
payloadExists = true

We can see here that the value of payloadExists has changed due to the if statement evaluating to true and thus changing the value of that variable.

Let’s fully continue execution and let the RPC return by calling continue.

1

(dlv) continue

You should now see the appropriate response from the RPC in your Nakama console API Explorer.

1
2
3

{
  "payloadExists": true
}

Further debugging #

The above demonstrates a very basic example of stepping through the execution of a custom server runtime RPC, inspecting the arguments and local variables and continuing execution.

Delve is a powerful debugger and if you would like to learn more about what commands are available please visit the official Delve CLI documentation.