Load test your Nakama deployment
We’ve helped a lot of studios launch their games, and we always recommend they load test before launch. What load tests give you is an informed estimate on the baseline hardware your anticipated load needs (more on that later).
We’re happy to run load tests with your team as part of the support plan. For general guidance, we’ve put together this guide covering the key steps.
Part 1. Decide what to test #
Know the shape of your load #
Game backends tend to be either more request-heavy or connection-heavy. An authoritative match, for example, runs your handler on every tick: 1000 matches at ten ticks per second is 10,000 executions every second. So the capacity for concurrent players matters.
You’ll also need to anticipate your game load in bursts. A content drop’s push notification can send a herd of players logging back in at the same time.
Set realistic targets #
Be realistic with the numbers. 100 req/s may sound low. For an authentication test, that’s 100 × 60 × 60 × 24, or 8.6 million registrations a day. In reality only the top-grossing games reach that.
Same goes for the hardware. On commodity hardware, a CPU runs 8 hyperthreads and that means 8 non-interrupted concurrent workloads at a time. If you are expecting 1000 req/s, size your hardware to your request logic accordingly.
In our experience, latency is the one teams get wrong most often. It’s tempting to benchmark the hop between Nakama and another cloud-hosted service such as GameLift, single-digit milliseconds inside a region. For a mobile player the dominant leg is device to cell tower to ISP, tens to low hundreds of milliseconds, before the request reaches any cloud network at all.
Focus on critical player paths #
Your first load test should focus on the paths that lock a player out of the game: authentication, session start, initial state sync. After those, the core game loop, covering the end-to-end player journey. This should be the loop that has the most player interactions in your game.
Pick the test types #
Run a smoke test first, with a handful of users. After that, stress and soak are the two that decide a launch. Ramp a stress test well above your expected peak until something breaks, and you get two numbers out of it: the cliff, and the safe operating point below it. A soak holds steady load for a long duration and catches what only shows up over time, like the storage object that grows every match or the connection that’s never returned.
Part 2. Run the tests and scale #
With a rough idea of what to test and what numbers to expect, the test itself becomes a search for the hardware that meets them. Start small and grow incrementally.
- Baseline on minimal hardware. Find out what one Nakama CPU and one database core actually carry.
- Scale up incrementally. Add resources and confirm throughput scales roughly linearly.
- Change one thing. Adjust code or configuration, then rerun.
Before you run #
You’ll need:
- A production-type Nakama deployment dedicated to load testing. Development environments aren’t scalable and can’t be used for load testing or any performance assessment.
- Notify our support team. Let us know which instance you’ll be testing and when, so the team can monitor and assist if needed.
- A review of your custom code. Before a significant run, our engineers can review your RPCs for performance anti-patterns. Common hotspots are external service calls, signed CDN URL generation, and validation or anti-cheat checks before wallet or state writes.
- Any add-ons you need, arranged ahead of time. Metric exporting, log exporting, and the database replica are paid add-ons, and continuous profiling also requires a commercial support plan. Block and mutex profiles are off by default, so ask for them before the run: they’re the ones that reveal lock contention, and without them you can test for a long time and never catch your worst contention.
- A load testing tool that supports WebSocket connections. We recommend Artillery for load testing Nakama, since it has first-class WebSocket support for testing Nakama’s real-time features. k6 is a documented alternative with WebSocket support.
Nakama configuration
Before running load tests, review these settings:
- JSVM pool counts: If you write JavaScript or TypeScript code, check your JSVM min and max pool counts. The default pool size is 64 (64 concurrent VM-based requests). For high-concurrency load tests, increase this value in your Nakama configuration. Each additional VM instance consumes memory, so scale your Nakama tier accordingly.
Run the test #
Scale your deployment proactively before the test begins. See Scaling for how to apply tier changes and how long they take to take effect. Leave at least 25% headroom so the deployment has capacity to absorb traffic spikes during the test.
For scaling advice specific to your load test scenario, contact us.
For complex games, we recommend splitting the test into phases and test them in order. Phase 1 is the critical paths that would lock a player out. Phase 2 is core game loop.
Watch the system as the run happens:
- Player-facing health on critical player journeys. Auth, state sync, and the core loop.
- The endpoints that error or slow down.
- Resource saturation: CPU, memory, database etc.
Utilization can look moderate while requests queue for a connection or a runtime VM. So watch the waits too. Climbing goroutine counts, database connection pool state, and block or mutex profiles.
If the run itself misbehaves, with throughput capping below the configured rate, sockets dropping, or runs finishing early, see Troubleshooting the test itself before you look at the backend.
Read the results and fix #
Perhaps the most difficult part of a load test is understanding what the results mean. It’s where most of our time with the studios goes. We can go through the results with you, point you at the tooling that gets you closer to the bottleneck, and help decide what to change before the next run. The tools below are the ones we’ll be looking at.
Diagnostic toolkit #
| Heroic Cloud tool | What it gives you in a load test | Availability |
|---|---|---|
| Graphs | Load balancer request count split by status code, Nakama CPU and memory per node, database CPU, and database query load. Export the selected range for offline analysis. | Included with every deployment |
| Top database queries | Your most resource-intensive SQL, ranked with an impact indicator. When database CPU is high, start here. | Included with every deployment |
| Logs | Full-text search, severity filtering, and date range selection across deployment output. Logs are sampled; contact us for unsampled export. | Included with every deployment |
| Nakama Console | Direct inspection of players, storage objects, leaderboards, and matches. Confirms your seeded accounts look how you intended and that a run wrote what it claimed. | Included with every deployment |
| Deployment audit | Who deployed which image, changed configuration, or triggered a reboot, with timestamps. Use it to attribute a result to the one thing you changed. | Included with every deployment |
| Data export | A full PostgreSQL snapshot you can restore locally and debug against production data volumes. | Included with every deployment |
| Metric exporting | Nakama application metrics, load balancer metrics, and your own custom module metrics, on a Prometheus endpoint you scrape into your own stack. | Paid add-on |
| Log exporting | Raw, unsampled logs shipped to an S3 bucket in near real-time, enriched with instance ID and timestamps. | Paid add-on |
| Database replica | A read-only copy of the database you can query without adding load to the primary. Confirm the replication lag with us before you correlate anything from it against a specific minute of your run. | Paid add-on |
| Continuous profiling | CPU usage, memory allocation, goroutine counts, mutex behavior, and lock contention, with differential analysis across two time frames. Covers database interactions too, surfacing the call stacks behind the queries. | Paid add-on and a commercial support plan |
See Dashboard metrics and logs for each view in detail, including what to watch on every chart.
Once you know what the run surfaced, the Symptom to fix reference maps it to the code change that fixes it.
Tune the platform #
Two tuning levers are Nakama-specific:
Runtime VM pool. The JavaScript and TypeScript runtime uses a VM pool, default 64. If profiles show requests queuing on the runtime rather than the database, raise it, at a memory cost you validate under load. Each VM instance holds its own copy of anything you cache at init, so a larger pool multiplies that memory. See Builders for the min and max pool settings. Lua runs in a sandboxed VM with the same pool sizing considerations; Go doesn’t use a pool. RPC throughput is nearly identical across all three runtimes for non-CPU-bound work, as the benchmarks show.
Scaling. Apply a tier change via Scaling between runs, not during one, since database scaling causes brief downtime. Scaling buys headroom for parallel work, so reach for it once per-request work is already lean.
Baseline performance #
We’ve stress-tested Nakama to over 2 million CCUs. Contact sales@heroiclabs.com for benchmarking reports and guidance on sizing for your specific workload.
Where to go next #
We’ve put together a guide on the design patterns and practices that hold up against load. Read the guide on Performance and scalability best practices, covering topics from storage design and caching to hooks and database access.
More information #
Tooling limits to know #
- Dashboard logs are truncated and rate-limited to manage volume. The logs are sampled. If you need unsampled logs exported, contact us.
- The metric endpoint is scraped once a minute. Lengthen your phases if you need the shape of a short spike.
- We’re working on making these add-ons self-service. Until then, contact us and we’ll enable them ahead of your run.
Troubleshooting the test itself #
When the numbers don’t make sense, check the test and the environment before the backend.
| Symptom | Likely cause | What to do |
|---|---|---|
| Throughput far below target, latency low | Running against a dev instance | Switch to a production-type instance |
| Throughput caps below the configured rate | Generator saturated or rate-limited by CPU or DNS | Split across generator nodes; keep generator CPU below roughly 70% |
| Runs finish early, generator sockets drop | Generator socket instability | Run the generator on a more stable host; confirm the applied load matched the config |
| Dependency calls time out only under load | Client timeout set too tight | Raise the client timeout and retry budget; pre-provision the dependency beforehand |
| Connection failures climbing with load | Socket, file descriptor, or ephemeral port limits on the generator, or too little headroom on the deployment | Split the generator across nodes; add CPU or memory headroom |
| RPCs queuing on the runtime rather than the database | JavaScript or Lua VM pool too small | Raise the pool count, accepting the memory cost |
Symptom to fix reference #
The fix for almost everything a load test surfaces is a code change. Performance and scalability best practices carries each of these in full, with examples.
| Symptom | Likely cause | Fix |
|---|---|---|
| High p95, many database calls per request | Per-row reads, repeated fetches down a call chain | Batch per-row calls; hoist loop-invariant calls |
| One endpoint degrades, others flat, scales with concurrency | Write contention on a shared record | Shard contended writes across keys |
| Latency flat then vertical at a concurrency point | Connection-pool exhaustion | Reduce round trips per request |
| Consistently slow at p50, modest p50-to-p95 ratio | Expensive work per call, uncached static data | Cache static data at init |
| Match stutter for all players at once | Blocking I/O in the tick loop | Keep the match loop I/O-free |
| A few heavy RPCs, unrelated calls fail under sustained load | Shared pool starved, cascading failure | Profile the heavy RPCs under sustained load and fix the handlers |
| Hot RPC slow with few database queries and a small response | Gathering analytics or event payloads inline on the hot path | Send events via the background queue; trim the data you collect |
See also #
- Scaling for tiers, costs, and how to apply scaling changes before your test.
- Nakama deployments for deployment configuration, built-in monitoring, and logging limits.
- Builders for runtime VM pool settings.
- Dashboard metrics and logs for the built-in graphs, top database queries, and log search.
- Additional add-ons for the database replica and continuous profiling.
- Performance and scalability best practices for the code changes that resolve what a test surfaces.
- Benchmarks for stock-server performance across hardware configurations.
