Storage Search

The Storage Engine supports configurable indices that automatically index the content of storage objects written through the storageWrite API.

Only a subset of the storage object value is indexed as configured. Search these indices using the query syntax .

Indices work best for finding cohorts of entities across many users, such as matching players for offline matchmaking or pulling a segment of accounts that share a trait. They aren’t built for reading back a single player’s own records. When you need every one of a player’s objects, complete and current, list those objects directly from the Storage Engine instead. See List objects for that approach.

Nakama populates the configured indices at startup and holds the maximum number of configured entries plus a threshold. Once that threshold is surpassed, it automatically evicts the oldest entries to favor new or recently updated entries. Nakama doesn’t reload evicted entries into the index.

Consistency and scale #

The search system built on the Storage Engine is eventually consistent. Nakama populates indices asynchronously, so a write may take a few milliseconds to appear in query results.

In a cluster, this means a query can return stale hits: entries that matched an earlier state of the index. By default, Nakama reads each matching object from the Storage Engine, so returned values are current even when a hit is stale. Index-only mode skips that read and returns values straight from the index.

An index also has a size limit that matters at scale. It holds at most maxEntries records. Once a collection grows past that, some objects never enter the index, and a query can’t return what the index doesn’t hold. Treat indices as bounded search structures, not a complete view of a collection.

Creating a new index #

Create and configure indices through the runtimes .

Index names must be unique. Each index is tied to a collection, and you must specify the top-level keys of the object fields to index. If an object has none of those keys, it isn’t indexed.

Set key to index only storage objects with that specific key. Omit it (or pass an empty string) to index all object keys that match the collection and fields.

Set a maximum entry count for the index.

The optional sortableFields parameter specifies which indexed fields support custom sort ordering when listing results. These must be a subset of fields.

Multiple indices can share a collection as long as their names are unique. Nakama automatically indexes any objects that match an index’s configuration on storage write.

1
2
3
4
5
6
7
8
9

name := "IndexName"
collection := "CollectionName"
key := "" // Set to empty string to match all keys instead
fields := []string{"field1", "field2"} // Only objects containing any of these keys and respective values are indexed.
sortableFields := []string{"field1"} // Must be a subset of fields. Leave empty if no custom sort ordering is needed.
maxEntries := 1000
indexOnly := false

err := initializer.RegisterStorageIndex(name, collection, key, fields, sortableFields, maxEntries, indexOnly)

1
2
3
4
5
6
7
8
9

local name = "IndexName"
local collection = "CollectionName"
local key = "KeyName" -- Set to empty string to match all keys instead
local fields = {"field1", "field2"} -- Only objects containing any of these keys and respective values are indexed.
local sortable_fields = {"field1"} -- Must be a subset of fields. Leave empty if no custom sort ordering is needed.
local max_entries = 1000
local index_only = false

local err = nk.register_storage_index(name, collection, key, fields, sortable_fields, max_entries, index_only)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

const name = "IndexName";
const collection = "CollectionName";
const key = "KeyName"; // Set to empty string to match all keys instead
const fields = ["field1", "field2"]; // Only objects containing any of these keys and respective values are indexed.
const sortableFields = ["field1"]; // Must be a subset of fields. Leave empty if no custom sort ordering is needed.
const maxEntries = 1000;
const indexOnly = false;

try {
    initializer.registerStorageIndex(name, collection, key, fields, sortableFields, maxEntries, indexOnly);
} catch(err) {
    // Handle error
}

When to use an index #

Reach for an index when you need to search across many users’ objects at once. Reach for a direct object list when you need one owner’s objects returned completely and in real time.

If you’re reading back data a player owns and expects to see in full, list it directly. Save the index for the queries only an index can answer.

Index-only mode #

By default, when a query runs against an index, Nakama first resolves the matching index entries and then fetches the corresponding storage objects from the database. This ensures returned values are always complete and up-to-date.

When indexOnly is set to true at registration time, the list operation returns values directly from the index without reading from the database. This reduces database reads and can improve performance, but has two drawbacks. The index only stores the fields specified at registration, so returned objects may contain partial data. Fields not included in fields are absent from the response. The index is also eventually consistent, so the returned values themselves may be stale. Use this mode only when both tradeoffs are acceptable for your use case.

Listing from an index #

Query the index using the powerful query syntax that also powers the Nakama matchmaker .

To filter by indexed field values, prefix the field key with value..

1
2
3
4
5
6
7
8

name := "index_name"
query := "+value.field1:1 value.field2:foo"
limit := 10
order := []string{} // optional, see Sorting results below
cursor := "" // empty string for first page

objects, nextCursor, err := nk.StorageIndexList(ctx, "", name, query, limit, order, cursor)
// Pass nextCursor as cursor in the next call to retrieve the following page.

1
2
3
4
5
6
7
8

local name = "index_name"
local query = "+value.field1:1 value.field2:foo"
local limit = 10
local order = {} -- optional, see Sorting results below
local cursor = nil -- nil for first page

local objects, next_cursor = nk.storage_index_list(name, query, limit, order, nil, cursor)
-- Pass next_cursor as cursor in the next call to retrieve the following page.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

const name = "index_name";
const query = "+value.field1:1 value.field2:foo";
const limit = 10;

try {
    let result = nk.storageIndexList(name, query, limit);
    let objects = result.objects;
    let nextCursor = result.cursor; // null when there are no further results
    // Pass nextCursor as the cursor argument to retrieve the following page.
} catch(err) {
    // Handle error
}

Each index entry also includes these queryable values automatically, in addition to the configured fields:

• update_time
• key
• user_id
• version
• create_time
• read
• write
• collection

These fields don’t need the value. prefix in queries.

Sorting results #

Use the optional order parameter to sort results by storage object fields. The prefix - before a field name indicates descending order. Declare the fields you want to sort by in sortableFields when registering the index.

1
2
3

order := []string{"value.field_name", "-value.other_field"}

objects, nextCursor, err := nk.StorageIndexList(ctx, "", "<name>", "<query>", 10, order, "")

1
2
3

local order = {"value.field_name", "-value.other_field"}

local objects, next_cursor = nk.storage_index_list("<name>", "<query>", 10, order)

1
2
3
4
5
6
7
8

const order = ["value.field_name", "-value.other_field"];

try {
    let result = nk.storageIndexList("<name>", "<query>", 10, order);
    let objects = result.objects;
} catch(err) {
    // Handle error
}

To sort results by the value of any of the sortable fields, their respective keys need to be prefixed with value..

Register a storage index filter #

Register a custom filtering function per index. This function is called for each storage object eligible for the index and must return a boolean value.

Return true to index the object. Return false to exclude it from the index and delete any previously indexed entry for it.

The registered function is also applied to all eligible entries when the index is populated.

1
2
3
4
5
6
7

name := "index_name"
fn := func(ctx context.Context, logger Logger, db *sql.DB, nk NakamaModule, write *StorageWrite) bool {
    // Inspect object to decide if it should be inserted or tentatively deleted.
    return true
}

err := initializer.RegisterStorageIndexFilter(name, fn);

1
2
3
4
5
6
7

local name = "index_name"
function indexFilter(context, logger, nk, storageWrite)
    -- Inspect object to decide if it should be inserted or tentatively deleted.
    return true
end

local err = nk.register_storage_index_filter(name, indexFilter)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

const name = "index_name";

let filterFn: nkruntime.StorageIndexFilterFunction = function (ctx: nkruntime.Context, logger: nkruntime.Logger, nk: nkruntime.Nakama, storageWrite: StorageWriteRequest): boolean {
      // Inspect object to decide if it should be inserted or tentatively deleted.
      return true;
}

try {
    initializer.RegisterStorageIndexFilter(name, filterFn);
} catch(err) {
    // handle error
}

Related Pages

• Collections
• Access Controls
• Modeling for Scalability