Storage Search #

The storage engine supports configurable indices that automatically index the content of storage objects written through the storageWrite API.

Only a subset of the storage object value is indexed as configured. These indices can then be searched using the query syntax.

The configured indices are populated at startup and only hold the maximum number of configured entries plus a threshold that, once surpassed, triggers the automatic eviction of the oldest entries.

Although possible, these indices are not meant to hold indexed representations of all objects in a given collection, but instead, enable complex queries to lookup cohorts of users or entities that enable different use-cases, such as offline matchmaking.

Creating a new index #

The indices are created and configured through the runtimes.

Index names must be unique. An index is always tied to a collection, and the top level keys of the fields of the object to index have to be specified. If no field with such key is found, the object is not indexed.

The key can be set to only index storage object with the given key, omitting this parameter leads all object keys (that match the collection and fields) to be indexed.

A maximum size is also required to be set.

As long as the names are unique, multiple indices can be created per collection, and any objects that match its configuration will be automatically indexed on storage write.

1
2
3
4
5
6
7
8

name := "IndexName"
collection := "CollectionName"
key := "" // Set to empty string to match all keys instead
fields := []string{"field1", "field2"} // Only objects containing any of these keys and respective values are indexed.
maxEntries := 1000
indexOnly := false

err := initializer.RegisterStorageIndex(name, collection, key, fields, maxEntries, indexOnly)

1
2
3
4
5
6
7
8

local name = "IndexName"
local collection = "CollectionName"
local key = "KeyName" -- Set to empty string to match all keys instead
local fields = {"field1", "field2"} -- Only objects containing any of these keys and respective values are indexed.
local max_entries = 1000
local index_only = false

local err = nk.register_storage_index(name, collection, key, fields, max_entries, index_only)

1
2
3
4
5
6
7
8

const name = "IndexName";
const collection = "CollectionName";
const key = "KeyName"; // Set to empty string to match all keys instead
const fields = ["field1", "field2"]; // Only objects containing any of these keys and respective values are indexed.
const maxEntries = 1000;
const indexOnly = false;

const err = initializer.registerStorageIndex(name, collection, key, fields, maxEntries, indexOnly);

Listing from an index #

The indices can be queried using the powerful query syntax that also powers the Nakama matchmaker.

To filter results by the value of any of the indexed fields, their respective keys need to be prefixed with value..

1
2
3
4
5

name := "index_name"
query := "+value.field1:1 value.field2:foo"
limit := 10

err := nk.StorageIndexList(name, query, limit)

1
2
3
4
5

local name = "index_name"
local query = "+value.field1:1 value.field2:foo"
local limit = 10

local err = nk.storage_index_list(name, query, limit)

1
2
3
4
5
6
7
8
9

const name = "index_name";
const query = "+value.field1:1 value.field2:foo";
const limit = 10;

try {
    let objects = nk.storageIndexList(name, query, limit);
} (catch err) {
    // Handle error
}

Besides the configured index fields, there are a number of queryable values that are automatically included for each index entry. These fields are:

• update_time
• key
• user_id
• version
• create_time
• read
• write
• collection

To query these fields, they do not need to be prefixed by value. in the query.

Sorting results #

You can use the optional order parameter to sort storage object fields. The prefix ‘-’ before a field name indicates descending order. All specified fields must be indexed and sortable.

1
2
3

order := []string{"value.field_name", "-value.other_field"}

err := nk.StorageIndexList("<name>", "<query>", 10, order)

1
2
3

local order = {"value.field_name", "-value.other_field"}

local err = nk.storage_index_list("<name>", "<query>", 10, order)

1
2
3
4
5
6
7

const order = ["value.field_name", "-value.other_field"];

try {
    let objects = nk.storageIndexList("<name>", "<query>", 10, order);
} (catch err) {
    // Handle error
}

To sort results by the value of any of the sortable fields, their respective keys need to be prefixed with value..

RegisterStorageIndexFilter #

It is possible to register a custom filtering function per index. This function will be called for each storage object that is eligible for the index, and has to return a boolean value.

If true is returned, the object will be indexed. If false is returned the object will not be indexed and will be deleted (if previously indexed).

The registered function is also applied to all eligible entries when the index is populated.

1
2
3
4
5
6
7

name := "index_name"
fn := func(ctx context.Context, logger Logger, db *sql.DB, nk NakamaModule, write *StorageWrite) bool {
    // Inspect object to decide if it should be inserted or tentatively deleted.
    return true
}

err := initializer.RegisterStorageIndexFilter(name, fn);

1
2
3
4
5
6
7

local name = "index_name"
function indexFilter(context, logger, nk, storageWrite)
    -- Inspect object to decide if it should be inserted or tentatively deleted.
    return true
end

local err = nk.register_storage_index_filter(name, indexFilter)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

const name = "index_name";

let filterFn: nkruntime.StorageIndexFilterFunction = function (ctx: nkruntime.Context, logger: nkruntime.Logger, nk: nkruntime.Nakama, storageWrite: StorageWriteRequest): boolean {
      // Inspect object to decide if it should be inserted or tentatively deleted.
      return true;
}

try {
    initializer.RegisterStorageIndexFilter(name, filterFn);
} (catch err) {
    // handle error
}

Related Pages

• Collections
• Access Controls
• Modeling for Scalability