In-Memory Storage Engine for Self-Managed Deployments

Specify In-Memory Storage Engine

To select the in-memory storage engine, specify:

• inMemory for the --storageEngine option, or the storage.engine setting if using a configuration file.

• --dbpath, or storage.dbPath if using a configuration file. Although the in-memory storage engine does not write data to the filesystem, it maintains in the --dbpath small metadata files and diagnostic data as well temporary files for building large indexes.

For example, from the command line:

mongod --storageEngine inMemory --dbpath <path>

Or, if using the YAML configuration file format:

storage:
   engine: inMemory
   dbPath: <path>

See inMemory Options for configuration options specific to this storage engine. Most mongod configuration options are available for use with in-memory storage engine except for those options that are related to data persistence, such as journaling or encryption at rest configuration.

Transaction (Read and Write) Concurrency

Starting in version 7.0, MongoDB uses a default algorithm to dynamically adjust the maximum number of concurrent storage engine transactions (read and write tickets). The dynamic concurrent storage engine transaction algorithm optimizes database throughput during cluster overload. The maximum number of concurrent storage engine transactions (read and write tickets) never exceeds 128 read tickets and 128 write tickets and may differ across nodes in a cluster. The maximum number of read tickets and write tickets within a single node are always equal.

To specify a maximum number of read and write transactions (read and write tickets) that the dynamic maximum can not exceed, use storageEngineConcurrentReadTransactions and storageEngineConcurrentWriteTransactions.

If you want to disable the dynamic concurrent storage engine transactions algorithm, file a support request to work with a MongoDB Technical Services Engineer.

Document Level Concurrency

The in-memory storage engine uses document-level concurrency control for write operations. As a result, multiple clients can modify different documents of a collection at the same time.

Memory Use

In-memory storage engine requires that all its data (including indexes, oplog if mongod instance is part of a replica set, etc.) must fit into the specified --inMemorySizeGB command-line option or storage.inMemory.engineConfig.inMemorySizeGB setting in the YAML configuration file.

By default, the in-memory storage engine uses 50% of physical RAM minus 1 GB.

If a write operation would cause the data to exceed the specified memory size, MongoDB returns with the error:

"WT_CACHE_FULL: operation would overflow cache"

To specify a new size, use the storage.inMemory.engineConfig.inMemorySizeGB setting in the YAML configuration file format:

storage:
   engine: inMemory
   dbPath: <path>
   inMemory:
      engineConfig:
         inMemorySizeGB: <newSize>

Or use the command-line option --inMemorySizeGB:

mongod --storageEngine inMemory --dbpath <path> --inMemorySizeGB <newSize>

Durability

The in-memory storage engine is non-persistent and does not write data to a persistent storage. Non-persisted data includes application data and system data, such as users, permissions, indexes, replica set configuration, sharded cluster configuration, etc.

As such, the concept of journal or waiting for data to become durable does not apply to the in-memory storage engine.

If any voting member of a replica set uses the in-memory storage engine, you must set writeConcernMajorityJournalDefault to false.

With writeConcernMajorityJournalDefault set to false, MongoDB does not wait for w: "majority" writes to be written to the on-disk journal before acknowledging the writes. As such, "majority" write operations could possibly roll back in the event of a transient loss (e.g. crash and restart) of a majority of nodes in a given replica set.

Write operations that specify a write concern journaled are acknowledged immediately. When an mongod instance shuts down, either as result of the shutdown command or due to a system error, recovery of in-memory data is impossible.

Transactions

Transactions are supported on replica sets and sharded clusters where:

• the primary uses the WiredTiger storage engine, and

• the secondary members use either the WiredTiger storage engine or the in-memory storage engines.

Deployment Architectures

In addition to running as standalones, mongod instances that use in-memory storage engine can run as part of a replica set or part of a sharded cluster.

sh.addShardTag("shardC", "inmem")

sh.addShardTag("shardA", "persisted")
sh.addShardTag("shardB", "persisted")

sh.addTagRange("test.analytics", { shardKey: MinKey }, { shardKey: MaxKey }, "inmem")

sh.addTagRange("salesdb.orders", { shardKey: MinKey }, { shardKey: MaxKey }, "persisted")