Class - mitto.iov2.store.rocksdb.RocksDbParams

JSON Schema

RocksDbParams

Configuration parameters for rocksdb store. Except for extremely rare situations, it should not be necessary to override any of the default values for these parameters – i.e., there is probably no need to understand these parameters and even less need to include any of them in your job configuration file. The parameters can be placed in a job configuration in one of two locations: the store section within the input section, used in conjunction with StoreInput Example store section: { "store": { ... "params": { "rocksdb": { "log_stats": 1000, "opts": { "block_based_table_factory": { "block_cache_size": 500000, "block_cache_compressed_size": 250000, }, }, }, ... } Example input section: { "input": { "use": "mitto.iov2.input#StoreInput", "params": { "rocksdb": { "log_stats": 1000, "opts": { "block_based_table_factory": { "block_cache_size": 500000, "block_cache_compressed_size": 250000, }, }, }, ... }
type	object
properties
chunk_size	Chunk Size
	Number of records to buffer prior to writing to disk. If 0, all records are buffered. Setting this to a non-zero value can reduce the amount of memory required by jobs inputting large amounts of data.
	type	integer
	examples	10000
	minimum	0
	default	0
log_stats	Log Stats
	Log rocksdb and process memory statistics every log_stats records. If 0, logging is not performed.
	type	integer
	examples	1000
	minimum	0
	default	0
opts	Opts
	default	create_if_missing	True
		max_open_files	0
		write_buffer_size	6291456
		max_write_buffer_number	3
		target_file_size_base	6291456
		max_background_flushes	4
		block_based_table_factory	bloom_filter_policy	10
			block_cache_size	524288000
			block_cache_compressed_size	262144000
	allOf	Opts
definitions
BlockBasedTableFactory	BlockBasedTableFactory
	Keyword args for rocksdb.BlockBasedTableFactory(). For context, the values are used in the following manner: table_factory = rocksdb.BlockBasedTableFactory( filter_policy=rocksdb.BloomFilterPolicy(bloom_filter_policy), block_cache=rocksdb.LRUCache(block_cache_size), block_cache_compressed=rocksdb.LRUCache(block_cache_compressed_size), ) These should not be changed from their defaults unless you know what you are doing. See the BlockBasedTableFactory documentation for more information.
	type	object
	properties
	bloom_filter_policy	Bloom Filter Policy
		Bits per key.
		type	integer
		minimum	0
		default	10
	block_cache_size	Block Cache Size
		Cache size in bytes.
		type	integer
		minimum	0
		default	524288000
	block_cache_compressed_size	Block Cache Compressed Size
		Cache size in bytes.
		type	integer
		minimum	0
		default	262144000
Opts	Opts
	Options passed to rocksdb when store is opened. For context, the values are used in the following manner: overridden_options = {"max_open_files": 1024, "create_if_missing": False} all_options = Opts(**overridden_options) rocksdb_options = rocksdb.Options(**all_options.dict()) These should not be changed from their defaults unless you know what you are doing. See rocksdb.Options for more information.
	type	object
	properties
	create_if_missing	Create If Missing
		type	boolean
		default	True
	max_open_files	Max Open Files
		If 0, the value is computed dynamically by Mitto. Otherwise, the value is used as the argument value.
		type	integer
		minimum	-1
		default	0
	write_buffer_size	Write Buffer Size
		type	integer
		minimum	1
		default	6291456
	max_write_buffer_number	Max Write Buffer Number
		type	integer
		minimum	1
		default	3
	target_file_size_base	Target File Size Base
		type	integer
		minimum	1
		default	6291456
	max_background_flushes	Max Background Flushes
		type	integer
		minimum	1
		default	4
	block_based_table_factory	Block Based Table Factory
		default	bloom_filter_policy	10
			block_cache_size	524288000
			block_cache_compressed_size	262144000
		allOf	BlockBasedTableFactory

JSON Schema Definitions

#/definitions/BlockBasedTableFactory

BlockBasedTableFactory

Keyword args for rocksdb.BlockBasedTableFactory(). For context, the values are used in the following manner: table_factory = rocksdb.BlockBasedTableFactory( filter_policy=rocksdb.BloomFilterPolicy(bloom_filter_policy), block_cache=rocksdb.LRUCache(block_cache_size), block_cache_compressed=rocksdb.LRUCache(block_cache_compressed_size), ) These should not be changed from their defaults unless you know what you are doing. See the BlockBasedTableFactory documentation for more information.
type	object
properties
bloom_filter_policy	Bloom Filter Policy
	Bits per key.
	type	integer
	minimum	0
	default	10
block_cache_size	Block Cache Size
	Cache size in bytes.
	type	integer
	minimum	0
	default	524288000
block_cache_compressed_size	Block Cache Compressed Size
	Cache size in bytes.
	type	integer
	minimum	0
	default	262144000

#/definitions/Opts

Opts

Options passed to rocksdb when store is opened. For context, the values are used in the following manner: overridden_options = {"max_open_files": 1024, "create_if_missing": False} all_options = Opts(**overridden_options) rocksdb_options = rocksdb.Options(**all_options.dict()) These should not be changed from their defaults unless you know what you are doing. See rocksdb.Options for more information.
type	object
properties
create_if_missing	Create If Missing
	type	boolean
	default	True
max_open_files	Max Open Files
	If 0, the value is computed dynamically by Mitto. Otherwise, the value is used as the argument value.
	type	integer
	minimum	-1
	default	0
write_buffer_size	Write Buffer Size
	type	integer
	minimum	1
	default	6291456
max_write_buffer_number	Max Write Buffer Number
	type	integer
	minimum	1
	default	3
target_file_size_base	Target File Size Base
	type	integer
	minimum	1
	default	6291456
max_background_flushes	Max Background Flushes
	type	integer
	minimum	1
	default	4
block_based_table_factory	Block Based Table Factory
	default	bloom_filter_policy	10
		block_cache_size	524288000
		block_cache_compressed_size	262144000
	allOf	BlockBasedTableFactory