My system is
% uname -or
FreeBSD 11.0-RELEASE-p2
Sphinx version is
% searchd --help
Sphinx 2.2.11-id64-release (95ae9a6)
Sphinx configuration:
index content_rt_template : common_template
{
type = rt
rt_mem_limit = 128M # 128M only...
rt_field = text
rt_attr_string = text
rt_field = title
rt_attr_string = title
rt_field = url
rt_attr_string = url
rt_attr_bigint = item_id
rt_attr_uint = source_id
rt_attr_timestamp = published_date
rt_attr_timestamp = created_date
}
common {
lemmatizer_base = /path/to/sphinx/
}
indexer
{
mem_limit = 128M # 128M only...
}
index content_rt_from_20170501_to_20170601 : content_rt_template
{
path = /path/to/sphinx/data/2017/content_rt_from_20170501_to_20170601
}
index content_rt_from_20170601_to_20170701 : content_rt_template
{
path = /path/to/sphinx/data/2017/content_rt_from_20170601_to_20170701
}
index content_rt_from_20171201_to_20180101 : content_rt_template
{
path = /path/to/sphinx/data/2017/content_rt_from_20171201_to_20180101
}
index content2017
{
type = distributed
local = content_rt_from_20170501_to_20170601
local = content_rt_from_20170601_to_20170701
local = content_rt_from_20171201_to_20180101
}
searchd
{
listen = 127.0.0.1:9417
listen = 9317:mysql41
log = /path/to/sphinx/log/searchd_2017.log
query_log = /path/to/sphinx/log/query_2017.log
read_timeout = 60
max_children = 30
pid_file = /path/to/sphinx/pid/searchd2017.pid
seamless_rotate = 0
preopen_indexes = 0
unlink_old = 1
workers = threads # for RT to work
binlog_path = /path/to/sphinx/data/2017/
}
Used memory before starting Sphinx:
Mem[||||||||| 5.33G/40.0G]
Swp[||||||||||||||||||||||||||||||3.35G/4.00G]
Log on Sphinx start:
% ./start.sh
Sphinx 2.2.11-id64-release (95ae9a6)
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
using config file '/path/to/sphinx/conf/content2017.conf'...
listening on 127.0.0.1:9417
listening on all interfaces, port=9317
WARNING: index 'common_template': key 'path' not found - NOT SERVING
WARNING: index 'content_rt_template': path must be specified - NOT SERVING
WARNING: failed to init process shared rwlock: process shared rwlock is not supported by FreeBSD; ALTER disabled
precaching index 'content_rt_from_20170501_to_20170601'
WARNING: failed to init process shared rwlock: process shared rwlock is not supported by FreeBSD; ALTER disabled
precaching index 'content_rt_from_20170601_to_20170701'
WARNING: failed to init process shared rwlock: process shared rwlock is not supported by FreeBSD; ALTER disabled
precaching index 'content_rt_from_20171201_to_20180101'
precached 3 indexes in 6.520 sec
Sphinx 2.2.11-id64-release (95ae9a6)
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
using config file '/path/to/sphinx/conf/content_dist.conf'...
listening on 127.0.0.1:9312
listening on all interfaces, port=9306
WARNING: index 'common_template': key 'path' not found - NOT SERVING
WARNING: index 'content_rt_template': path must be specified - NOT SERVING
WARNING: failed to init process shared rwlock: process shared rwlock is not supported by FreeBSD; ALTER disabled
precaching index 'content_snippet'
precached 1 indexes in 0.064 sec
Used memory after Sphinx started:
Mem[||||||||||||||| 11.6G/40.0G]
Swp[||||||||||||||||||||||||||||||3.34G/4.00G]
Sphinx use about 6G. But according to mem_limit and rt_mem_limit must use not more than 128 * 3 = 384M.
What may be reason of using lot of memory?
Maybe reason in this warning?
WARNING: failed to init process shared rwlock: process shared rwlock
UPD
I has tryed it on Ubuntu 16.04 and situation is same.
The rt_mem_limit only limits the size of the ram chunk itself. Any disk chunks will use their own memory.
Typically its attributes that compose the biggest part , as by default, all are held in memory. Can cut down memory with http://sphinxsearch.com/docs/current.html#conf-ondisk-attrs