Search code examples
influxdb

Influxdb after sometime loses index of series


I'm running influxdb from a remote file system (CIFS) on a raspberry.

After start it works pretty good for few hours and then when I try to list the series (using show series;) I get following error:

ERR: SHOW SERIES [panic:runtime error: index out of range]

After restarting application everything works fine again for few hours. I tried adding more logging but there are no errors there or useful info entries.

I'm quite puzzled why this happens, googling returns mostly error for golang.

EDIT: Actually I found some logs for this:

lvl=error msg="SHOW SERIES [panic:runtime error: index out of range] goroutine 10617 [running]:\nruntime/debug.Stack(0x31e91d0, 0x30ecf00, 0x
b)\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x80\ngithub.com/influxdata/influxdb/query.(*Executor).recover(0x2ebfe60, 0x31e91d0, 0x33da5c0)\n\t/go/src/github.com/influxdata/influxdb/query/executor.go:3
94 +0x88\npanic(0xad41a8, 0x15fd528)\n\t/usr/local/go/src/runtime/panic.go:513 +0x194\nencoding/binary.bigEndian.Uint16(...)\n\t/usr/local/go/src/encoding/binary/binary.go:100\ngithub.com/influxdata/influxdb/
tsdb.ReadSeriesKeyMeasurement(...)\n\t/go/src/github.com/influxdata/influxdb/tsdb/series_file.go:338\ngithub.com/influxdata/influxdb/tsdb.parseSeriesKey(0x63e190ed, 0x1, 0x3fff13, 0x0, 0x0, 0x0, 0x1, 0x1, 0x3
0b7620, 0x0, ...)\n\t/go/src/github.com/influxdata/influxdb/tsdb/series_file.go:375 +0x350\ngithub.com/influxdata/influxdb/tsdb.ParseSeriesKey(0x63e190ed, 0x1, 0x3fff13, 0x30, 0x0, 0x0, 0x1ce08, 0x2cf6108, 0x
2cbb9e0)\n\t/go/src/github.com/influxdata/influxdb/tsdb/series_file.go:358 +0x40\ngithub.com/influxdata/influxdb/tsdb.(*seriesPointIterator).Next(0x2f18000, 0x9e4448, 0x9e4448, 0x2cf60d0)\n\t/go/src/github.co
m/influxdata/influxdb/tsdb/index.go:842 +0x40\ngithub.com/influxdata/influxdb/query.(*floatInterruptIterator).Next(0x348c190, 0x3988640, 0x2cbb9e0, 0x1ad84)\n\t/go/src/github.com/influxdata/influxdb/query/ite
rator.gen.go:941 +0x44\ngithub.com/influxdata/influxdb/query.(*floatFastDedupeIterator).Next(0x348c1a0, 0x39886c0, 0x30eced8, 0x4)\n\t/go/src/github.com/influxdata/influxdb/query/iterator.go:1301 +0x28\ngithu
b.com/influxdata/influxdb/query.(*bufFloatIterator).Next(0x348c1c0, 0x363ae4, 0x39886c0, 0x30eced8)\n\t/go/src/github.com/influxdata/influxdb/query/iterator.gen.go:90 +0x74\ngithub.com/influxdata/influxdb/que
ry.(*bufFloatIterator).peek(0x348c1c0, 0x2e61080, 0x1, 0x1)\n\t/go/src/github.com/influxdata/influxdb/query/iterator.gen.go:65 +0x1c\ngithub.com/influxdata/influxdb/query.(*floatIteratorScanner).Peek(0x39886a
0, 0
Apr 02 19:30:20 pi2 influxd[23263]: x2, 0x2, 0x30eced8, 0x4, 0xa8ff70, 0x2e61080, 0x1)\n\t/go/src/github.com/influxdata/influxdb/query/iterator.gen.go:516 +0x2c\ngithub.com/influxdata/influxdb/query.(*scanner
Cursor).scan(0x2d4a4b0, 0x39886c0, 0x0, 0x0, 0x3f9e74, 0x10, 0xb180c0, 0x1, 0x1ad84)\n\t/go/src/github.com/influxdata/influxdb/query/cursor.go:235 +0x28\ngithub.com/influxdata/influxdb/query.(*scannerCursor).
scan-fm(0x39886c0, 0xd53478, 0x348c130, 0x4, 0x1, 0x328a8cf, 0x18, 0x8)\n\t/go/src/github.com/influxdata/influxdb/query/cursor.go:230 +0x24\ngithub.com/influxdata/influxdb/query.(*scannerCursorBase).Scan(0x2d
4a4b8, 0x328a8d0, 0x0)\n\t/go/src/github.com/influxdata/influxdb/query/cursor.go:175 +0x38\ngithub.com/influxdata/influxdb/query.(*Emitter).Emit(0x3d78340, 0xc1e884, 0x3d78340, 0x3d78340, 0x440a0bc)\n\t/go/sr
c/github.com/influxdata/influxdb/query/emitter.go:41 +0x40\ngithub.com/influxdata/influxdb/coordinator.(*StatementExecutor).executeSelectStatement(0x2c36a40, 0x316e880, 0x2e231f0, 0x0, 0x0)\n\t/go/src/github.
com/influxdata/influxdb/coordinator/statement_executor.go:561 +0xfc\ngithub.com/influxdata/influxdb/coordinator.(*StatementExecutor).ExecuteStatement(0x2c36a40, 0xd558d8, 0x316e880, 0x2e231f0, 0x1, 0x1)\n\t/g
o/src/github.com/influxdata/influxdb/coordinator/statement_executor.go:64 +0x1f90\ngithub.com/influxdata/influxdb/query.(*Executor).executeQuery(0x2ebfe60, 0x31e91d0, 0x440a0bc, 0xa, 0x0, 0x0, 0xd561f8, 0x161
9f60, 0x2710, 0x0, ...)\n\t/go/src/github.com/influxdata/influxdb/query/executor.go:334 +0xb50\ncreated by github.com/influxdata/influxdb/query.(*Executor).ExecuteQuery\n\t/go/src/github.com/influxdata/influx
db/query/executor.go:236 +0x6c\n" log_id=0EZUK5qG000 service=query

but they don't help much :(


Solution

  • CIFS is a problem. Probably InfluxDB files are corrupted - see https://github.com/influxdata/influxdb/issues/10309

    InfluxDB isn't supported on network file systems (e.g. NFS, GlusterFS).