We are experiencing a problem involving missing records using a secondary key. A set namely user contains the following bins: name, email, password, role and private_token- with a secondary key on the bin private_token.
Regarding the issue: As of the previous week, four nodes in Google Cloud. The secondary key of the user set was working fine. During the end of last week, we created a new node and now this specific set stopped returning any data when querying it using a secondary key. Regarding all the other set having a similar configuration, they kept returning the correct values.
The namespace n1 of the user set is using a non local SSD storage.
In order to troubleshoot the issue, we created a new namespace, namely test, that is stored in memory. Here is a script that fails in the n1 namespace but succeed in the test namespace:
INSERT INTO test.user (PK, name, email, password, role, private_token) VALUES ('test@test.com', 'Test 1', 'test@test.com', 'password_test_1', 'role_test_1', 'private_token_1')
INSERT INTO test.user (PK, name, email, password, role, private_token) VALUES ('test2@test.com', 'Test 2', 'test2@test.com', 'password_test_2', 'role_test_2', 'private_token_2')
CREATE INDEX user_private_token ON test.user(private_token) STRING
SELECT * FROM test.user WHERE private_token = 'private_token_2'
Here is the output in test namespace:
+----------+------------------+-------------------+---------------+-------------------+
| name | email | password | role | private_token |
+----------+------------------+-------------------+---------------+-------------------+
| "Test 2" | "test2@test.com" | "password_test_2" | "role_test_2" | "private_token_2" |
+----------+------------------+-------------------+---------------+-------------------+
1 row in set (0.001 secs)
And here in n1 namespace:
0 rows in set (0.001 secs)
Version of aerospike:
$ dpkg -l | grep aerospike
ii aerospike-amc-community 3.6.4 all AMC Community Version is a web UI based monitoring tool for Aerospike Community Edition Server releases after 2.6.3 (March 2013)
ii aerospike-server-community 3.6.4-1 amd64 The Aerospike distributed datastore allows fully scalable and reliable data storage with elastic server properties.
ii aerospike-tools 3.6.3 amd64 Aerospike server tools.
Trying to troubleshoot in asadm:
$ asadm
Aerospike Interactive Shell, version 0.0.13
Found 6 nodes
Online: 10.240.0.3:3000, 10.240.0.4:3000, 10.240.0.5:3000, 10.240.0.2:3000, 10.240.0.6:3000
Offline: 10.240.0.9:3000
Admin> info network
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Network Information~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Node Node Fqdn Ip Client Current HB HB
. Id . . Conns Time Self Foreign
10.240.0.9 000000000000000 10.240.0.9:3000 10.240.0.9:3000 N/E N/E N/E N/E
aerospike-vm-1 BB90300F00A0142 aerospike-vm-1.c.citiservi-ads.internal:3000 10.240.0.3:3000 38 193339236 0 249014840
aerospike-vm-2 BB90500F00A0142 aerospike-vm-2.c.citiservi-ads.internal:3000 10.240.0.5:3000 37 193339236 0 194991137
aerospike-vm-3 BB90400F00A0142 aerospike-vm-3.c.citiservi-ads.internal:3000 10.240.0.4:3000 34 193339236 0 229395269
aerospike-vm-4 BB90200F00A0142 aerospike-vm-4.c.citiservi-ads.internal:3000 10.240.0.2:3000 45 193339236 0 224245394
aerospike-vm-5 *BB90600F00A0142 aerospike-vm-5.c.citiservi-ads.internal:3000 10.240.0.6:3000 37 193339236 0 17162917
Number of rows: 6
Admin> info service
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Service Information~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Node Build Cluster Cluster Cluster Free Free Migrates Principal Objects Uptime
. . Size Visibility Integrity Disk% Mem% (tx,rx,q) . . .
10.240.0.9 N/E N/E N/E N/E N/E N/E N/E N/E N/E N/E
aerospike-vm-1 3.6.4 5 True True 61 68 (0,0,0) aerospike-vm-5 58.618 M 1710:56:06
aerospike-vm-2 3.6.4 5 True True 61 69 (0,0,0) aerospike-vm-5 57.594 M 1472:30:21
aerospike-vm-3 3.6.4 5 True True 61 68 (0,0,0) aerospike-vm-5 57.430 M 1638:33:39
aerospike-vm-4 3.6.4 5 True True 61 68 (0,0,0) aerospike-vm-5 58.242 M 1824:25:55
aerospike-vm-5 3.6.4 5 True True 61 71 (0,0,0) aerospike-vm-5 57.774 M 102:48:57
Number of rows: 6
Admin> info sindex
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/usr/bin/asadm/__main__.py", line 20, in <module>
File "/usr/bin/asadm/asadm.py", line 339, in main
File "/usr/lib/python2.7/cmd.py", line 141, in cmdloop
line = self.precmd(line)
File "/usr/bin/asadm/asadm.py", line 108, in precmd
File "/usr/bin/asadm/lib/controllerlib.py", line 176, in execute
File "/usr/bin/asadm/lib/controllerlib.py", line 120, in __call__
File "/usr/bin/asadm/lib/controllerlib.py", line 176, in execute
File "/usr/bin/asadm/lib/controller.py", line 158, in do_sindex
TypeError: string indices must be integers, not str
How can I fix the set and have a working index again?
Thanks in advance
Unlikely to still be an issue, but the Managing Indexes article of the Operations manual has information on how to repair an index. There has also been quite a lot of work done on secondary indexes since the version 3.6 you where quoting.