Search code examples
pythonredisnosqlmlopsmlrun

Issue, iteration cross KV (NoSql/Redis) Target

I would like to iterated cross KV (NoSql,Redis) Target in MLRun without specification of primary key(s) also, but I saw only ability to get specific KV item(s) based on specific key(s). I have to use only unique items, not duplicit items.

I chose NoSql/Redis Target because key operations in-memory are critical for quick responce (based on specific key), but from time to time I have to iterate cross whole collection of keys (and it is case of this question).

You can see part of my code for getting values based on keys:

import mlrun 
import mlrun.feature_store as fs
...

svc=fs.get_online_feature_service(fs.FeatureVector("myVector", ["client.*"]))
resp = svc.get([{"cuid": 1926, "key1": 100023}])

Do you know, how to iterate cross items in KV (NoSql, Redis) target in MLRun CE/Paid editions (version 1.2.1)?

Solution

  • I am using these solutions for iteration cross keys in KV storage:

    1. NoSqlTarget

    Iteration via v3io (it is not pure part of MLRun API, but it is part of MLRun distribution packages). More information about v3io python SDK see and iteration cross KV items (cursor usage) see sample

    import v3io.dataplane

v3io_client = v3io.dataplane.Client(endpoint='https://v3io-webapi:8081', access_key='some_access_key')

# create a query, and use an items cursor to iterate the results
items_cursor = v3io_client.kv.new_cursor(container='users',
                                         table_path='/user-profile',
                                         attribute_names=['income'],
                                         filter_expression='income > 150')

# print the output
for item in items_cursor.all():
    print(item)

    BTW: NoSqlTarget is available only for MLRun Enterprise edition

    2. RedisTarget

    You can use easy iteration cross KV items, it is part of Redis API

    import redis

r = redis.StrictRedis(host='localhost', port=6379, db=0)
for key in r.keys('*'):
    r.delete(key)

    It is possible to use commandline also via redis-cli see sample:

    redis-cli keys users*

    or remove from redis specific keys based on list of keys:

    redis-cli keys users* | xargs redis-cli del

    BTW: RedisTarget is available for MLRun CE and Enterprise editions