Search code examples
amazon-web-servicesopensearchamazon-opensearch

Does AWS Opensearch Support ML Commons Library?

I'm trying to run a simple code example to train and predict a ML algorithm using the Opensearch ML Commons plugin on a AWS managed Opensearch cluster. The cluster is on OS version 2.5, which definitely supports the plugin. The code example can be found here .

When I run the sample code in my console I get the following error message:

"type": "m_l_resource_not_found_exception",
    "reason": "No eligible node found to execute this request. It's best practice to provision ML nodes to serve your models. You can disable this setting to serve the model on your data node for development purposes by disabling the \"plugins.ml_commons.only_run_on_ml_node\" configuration using the _cluster/setting api"

However after trying to update the settings I get _cluster/settings payload not allowed, it doesn't seem like the AWS OS version supports the cluster/settings API except for these

/_cluster/settings for several properties:

action.auto_create_index

action.search.shard_count.limit

indices.breaker.fielddata.limit

indices.breaker.request.limit

indices.breaker.total.limit

cluster.max_shards_per_node

source

A few questions:

  1. Am I correct in my assumption that I can't update this cluster setting, meaning that I can't run ML Commons on an AWS version of Opensearch
  2. Is there a way to configure an ML node on a AWS OS. All examples seen require you to update the opensearch.yml file, which I can't do because I'm using a hosted version.
  3. Is there some security permission that I don't have that could allow me to change these cluster settings?

Thanks!

Solution

  • Amazon OpenSearch Service supports ML Commons for versions >= 1.3

    One of three things is likely occurring here:

    1. Instance Types: certain operations (e.g. anomaly detection, ML Commons) are not supported on all instances. While AWS do not specify which instances are supported for ML Commons, it is likely that t2.micro and t2.small nodes will not be supported — this is the case with anomaly detection.

    2. Access Control: you must enable fine-grained access control (FGAC) in order to use non-standard plugins. kNN, SQL, and other CRUD-oriented actions, of course, are supported without FGAC. Although there is no documentation to confirm this, I imagine that ML Commons, as with Anomaly Detection, requires FGAC as the plugin involves significant modifications to cluster/node configuration.

      • Additionally, you’ll need to map the ml_full_access backend role to your user, even if you already have the all_access role. More on this here.

    3. Formatting of _settings API call: Amazon’s flavour of OpenSearch requires “PUT requests to the _cluster/settings API that use the "flat" settings form” [source]