Search code examples
pythonamazon-web-servicesaws-lambdaopensearchamazon-opensearch

How to issue POST requests to opensearch cluster in VPC?


I've deployed an opensearch cluster into a VPC. I have a VPC endpoint and can successfully issue GET requests to "VPC endpoint URL"/_cluster/settings to get the cluster config, but issuing POST requests to the same URL does not work. Should issuing POST requests to "VPC endpoint URL"/_cluster/settings work?

Looking at the documentation - https://docs.aws.amazon.com/opensearch-service/latest/developerguide/configuration-api.html#configuration-api-actions-describedomainconfig - it says to -

POST https://es.us-east-1.amazonaws.com/2021-01-01/opensearch/domain/domain-name/config
"SnapshotOptions": {
    "AutomatedSnapshotStartHour": 3
  }

Since that is a public URL, I can't do that with my cluster. I also tried a POST request to "VPC endpoint URL"/config without any luck.

My working GET request -

def lambda_handler(event, context):
    x = requests.get('https://vpc-<private endpoint>.us-east-1.es.amazonaws.com/_cluster/settings')

Successful output from GET request -

Function Logs
:"5s","max_index_buffer_size":"-1","shard_inactive_time":"5m",...

My failing POST request -

def lambda_handler(event, context):
    url = 'https://vpc-<private endpoint>.us-east-1.es.amazonaws.com/_cluster/settings'
    myobj = {"SnapshotOptions": {
    "AutomatedSnapshotStartHour": 3
  } }
    x = requests.post(url, json = myobj)

Error Message -

Function Logs
START RequestId: b483f2ca-0051-468a-81cf-8a771a667bd2 Version: $LATEST
{"Message":"Your request: '/_cluster/settings' is not allowed for verb: POST"}

Solution

  • The resolution I discovered was signing the request. - https://docs.aws.amazon.com/opensearch-service/latest/developerguide/request-signing.html

    Ex. -

    from opensearchpy import OpenSearch, RequestsHttpConnection, AWSV4SignerAuth
    import boto3
    
    host = '' # cluster endpoint, for example: my-test-domain.us-east-1.es.amazonaws.com
    region = '' # e.g. us-west-1
    
    credentials = boto3.Session().get_credentials()
    auth = AWSV4SignerAuth(credentials, region)
    index_name = 'movies'
    
    client = OpenSearch(
        hosts = [{'host': host, 'port': 443}],
        http_auth = auth,
        use_ssl = True,
        verify_certs = True,
        connection_class = RequestsHttpConnection
    )
    
    q = 'miller'
    query = {
      'size': 5,
      'query': {
        'multi_match': {
          'query': q,
          'fields': ['title^2', 'director']
        }
      }
    }
    
    response = client.search(
        body = query,
        index = index_name
    )
    
    print('\nSearch results:')
    print(response)