Search code examples
aws-lambdaserverless-frameworkamazon-kinesis

Mysterious "Ingestion" Serverless error in AWS Lambda logs

I'm encountering a weird error when one of my Lambda functions is invoked. I have never seen this error. Google searches haven't helped at all. Here's what it looks like :

Ingestion server error 
{
    "request": {
        "url": "https://core.serverless.com/ingestion/kinesis/v1/request-response",
        "headers": {
            "accept-encoding": "gzip",
            "content-type": "application/json",
            "serverless_token": "[redacted]"
        }
    },
    "response": {
        "status": 500,
        "text": "{\"message\":\"Something went wrong forwarding RequestResponseEvent data.\"}"
    }
}

I'm pretty sure this has nothing to do with my code since my function works. Occasionally though, for periods of up to about an hour, my function fails with this error. Looking at the error it seems that the Serverless Framework is trying to redirect some data to their servers, for some reason, and failing in the process.

Why is this happening? What is Serverless trying to achieve? What can I do to stop this error from happening? I'm desperate for any insight.

I've also stumbled upon some environment variables that Serverless seems to be setting automatically :

Key Value
AWS_LAMBDA_EXEC_WRAPPER /opt/otel-extension/internal/exec-wrapper.sh
OTEL_RESOURCE_ATTRIBUTES sls_service_name=ODAC-API,sls_stage=dev,sls_org_id=[redacted]
SLS_OTEL_REPORT_LOGS_URL https://core.serverless.com/ingestion/kinesis/v1/logs
SLS_OTEL_REPORT_METRICS_URL https://core.serverless.com/ingestion/kinesis/v1/metrics
SLS_OTEL_REPORT_REQUEST_HEADERS serverless_token=[redacted]
SLS_OTEL_REPORT_REQUEST_RESPONSE_URL https://core.serverless.com/ingestion/kinesis/v1/request-response
SLS_OTEL_REPORT_TRACES_URL https://core.serverless.com/ingestion/kinesis/v1/traces
SLS_OTEL_USER_SETTINGS {}

I have no idea what these are for but it seems related (same url).

Could it have something to do with the sls --console command that I ran? Has it setup some mechanisms to monitor my functions? If so, how can I undo this?

This is really bugging me. Google seems to know nothing of this error, whatever I try to search. Has anyone encountered this before?

Solution

  • This appears to be a backend error coming from the Serverless.com's API.

    My guess is you can set console: false in your serverless.yml file and then re-deploy your service to remove the instrumentation - but I haven't confirmed that, it seems absent from their documentation