Search code examples
machine-learningamazon-sagemakeramazon-sagemaker-studio

Sagemaker: Missing expected ":" in Code I did not write

I am currently following this tutorial to use my own model for inference on AWS: https://aws.amazon.com/blogs/startups/how-startups-deploy-pretrained-models-on-amazon-sagemaker/

But when using the model it fails and the error logs for deploying the endpoint are:

"2023-09-19T08:47:49,767 [INFO ] W-9002-model_1.0-stdout MODEL_LOG -   File ""/opt/conda/lib/python3.10/site-packages/sagemaker_inference/transformer.py"", line 206, in _validate_user_module_and_set_functions"
"2023-09-19T08:47:49,767 [INFO ] W-9002-model_1.0-stdout MODEL_LOG -     user_module = importlib.import_module(user_module_name)"
"2023-09-19T08:47:49,767 [INFO ] W-9002-model_1.0-stdout MODEL_LOG -   File ""/opt/conda/lib/python3.10/importlib/__init__.py"", line 126, in import_module"
"2023-09-19T08:47:49,767 [INFO ] W-9002-model_1.0-stdout MODEL_LOG -     return _bootstrap._gcd_import(name[level:], package, level)"
"2023-09-19T08:47:49,767 [INFO ] W-9002-model_1.0-stdout MODEL_LOG -   File ""<frozen importlib._bootstrap>"", line 1050, in _gcd_import"
"2023-09-19T08:47:49,767 [INFO ] W-9002-model_1.0-stdout MODEL_LOG -   File ""<frozen importlib._bootstrap>"", line 1027, in _find_and_load"
"2023-09-19T08:47:49,767 [INFO ] W-9002-model_1.0-stdout MODEL_LOG -   File ""<frozen importlib._bootstrap>"", line 1006, in _find_and_load_unlocked"
"2023-09-19T08:47:49,767 [INFO ] W-9002-model_1.0-stdout MODEL_LOG -   File ""<frozen importlib._bootstrap>"", line 688, in _load_unlocked"
"2023-09-19T08:47:49,767 [INFO ] W-9002-model_1.0-stdout MODEL_LOG -   File ""<frozen importlib._bootstrap_external>"", line 879, in exec_module"
"2023-09-19T08:47:49,768 [INFO ] W-9002-model_1.0-stdout MODEL_LOG -   File ""<frozen importlib._bootstrap_external>"", line 1017, in get_code"
"2023-09-19T08:47:49,768 [INFO ] W-9002-model_1.0-stdout MODEL_LOG -   File ""<frozen importlib._bootstrap_external>"", line 947, in source_to_code"
"2023-09-19T08:47:49,768 [INFO ] W-9002-model_1.0-stdout MODEL_LOG -   File ""<frozen importlib._bootstrap>"", line 241, in _call_with_frames_removed"
"2023-09-19T08:47:49,768 [INFO ] W-9002-model_1.0-stdout MODEL_LOG -   File ""/opt/ml/model/code/inference.py"", line 15"
"2023-09-19T08:47:49,768 [INFO ] W-9002-model_1.0-stdout MODEL_LOG -     def create_connection(host, user, pw, db_name, port)"
"2023-09-19T08:47:49,768 [INFO ] W-9002-model_1.0-stdout MODEL_LOG -                                                         ^"
"2023-09-19T08:47:49,768 [INFO ] W-9002-model_1.0-stdout MODEL_LOG - SyntaxError: expected ':'"

Does anyone have similar experience with this sort of behaviour/error?

For anyone interested, here is the whole error log: https://github.com/RobertReichel1993/logs

I tried using different models or just the plain tutorial but the error persists.

Solution

  • The example seems to use torchserve which may not work well with LLM's. Refer to the below example for different ways of deploying LLM's on SageMaker.

    1. Using SageMaker LMI container - https://github.com/aws-samples/sagemaker-hosting/tree/main/GenAI-Hosting/Large-Language-Model-Hosting/LLM-Deployment-SageMaker/LMI/Falcon-40b-and-7b
    2. Using Huggingface TGI container - https://github.com/aws-samples/sagemaker-hosting/tree/main/GenAI-Hosting/Large-Language-Model-Hosting/LLM-Deployment-SageMaker/TGI/Llama2-7B