Search code examples
amazon-web-servicesvisual-studio-codeamazon-sagemaker

Lifecycle Configuration File fails to auto-terminate amazon sagemaker studio instance


I am trying to configure a Lifecycle Configuration (LCC) file in order to implement an auto-terminate feature upon reaching an idle threshold for sagemaker's code editor. I have followed the solution in this question. I am able to see and select the LCC file in the amazon sagemaker studio UI after successful completion of the steps but the instances do not shutdown after 1 hour of idle time (example below is 60 seconds for testing).

This is the LCC file:

# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
# SPDX-License-Identifier: MIT-0

#!/bin/bash
set -eux
ASI_VERSION=0.3.0

# User variables [update as needed]
IDLE_TIME_IN_SECONDS=60  # in seconds, change this to desired idleness time before app shuts down

# System variables [do not change if not needed]
CONDA_HOME=/opt/conda/bin
LOG_FILE=/var/log/apps/app_container.log # Writing to app_container.log delivers logs to CW logs.
SOLUTION_DIR=/var/tmp/auto-stop-idle # Do not use /home/sagemaker-user
PYTHON_PACKAGE=sagemaker_code_editor_auto_shut_down-$ASI_VERSION.tar.gz
PYTHON_SCRIPT_PATH=$SOLUTION_DIR/sagemaker_code_editor_auto_shut_down/auto_stop_idle.py

# Installing cron
sudo apt-get update -y
sudo sh -c 'printf "#!/bin/sh\nexit 0" > /usr/sbin/policy-rc.d'
sudo apt-get install -y cron

# Creating solution directory.
sudo mkdir -p $SOLUTION_DIR

# Downloading autostop idle Python package.
echo "Downloading autostop idle Python package..."
curl -LO --output-dir /var/tmp/ https://github.com/aws-samples/sagemaker-studio-apps-lifecycle-config-examples/releases/download/v$ASI_VERSION/$PYTHON_PACKAGE
sudo $CONDA_HOME/pip install -U -t $SOLUTION_DIR /var/tmp/$PYTHON_PACKAGE

# Touch file to ensure idleness timer is reset to 0
echo "Touching file to reset idleness timer"
touch /opt/amazon/sagemaker/sagemaker-code-editor-server-data/data/User/History/startup_timestamp

# Setting container credential URI variable to /etc/environment to make it available to cron
sudo /bin/bash -c "echo 'AWS_CONTAINER_CREDENTIALS_RELATIVE_URI=$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI' >> /etc/environment"

# Add script to crontab for root.
echo "Adding autostop idle Python script to crontab..."
echo "*/2 * * * * /bin/bash -ic '$CONDA_HOME/python $PYTHON_SCRIPT_PATH --time $IDLE_TIME_IN_SECONDS --region $AWS_DEFAULT_REGION >> $LOG_FILE'" | sudo crontab -

Solution

  • Apparently there is a bug in the latest sagemaker distribution image (1.6, seems to have been solved in 1.7) that prevented cron from installing.

    Replacing this:

    sudo apt-get install -y cron
    

    with this:

    # Check if cron needs to be installed
    status="$(dpkg-query -W --showformat='${db:Status-Status}' "cron" 2>&1)"
    if [ ! $? = 0 ] || [ ! "$status" = installed ]; then
        # Fixing invoke-rc.d: policy-rc.d denied execution of restart.
        sudo /bin/bash -c "echo '#!/bin/sh
        exit 0' > /usr/sbin/policy-rc.d"
    
        # Installing cron.
        echo "Installing cron..."
        sudo apt install cron
    else
        echo "Package cron is already installed."
        sudo cron
    fi
    

    ...solved my problem.