Search code examples
amazon-web-servicesaws-lambdaamazon-efs

AWS EFS too slow when i use git & npm install


I'm using aws S3 storage and cloudfront for my dozens of static web sites. Also i'm using aws lambda with nodejs and EFS for git, node_modules and build cache files.

When i try git clone, npm install and npm run build from EFS it's working to slow. But when i try from lambda /tmp folder it's working x10 faster than the EFS storage.

I need a storage like EFS because i storage dozens of web sites git, node package and cache files. So how can i increase EFS performance.


Solution

  • If you have used the standard settings for EFS you will be utilizing burstable credits which are depleted the more file changes you make.

    Depending on the file size and the number of changes on the EFS mount you nay be depleting the available credits which would provide performance problems for any application attached to the EFS mount. You can detect this by looking at the BurstCreditBalance CloudWatch metric, also keep an eye for any flatlining of TotalIOBytes as this might suggest it has reached its maximum throughput.

    When you perform the git clone you could also use the --depth with a value of 1 to create a shallow clone. This option will get only the latest commit, as opposed to cloning the entire git history too.

    A improvement to this workflow I would suggest reconsidering to use the following technologies to provide the workflow for what you want. Rather than a Lambda function create a CodePipeline pipeline that will trigger a CodeBuild job. This CodeBuild job would be responsible for running the npm install task for you as well as any other actions.

    Part of CodePipeline's flow is that it will store the legacy artifact in S3 along the way, so that you have a copy of it. The CodePipeline can also deploy to your S3 bucket as well at the end.

    A couple of links that might be useful for you: