Gitlab CI/CD Terraform + AWS diretory issue

Question

I am trying to setup gtilab pipeline for serverless application deployment in AWS. Application infrastructure are deployed using Terraform. I am using hashocorp/terraform:ligh image in .gitlab-ci.yml but it is failing in plan stage

here is my .gitlab-ci.yml file

image:
  name: hashicorp/terraform:light
  entrypoint:
    - 'usr/bin/env'
    - 'PATH=/usr/localsbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'

stages:
  - fmt
  - validate
  - plan
  - apply

variables:
  TF_ROOT: "terraform"
  AWS_ACCESS_KEY_ID: "$AWS_ACCESS_KEY_ID"
  AWS_SECRET_ACCESS_KEY: "$AWS_SECRET_ACCESS_KEY"
  AWS_DEFAULT_REGION: "$AWS_DEFAULT_REGION"
  TF_BACKEND_BUCKET: "$TF_BACKEND_BUCKET"
  TF_BACKEND_KEY: "$TF_BACKEND_KEY"
  TF_BACKEND_DYNAMODB_TABLE: "$TF_BACKEND_DYNAMODB_TABLE"

before_script:
  - terraform --version
  - mkdir -p $TF_ROOT
  - cd $TF_ROOT
  - terraform init -reconfigure -backend-config="bucket=$TF_BACKEND_BUCKET" -backend-config="key=$TF_BACKEND_KEY" -backend-config="region=$AWS_DEFAULT_REGION" -backend-config="dynamodb_table=$TF_BACKEND_DYNAMODB_TABLE"
  - echo "$TR_ROOT"
  - echo "$TFVARS_FILE"
  - ls -al

format:
  stage: fmt
  script:
    - terraform fmt
  only:
    - branches

validate:
  stage: validate
  script:
    - terraform validate
  only:
    - branches

plan:
  stage: plan
  script:
    - terraform plan -var-file="$TFVARS_FILE" -out=tfplan
  only:
    - branches

apply:
  stage: apply
  script:
    - terraform apply -var-file="$TFVARS_FILE" -auto-approve -out=tfplan
  environment:
    name: dev
    url: "https://mycompany.awsapps.com/start/#"
  only:
    - main
  when: manual

Stage fmt and validate are running fine but failing at plan with below error.

• I do not understand why it is saying there are no tf files whereas there are tf files in project folder.

• Is below command going to buildempty directory every time, I run the pipeline?

• Can somebody help to fix the below and also help me to understand how image and directory works in gitlab?

  $ mkdir $TF_ROOT
   $ cd $TF_ROOT
   $ terraform init -reconfigure -backend-config="bucket=bucket=$TF_BACKEND_BUCKET" -backend-config="key=$TF_BACKEND_KEY" -backend-config="region=$AWS_DEFAULT_REGION" -backend-config="dynamodb_table=$TF_BACKEND_DYNAMODB_TABLE"
   Terraform initialized in an empty directory!
   The directory has no Terraform configuration files. You may begin working
   with Terraform immediately by creating Terraform configuration files.
   $ terraform plan -var-file="$TFVARS_FILE" -out=tfplan
   ╷
   │ Error: Failed to read variables file
   │ 
   │ Given variables file  does not exist.
   ╵
  

Answer 1

Removing - mkdir -p $TF_ROOT and - cd $TF_ROOT helped to fix the issue. -mkdir was creating directory under parent directory and main.tf file is within parent directory, not sub-directory.

image:
  name: hashicorp/terraform:light
  entrypoint:
    - 'usr/bin/env'
    - 'PATH=/usr/localsbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'

stages:
  - fmt
  - validate
  - plan
  - apply

variables:
  TF_ROOT: "terraform"
  AWS_ACCESS_KEY_ID: "$AWS_ACCESS_KEY_ID"
  AWS_SECRET_ACCESS_KEY: "$AWS_SECRET_ACCESS_KEY"
  AWS_DEFAULT_REGION: "$AWS_DEFAULT_REGION"
  TF_BACKEND_BUCKET: "$TF_BACKEND_BUCKET"
  TF_BACKEND_KEY: "$TF_BACKEND_KEY"
  TF_BACKEND_DYNAMODB_TABLE: "$TF_BACKEND_DYNAMODB_TABLE"

before_script:
  - terraform --version
  - terraform init -reconfigure -backend-config="bucket=$TF_BACKEND_BUCKET" -backend-config="key=$TF_BACKEND_KEY" -backend-config="region=$AWS_DEFAULT_REGION" -backend-config="dynamodb_table=$TF_BACKEND_DYNAMODB_TABLE"
  - echo "$TR_ROOT"
  - echo "$TFVARS_FILE"
  - ls -al

format:
  stage: fmt
  script:
    - terraform fmt
  only:
    - branches

validate:
  stage: validate
  script:
    - terraform validate
  only:
    - branches

plan:
  stage: plan
  script:
    - terraform plan -var-file="$TFVARS_FILE" -out=tfplan
  only:
    - branches

apply:
  stage: apply
  script:
    - terraform apply -var-file="$TFVARS_FILE" -auto-approve -out=tfplan
  environment:
    name: dev
    url: "https://mycompany.awsapps.com/start/#"
  only:
    - main
  when: manual