Why is my Azure DevOps pipeline outputting files to the server that doesn't exist in the published artifact?

Question

I have a .NET Core 8.0 web app which has both an appsettings.json file and an appsettings.Test.json file. I have an Azure DevOps pipeline that builds and deploys the app to a server. We are migrating away from using the appsettings.Test.json file and instead injecting environment variables into the appsettings.json file via the pipeline. How do I prevent the appsettings.Test.json file that is still in the repo (and will stay there for a little while until we've finished migrating and verified all functionality) from being outputed to the server?

The file in question is not visible in the published artifact.

But it ends up on the server anyway and it reappears after each pipeline run even if I have deleted it manually:

I have tried adding the following in our .csproj file in different combinations but nothing seems to work. Based on what is said here: ASP.NET Core: Exclude or include files on publish

  <ItemGroup>
    <Content Remove="appsettings.*.json" />
  </ItemGroup>

  <ItemGroup>
    <None Include="appsettings.*.json" />
  </ItemGroup>

  <ItemGroup>
    <Content Update="appsettings.*.json" CopyToPublishDirectory="Never" />
  </ItemGroup>

My Pipeline looks like this:

trigger:
  batch: true
  branches:
    include:
    - test
  paths:
   exclude:
     - src/ETLService/*

pool:
  vmImage: 'windows-latest'

variables:
  buildPlatform: 'Any CPU'
  buildConfiguration: 'Release'
  solution: '**/WebHost.sln'
  packageFolderName: 'WebHost'
  artifactName: 'WebAPI'
  IISWebsiteName: 'API'

stages: 
  - stage: BuildAndTest
    displayName: 'Build and test'
    jobs:
      - job: 
        displayName: '${{ variables.artifactName }}'
        steps:
            - template: PipelineTemplates/BuildAndTest.yml
            - template: PipelineTemplates/PublishWebArtifact.yml
  - stage: DeployToTest
    displayName: 'Deploy to test'
    variables:
    - group: APITest
    jobs:
      - deployment: DeployAPI
        displayName: 'API'
        environment: 'Test.TEST-SERVER'
        strategy:
          runOnce:
            deploy:
              steps:
                - checkout: none
                - download: 'current'
                  name: 'DownloadBuildArtifacts'
                  displayName: 'Download build artifacts'
                  artifact: '${{ variables.artifactName }}'
                - template: PipelineTemplates/DeployToIIS.yml

BuildAndTest.yml

steps:
  - task: NuGetToolInstaller@1
    displayName: "Nuget install"
    
  - task: NuGetCommand@2
    displayName: "Nuget restore"
    inputs:
      restoreSolution: '$(solution)'
      
  - task: DotNetCoreCLI@2
    displayName: 'Dotnet build'
    inputs:
      command: 'build'
      projects: '$(solution)'
      arguments: '--configuration $(buildConfiguration)'

  - task: DotNetCoreCLI@2
    displayName: 'Dotnet test'
    inputs:
      command: 'test'
      projects: '$(solution)'
      arguments: '--configuration $(buildConfiguration)'

PublishWebArtifact.yml

steps:
  - task: DotNetCoreCLI@2
    displayName: "Dotnet Publish"
    inputs:
      command: 'publish'
      publishWebProjects: true
      arguments: '--configuration $(BuildConfiguration) --output $(Build.ArtifactStagingDirectory)'
      zipAfterPublish: false

  - task: PublishBuildArtifacts@1
    displayName: "Publish Web Artifact"
    inputs:
      PathtoPublish: '$(Build.ArtifactStagingDirectory)'
      ArtifactName: '$(artifactName)'
      publishLocation: 'Container'

DeployToIIS.yml

steps:
  - task: IISWebAppManagementOnMachineGroup@0
    name: 'StopIIS'
    displayName: 'Stop IIS website'
    inputs:
      IISDeploymentType: 'IISWebsite'
      ActionIISWebsite: 'StopWebsite'
      StartStopWebsiteName: '$(IISWebsiteName)'
  - task: IISWebAppDeploymentOnMachineGroup@0
    inputs:
      WebSiteName: '$(IISWebsiteName)'
      Package: '$(Pipeline.Workspace)\$(artifactName)\$(packageFolderName)'
      TakeAppOfflineFlag: true
      JSONFiles: 'appsettings.json'
  - task: IISWebAppManagementOnMachineGroup@0
    name: 'StartIIS'
    displayName: 'Start IIS website'
    inputs:
      IISDeploymentType: 'IISWebsite'
      ActionIISWebsite: 'StartWebsite'
      StartStopWebsiteName: '$(IISWebsiteName)'

Answer 1

This seems to have been a typical case of wrongful assumptions. There was nothing wrong with my publishing of the artifact or my path-references deploying it. I had assumed that when I run a pipeline the agent working on it has a clean working directory to start with. This is not the case, and I never double checked this until I finally stumled over this post: How to clean target directory before downloading artifact into it?

So in my $(Pipeline.Workspace) folder I had remnants from a previous run where the files WERE outputted, and those versions were never cleaned out and kept being deployed. Unfortunately they got new timestamps on the target server every time so I thought they were newly generated just like the rest of the files.

The solution was to add the clean parameter in my deployment task in the pipeline:

- deployment: DeployAPI
  displayName: 'API'
  environment: 'Test.TEST-SERVER'
  workspace: # Added
    clean: all # Added