We are in the process of implementing a CI/CD Pipeline and use TFS as our code repository and build & release tool. I have the following concrete questions:
Let's see if I can break this down for you.
We currently store the libraries and 3rd party tools that we require during our build in our code repository. We would like to analyze other ways of storing and accessing 3rd party tools and libraries.
- Would Artifactory be a right tool to store them into?
Yes, Artifactory and other package repository servers typically have a binary/raw repository that would be a good option. Think of Artifactory as a place to put your production use artifacts. So those libraries, 3rd party tools, 3rd party software, modules, etc could all be stored in different types of repositories in Artifactory. Artifactory would be an enterprise level package repository manager, optimized to do that, handle High Available, and more. You store, secure, and deploy binaries, packages, software, etc here. That can be to Dev, Test, Stage, and Production environments.
- As far as I understand Artifactory should only be used for storing build artifacts that can be thrown away and recreated.
I think this is a bit off - it's close. You could possibly store build artifacts that could be thrown away and recreated, but you can store so much more there as well. Stating it in this way doesn't really do enough justice into what it actually can do.
- OR is using Chocolatey a better option?
This is not a competing option to Artifactory. Chocolatey packages can be stored in Artifactory (Pro) in a NuGet type repository. The binaries would either be IN the Chocolatey packages or could be in a binary/raw repository.
Artifactory is a package repository store, where Chocolatey is a package manager (software management and deployment) for Windows.
As far as I understand we would need to create Chocolatey packages from our 3rd party tools and libraries. Where does: the source of those packages e.g. (.exe, .dll, .zip, .msi) usually reside?
- In an UNC file location?
- Or in a binary repository like Artifactory?
- Is using a binary repository to store build-time dependencies the correct approach?
You forgot the often most used and recommended approach:
In the Chocolatey package itself is the most recommended place to keep the binaries that a package represents. It's the most deterministic and reliable method for packages.
The problem is you may be looking at the community package repository as an example of packaging - we'd recommend NOT doing that as it doesn't represent but maybe 5% of packages out there. We don't recommend following that approach as it is not a reliable approach.
- It would need to reside there permanently and every new version is going to add to the repository size.
This is true, however Artifactory does have a culling approach (I believe) and it is optimized for these types of scenarios. We go over system requirements recommendations for different commercial repository solutions at https://chocolatey.org/docs/how-to-host-feed#commercial-repository-system-requirements.
- the Chocolatey packages themselves reside?
- In an UNC file location?
They absolutely could, but do keep in mind how permissions work with file shares and Windows permissions - https://chocolatey.org/docs/how-to-host-feed#local-folder-permissions
- Or in a binary repository like Artifactory?
No, that would be a NuGet OData repository available in Artifactory Pro. Yes, a NuGet Artifactory repository would be a great place, and one could use multiple repos for different environments, permissions, etc. Whatever makes sense for your environment.
- Is using a binary repository to store build-time dependencies the correct approach? It would need to reside there permanently and every new version is going to add to the repository size.
I think this was handled elsewhere.
- If we store the 3rd party tools and libraries outside of our code repository
- Do we need to use Chef and Chocolatey to access them?
- OR can we access them directly from TFS using Chocolatey without having to use Chef in the build process?
You can do either typically. If you do put the 3rd party tools into software deployment packages (aka Chocolatey packages), you will need Chocolatey to manage the deployment.
- Am I correct into thinking that Chef is primarily used to setup build environments with the required software and environment variables before starting a build process?
I would say you are misunderstanding Chef - it's a configuration management solution. It can set up build environments and keep them in a desired state, but that's limiting to the capabilities. Chef (and other configuration managers) are used to script out a state of expectations (desired state) for your infrastructure (infrastructure as code), which you can check into source control and test entire infrastructure changes before you deploy possibly using continuous integration (CI) servers like Jenkins, TeamCity, TFS to do so (which brings about testing your infrastructure, test driven infra, etc), etc. For developers this is somewhat intuitive, but for folks on the operations side of the house, this is all newer to bring these development practices to operations. I call it modern automation but some folks call these kinds of shifts DevOps.
You could use a Chef + Chocolatey + Artifactory solution to manage software and machine configurations across your organization, not just development environments. I think it's possible you are approaching all of these tools from the frame of reference of a developer type position and thus are just thinking in the context of provisioning and not longer term management of deployments, configuration, etc that an operations / system administrator might be looking at. These tools all certainly add something to the mix, but they are all complimentary and not at all competing solutions. For many larger organizations, putting these or similar components in place produces an architecture that can handle the critical infrastructure needs of today's organizations.