Is it possible to use an agent pool across multiple projects within the same ADO organization?
Yes
In Azure Pipelines, pools are scoped to the entire organization; so you can share the agent machines across projects.
You're a member of the infrastructure team and would like to set up a pool of agents for use in all projects. First make sure you're a member of a group in All agent pools with the Administrator role by navigating to agent pools page in your organization settings. Next create a New agent pool and select the option to Auto-provision corresponding agent pools in all projects while creating the pool. This setting ensures all projects have access to this agent pool. Finally install and configure agents to be part of that agent pool.