If I create an Azure Artifact feed and reference it form a build, will the build continue to function if I am, as feed owner, removed from the organisation?
Yes, the feed will continue to exist, and assuming you gave the appropriate build identity access to the feed, your builds will continue to have access to the feed.
Members of the Project Collection Administrators group (effectively, your organization administrators) will remain owners of the feed and can give permissions to other users.