I have setup snowpipe to continuously load data in tables from an S3 bucket. This has been running about a month now (i.e. > 14 days). There is data in the bucket from before snowpipe was setup and we need to load those files into snowflake also. Snowpipe apparently only maintains copy history data for 14 days. What would be a good way to identify the files that have not yet been ingested into tables and bulk import them?
Did you try below view
SNOWFLAKE.ACCOUNT_USAGE.COPY_HISTORY . It stores last one year load history data from both copy into command as well as the snowpipe load history
Get the list of files loaded using snowpipe then you can plan for all remaining files load .
Please check the usage note on latency as well
https://docs.snowflake.com/en/sql-reference/account-usage/copy_history.html