I am working on a large project that involves taking thousands (30,000+) static web pages and turning it into to a CMS.
The issue is many of these pages are duplicates within their directories. I want to keep the SEO intact by using 301 redirects, however, I am not sure how to go about doing such a large redirect (301).
Here is an example of the current directory structure for the pages.
/page.html /folder/page.html /folder/subfolder/page.html /folder/subfolder/anotherfolder/page.html
As you can see page.html is duplicated in all the directories.
For the new CMS the URL to that page would just be /page.html.
You should be redirected straight to /page.html
Options +FollowSymlinks
RewriteEngine on
RewriteRule ^(.*)page.html /page.html [R=301,NC]
This will always redirect http://www.foo.com/something/something/something/page.html back to http://www.foo.com/page.html using a 301 hard redirect.
The rewrite rule does this by looking at the URL, determining if anything before page.html is included (excluding the domain itself) and if it is, will 301 redirect. So you can literally use any sub-level, and as long as it ends with page.html, it will redirect to /page.html in the root directory.
In case you're wondering what [R=301,NC] means,
R // means simple redirect
R=301 // means redirect with a 301 header
NC // means no-case, or case insensitive
L // can also be used to say 'ignore all the other rules after this'