I have a Blazor Web Assembly project that consists of a Shared Prohect, Client project and a Server project and am wondering where to put Robots.txt to block crawlers.
It looks like my options are:
But not sure what the best practice would be?
Before discussing your options, a little review about the "server" and "client" project. A Blazor WASM is when compiled - in essence - just a bunch of static files. Your server project serves these static files to clients.
Let's start to discuss your options.
The result will be a robots.txt file in the wwwroot folder of the server project. If the server is an API and an MVC project and you want to enable crawlers, keep in mind to write your file properly.
1b) That said, it is also possible to create the robots.txt in the server project's wwwroot directory.
Middleware, in the sense of intercepting an HTTP request, can only be written on the server. Again, the client is just some static files.
The middleware would need to intercept an HTTP GET request for the robots.txt. If you have certain rules in mind, why some request should be served and others not. Then a middleware is a good option. Otherwise, a middleware adds the same value as robots.txt file while having a lot of more complexity.
The routing system is quite flexible, and with some tricks, it should be possible to direct a request for robot.txt to a controller. But this way is even more complex than the second method, with the same drawbacks.
My suggestion is to create a robot.txt file in the wwwroot directory of the client project unless you have complicated rules to "restrict access" to this file. If so, go with a middleware.
If you want to "test" the result, I find it quite insightful to have a dotnet publish of your server project. You can see what the result - in this case, static files - will look like if they are hosted.