What architecture should I go for? There is a worker process that takes messages off a queue and processes one by one. Now I can have many windows service instances do this job, or I can have one WCF service hosted as a windows service to act as some sort of a server on which I can start a thread for each instance.
Which approach would scale better? I want viewpoints when we are talking about cloud-like infrastructure where scaling is very quick, as well as, non-cloud-based infrastructure.
The most scalable approach in cloud would be to have an instance configured with handling services with WCF or Web-api(preferable if you are designing RESTFUL services). Such instance would be scalable. In case of Azure consider this as Web-role hosted with IIS. IIS is designed to scale with incoming requests from client. Another advantage is you get IIS and asp.net infrastructure to manage security and other stuff. However if you do not need that use windows service hosted WCF or Web-api services. The web-service instances would then queue the messages or say work loads into a queue. In case of Azure the queue can be a service bus queue. Then you can have windows services or Azure worker roles to pull the work items from the queue. Usually a single worker pulls multiple messages from queue and processes it. While doing so those messages are not available to other workers. After the worker is done remove the messages or work loads from the queue. Because after some time they will be visible to other workers there are settings for visibility like in amazon SQS there is visibility timeout setting. *Pull 5 to 10 messages in a single worker and run them as a thread pool threads as a "Task" parallely.
Note that the architecture with scalable service instances at the front end, receiving incoming requests AND worker roles at the back-end clearing the work items as soon as possible is dependent on the distributed, robust and scalable queue that cloud provides like Azure service bus and Amazon SQS. Otherwise you may have contention problems.
For in house(non-cloud) deployment normally if there is no distributed queue, here you can have the IIS hosted service instances do everything or the windows service do everything or combination. I suggeste to have HTTP services hosted with IIS to serve web pages and receive incoming requests(REST services) . IIS is good in that. However IIS pool can get recycled and may not be running if there are no requests for long time. Hence if there is requirement to run scheduled tasks or jobs windows service at the back end is good at this. Having windows service to do everything is certainly doable but from my experience using IIS and asp.net for incoming request helps in productivity. You can share the security with services and web app. I prefer to have IIS at front end and windows service at the back end. With this I do not have to manage security with windows service. Try NServiceBus for inhouse queue https://github.com/NServiceBus/NServiceBus. I have not evaluated it.