We need to be able to analyse an HD video of a crowd scene from above (think train station) and be able to detect all moving objects and perform collision detection. A virtual object will effectively be superimposed on the image and must respond to the real objects. This needs to happen in real time.
Ideally I am looking for a library that can be consumed from C# to do this. This is way out of my comfort zone so I am just looking for some pointers as to whether this is an easy or infeasibly difficult problem and what tools would be up to the job. Is OpenCV suitable for example?
If there are performance limitations on how many objects can be processed what would these be?
Short answer: Not quite
Long answer: There are some libraries performing what you want to do, but all of them, as I know, are commercial stuff, they are expensive, and they are used in security applications. It's quite unlikely that you will get an SDK at a reasonable price, and be flexible enough to let you play with it. (Most likely it will output just number of people, or things like that)
If you still want to do the job, do not get discouraged: OpenCV is a great tool to help you, but it will definitely eat a significant amount of your time. Google crowd counting/tracking/detection in video, find a good paper, and implement it. But be aware that it may need deep mathematical and image processing knowledge to do it.
Processing speed advice: It will not work in real time, unless you write a parallelizable algorithm, and run it on multiple GPUs/CPUs, and tune it carefully. However, with a good implementation, you should have reasonable speed.
And, last advice: if you really want to do it, to work in C++ will be the least difficult part. It's much better to use the official opencv (c++) than the C# wrapper (better docs, up-to-date functions, all the functionalities are there for you). I do not know of any serious project using .NET wrappers - just students playing around.