My application has a load balancer which distributes payloads to two test servers. I need to test the load balancing using jmeter. Do I need to have an instance of jmeter run on all these servers to test this? How can I test this efficiently using jmeter? I also have to take out one of the test servers during load and see the response as well. Please advise how to proceed.
You should test the configuration you will be using in production, i.e. use single endpoint (which is the loadbalancer) instead of direct backend node hosts. You should not be running JMeter on any of the backend machines to avoid mutual interference. If single JMeter instance will not be enough to create the anticipated load and you will go for the distributed testing JMeter slave nodes (load generators) need to be installed outside the application under test.
Also if your Load Balancer in its turn has multiple IP addresses make sure your add DNS Cache Manager to your Test Plan to avoid IP caching on OS/JVM levels