Search code examples
jmeterperformance-testing

Jmeter Distributed testing results 403/Forbidden

Run jmeter distributed testing,all machines are gcp instances.Able to trigger tests from master to slaves.Problem is test results getting 100% Error rate 403/Forbidden. This happens on all the vms.I'm running a sample script with no external data files e.g csv,so it should be all pass. On running the tests on my local they pass just fine. Master & slaves using same version of jmeter,java and plugins

server.log of one of the vms

    2022-09-02 07:12:13,318 INFO o.a.j.e.RemoteJMeterEngineImpl: Running test
2022-09-02 07:12:13,330 INFO o.a.j.e.StandardJMeterEngine: Running the test!
2022-09-02 07:12:13,330 INFO o.a.j.s.SampleEvent: List of sample_variables: []
2022-09-02 07:12:13,356 INFO o.a.j.e.u.CompoundVariable: Note: Function class names must contain the string: '.functions.'
2022-09-02 07:12:13,356 INFO o.a.j.e.u.CompoundVariable: Note: Function class names must not contain the string: '.gui.'
2022-09-02 07:12:13,822 INFO o.a.j.e.StandardJMeterEngine: Starting ThreadGroup: 1 : Users
2022-09-02 07:12:13,822 INFO o.a.j.e.StandardJMeterEngine: Starting 10 threads for group Users.
2022-09-02 07:12:13,822 INFO o.a.j.e.StandardJMeterEngine: Thread will continue on error
2022-09-02 07:12:13,823 INFO o.a.j.t.ThreadGroup: Starting thread group... number=1 threads=10 ramp-up=10 delayedStart=false
2022-09-02 07:12:13,851 INFO o.a.j.s.SampleResult: Note: Sample TimeStamps are START times
2022-09-02 07:12:13,851 INFO o.a.j.s.SampleResult: sampleresult.default.encoding is set to ISO-8859-1
2022-09-02 07:12:13,851 INFO o.a.j.s.SampleResult: sampleresult.useNanoTime=true
2022-09-02 07:12:13,851 INFO o.a.j.s.SampleResult: sampleresult.nanoThreadSleep=5000
2022-09-02 07:12:13,854 INFO o.a.j.t.JMeterThread: Thread started: 10.10.52.197-Users 1-1
2022-09-02 07:12:13,912 INFO o.a.j.p.h.s.HTTPHCAbstractImpl: Local host = jmeter-worker-prf-002
2022-09-02 07:12:13,913 INFO o.a.j.t.ThreadGroup: Started thread group number 1
2022-09-02 07:12:13,913 INFO o.a.j.e.StandardJMeterEngine: All thread groups have been started
2022-09-02 07:12:13,928 INFO o.a.j.p.h.s.HTTPHC4Impl: HTTP request retry count = 0
2022-09-02 07:12:14,024 INFO o.a.j.p.h.s.h.LazyLayeredConnectionSocketFactory: Setting up HTTPS TrustAll Socket Factory
2022-09-02 07:12:14,029 INFO o.a.j.u.JsseSSLManager: Using default SSL protocol: TLS
2022-09-02 07:12:14,029 INFO o.a.j.u.JsseSSLManager: SSL session context: per-thread
2022-09-02 07:12:14,034 INFO o.a.j.u.SSLManager: JmeterKeyStore Location:  type JKS
2022-09-02 07:12:14,035 INFO o.a.j.u.SSLManager: KeyStore created OK
2022-09-02 07:12:14,035 WARN o.a.j.u.SSLManager: Keystore file not found, loading empty keystore
2022-09-02 07:12:14,504 INFO o.a.j.t.JMeterThread: Thread is done: 10.10.52.197-Users 1-1
2022-09-02 07:12:14,504 INFO o.a.j.t.JMeterThread: Thread finished: 10.10.52.197-Users 1-1
2022-09-02 07:12:14,852 INFO o.a.j.t.JMeterThread: Thread started: 10.10.52.197-Users 1-2
2022-09-02 07:12:15,094 INFO o.a.j.t.JMeterThread: Thread is done: 10.10.52.197-Users 1-2
2022-09-02 07:12:15,094 INFO o.a.j.t.JMeterThread: Thread finished: 10.10.52.197-Users 1-2
2022-09-02 07:12:15,847 INFO o.a.j.t.JMeterThread: Thread started: 10.10.52.197-Users 1-3
2022-09-02 07:12:16,076 INFO o.a.j.t.JMeterThread: Thread is done: 10.10.52.197-Users 1-3
2022-09-02 07:12:16,076 INFO o.a.j.t.JMeterThread: Thread finished: 10.10.52.197-Users 1-3
2022-09-02 07:12:16,840 INFO o.a.j.t.JMeterThread: Thread started: 10.10.52.197-Users 1-4
2022-09-02 07:12:17,059 INFO o.a.j.t.JMeterThread: Thread is done: 10.10.52.197-Users 1-4
2022-09-02 07:12:17,059 INFO o.a.j.t.JMeterThread: Thread finished: 10.10.52.197-Users 1-4
2022-09-02 07:12:17,840 INFO o.a.j.t.JMeterThread: Thread started: 10.10.52.197-Users 1-5
2022-09-02 07:12:18,063 INFO o.a.j.t.JMeterThread: Thread is done: 10.10.52.197-Users 1-5
2022-09-02 07:12:18,063 INFO o.a.j.t.JMeterThread: Thread finished: 10.10.52.197-Users 1-5
2022-09-02 07:12:18,831 INFO o.a.j.t.JMeterThread: Thread started: 10.10.52.197-Users 1-6
2022-09-02 07:12:19,052 INFO o.a.j.t.JMeterThread: Thread is done: 10.10.52.197-Users 1-6
2022-09-02 07:12:19,052 INFO o.a.j.t.JMeterThread: Thread finished: 10.10.52.197-Users 1-6
2022-09-02 07:12:19,843 INFO o.a.j.t.JMeterThread: Thread started: 10.10.52.197-Users 1-7
2022-09-02 07:12:20,076 INFO o.a.j.t.JMeterThread: Thread is done: 10.10.52.197-Users 1-7
2022-09-02 07:12:20,076 INFO o.a.j.t.JMeterThread: Thread finished: 10.10.52.197-Users 1-7
2022-09-02 07:12:20,829 INFO o.a.j.t.JMeterThread: Thread started: 10.10.52.197-Users 1-8
2022-09-02 07:12:21,148 INFO o.a.j.t.JMeterThread: Thread is done: 10.10.52.197-Users 1-8
2022-09-02 07:12:21,148 INFO o.a.j.t.JMeterThread: Thread finished: 10.10.52.197-Users 1-8
2022-09-02 07:12:21,845 INFO o.a.j.t.JMeterThread: Thread started: 10.10.52.197-Users 1-9
2022-09-02 07:12:22,052 INFO o.a.j.t.JMeterThread: Thread is done: 10.10.52.197-Users 1-9
2022-09-02 07:12:22,052 INFO o.a.j.t.JMeterThread: Thread finished: 10.10.52.197-Users 1-9
2022-09-02 07:12:22,841 INFO o.a.j.t.JMeterThread: Thread started: 10.10.52.197-Users 1-10
2022-09-02 07:12:23,092 INFO o.a.j.t.JMeterThread: Thread is done: 10.10.52.197-Users 1-10
2022-09-02 07:12:23,092 INFO o.a.j.t.JMeterThread: Thread finished: 10.10.52.197-Users 1-10

the only error i see is

2022-09-02 07:12:14,035 WARN o.a.j.u.SSLManager: Keystore file not found, loading empty keystore

could that be the issue?though i didnt load any keystore on local

Solution

  • jmeter.log file doesn't tell the full story (at least until you enable debug logging)

    First of all try temporarily enabling saving of request and response details for local and remote runs and compare resulting .jtl files. By default JMeter stores metrics in CSV format so you either need to amend Results File Configuration or add a Listener like Simple Data Writer and configure what and where to store

    enter image description here

    if the requests are the same enable debug logging and compare jmeter.log files. If there are no suspicious entries there try running your test from a machine on GCP in command-line non-GUI mode (not-distributed test run) as it might be the case your application doesn't allow incoming traffic from there