Search code examples
testingistiomirroring

Prevent repeat write API calls whilst traffic mirroring

I'm looking at using traffic mirroring with Istio to dark test releases.

The mirror'd traffic will mean write APIs like order & payment, etc are called multiple times which I don't want else I'll be charging the customer twice & sending them a duplicate product.

Is there a standard way to prevent this (stubbing seems an odd thing to do in production) or is mirroring only really applicable for read APIs?

Solution

  • Issue

    There is diagram of mirroring setup wih traffic flows.

    enter image description here

    Although these mirror requests are mirrored as “fire and forget” and the reply from the mirror service is just dropped (by the envoy proxy sidecar) to /dev/null and not returned to the caller, it's still uses this api.

    Solution

    As mentioned in comments

    In my opinion you should add path for your testing purposes with some custom header, so this could be tested only by you or your organization, and the customer shouldn't be involved in that.

    This topic is described in detail here by Christian Posta.

    When we deploy a new version of our service and mirror traffic to the test cluster, we need to be mindful of impact on the rest of the environment. Our service will typically need to collaborate with other services (query for data, update data, etc). This may not be a problem if the collaboration with other services is simply reads or GET requests and those collaborators are able to take on additional load. But if our services mutates data in our collaborators, we need to make sure those calls get directed to test doubles and not the real production traffic.

    There are a few approaches you may consider, all of them are described in the link above:

    • Stubbing out collaborating services for certain test profiles
    • Synthetic transactions
    • Virtualizing the test-cluster’s database
    • Materializing the test-cluster’s database

    In practice, mirroring production traffic to our test cluster (whether that cluster exists in production or in non-production environments) is a very powerful way to reduce the risk of new deployments. Big webops companies like Twitter and Amazon have been doing this for years. There are some challenges that come along with this approach, but there exist decent solutions as discussed in the patterns above.