Search code examples
firebasesplitab-testingfirebase-ab-testing

From which step does Firebase split the traffic by variants in AB-test


If I have a productOpen activation event (a custom chat-opening event inside the app), it starts counting from this event to evaluate the results as stated in Firebase ab-testing documentation.

The question is, at what point does the traffic split for all tests inside Firebase occur? With the startSession event (by default, opening the app), or with the selected activation event or something else?

I'll be very grateful for the answer!!


Solution

  • You need to differentiate between taking part in the test and playing into results of a test.

    Users will get values from one of the test variants if they are part of the target group. They will get that value right when the device fetches and activates data from Remote Config.

    However, at that point they will not necessarily be part of the results of the test. That's what the activation event is for, just as you correctly mentioned in your comment to your question.

    Btw, I've just recently raised a related question whether users would also leave a test at some point if the target requirements are not fulfilled anymore. However there is has been no reply yet. Same for my other related question on whether the user counts into the test if the activation event happened before the test was rolled out. Those questions might also be of relevant for you looking at the scenario you describe.

    Generally the documentation of A/B testing leaves a lot to be desired, but it is still beta, so here's hopes that all those questions will be answered at some point.