We have recently started using Sitecore A/B Testing and I am getting lots of questions about how the scoring works. I have been through the relevant Sitecore DMS documents but I still am not 100% sure if I understand how the scoring works.
My basic understanding is that the scores are based on Value Per Visit and my assumption is that the the value relates to the whole visit and not just the specific components we may be trying to optimize with the A/B Test.
For example, if option A has a goal associated with it worth 5 points, anyone presented with this option would get 5 points PLUS any other goal values they trigger during that visit to the site. That might add 5, 10, 50 or more to the visit score and then the option A score would be "total visits score/total visits".
Can anyone confirm if my assumptions are correct or explain where I may be off base? Can a user presented with option B change the score for option A?
By default, the Engagement Value is calculated on a per-visit basis. So your assumption is basically correct - and it does make it hard to test how a particular component variation does against another.
That being said; there are tools to help you.
We're currently implementing SBOS Accellerators into our solution. We have the same issue you are describing, and need a more fine-grained approach to testing.
Basically SBOS accellerators will allow you to track individual personalisation performance, not "just" looking at the overall Engagement Value.
Lars Petersen blogs about it here: http://www.larsdk.dk/2014/01/must-have-marketplace-modules-for-sitecore-digital-marketing-system/
Marketplace link for the module here: http://marketplace.sitecore.net/en/Modules/SBOS_Accelerators.aspx
We found a few issues in testing the module, but none were really severe. I know these issues are being fixed if they haven't already.