A/B testing

Change one thing, split the traffic at random, and compare. Version A is the current page; version B has, say, a different headline. Because both run at once to similar visitors, any difference in results is due to the change and not to the day of the week.

Arbitrageurs test at each stage of the funnel: ads (see creative testing), article headlines and layouts, the number and wording of related search terms, and even feed providers against each other. The measure of success should be revenue per visit, not click rate alone, since a change can raise clicks while lowering their value.

Three disciplines keep tests honest. Enough data: small differences on a few hundred visits are noise. One change at a time. And patience with revenue, which is estimated and delayed. There is also a boundary. Layout and wording tests must stay inside the feed's rules, so experiments that make ads look like content, or that nudge people to click, are off limits whatever they do to the numbers (see RSOC policies and AFS program policies).

An example

Say 10,000 visitors are split evenly. Version A: 45% lander CTR, $0.27 per visit. Version B: 52% lander CTR but lower-value clicks, $0.26 per visit. B "wins" on clicks and loses on revenue, so A stays.