[AB Test] DS Ads, ab test case study


After advertisers submitted their advertisements, there’s a chance the advertisement got rejected because of illegal or other issues; there’s also a chance that the rejection is wrongly made. We now have a new feature, an ‘Appeal’ bottom on the page when the advertisement submitted is rejected. Advertizers can click on this ‘Appeal’ bottom and resubmit the same advertisement again, and can probably get accepted this time, or got rejected for the second time.


  1. how will you define the impact brought by this feature? (revenue)
    A: Business metric: Revenue rescued, Revenue rescued/per customer
    Guardrail metric: legal ads approved/per reviewer/per day
  2. how will you measure the impact of this feature?
    A: Run A/B testing on the randomization unit (i.e. advertiser) on the control group and treatment group.
  3. Some advertisers that got rejected may not use that bottom; they remake an advertisement and resubmit it. How will you consider those revenues?
    A: We can try to find those ‘duplicate’ advertisement record and remove the first one that was rejected and replace it with the successful one followed. Since the successful contribution of the second time transaction is not because of the new feature, we could count them in.
  4. what is the control group and treatment group?
    A: Firstly, randomly assign the treatment to advertisers who got rejected. And then to get the final control and treatment groups by conducting pair-wise matching for those advertisers in both groups based on their similar attributes.
  5. what is the bias you should be looking for? will these biases lead to overestimation or underestimation?
    A: There might be potential interference between treatment group and control group such that advertisers in the control group may be told by other advertisers in the treatment group that they can try to remake and resubmit their advisement again. This can cause the test being underestimated.
  6. say there is only 30% of advertisers in the treatment group click the ‘Appeal’ bottom, how will you compare the control and treatment group?
    A: ? 【我疑惑这个问题的问点是什么,而且感觉这个题和treatment和control group的定义还有关系】
  7. what would you do to increase the advertisers use the ‘Appeal’ bottom?
    A: Pop-up notifications to advertisers who got rejected with notes reminder that people who really think they don’t break any rules can for sure resubmit their application to specialist for review.

以上是我的答案, 请问大家第六题是怎么考虑的,以及对于3-7题大家是否有不同的看法,欢迎一起讨论和分享思路。感谢!

我个人认为,这个Q6是想让你讨论ab testing enrollment阶段的一些处理方法。如果这个实验是想检测“advertisers的满意度是否在使用了appeal button之后有提高”的话,那么你的treatment可能应该只把clicked this button的user加进来。但是这样又可能有selection bias, 因为click button这个行为本身可能代表了一类用户特点。因此进一步的,可能需要在control group针对treatment的user做matching, 希望做到的是,control和treatment group user是randomize且都很有代表性的。