Background:
After advertisers submitted their advertisements, there’s a chance the advertisement got rejected because of illegal or other issues; there’s also a chance that the rejection is wrongly made. We now have a new feature, an ‘Appeal’ bottom on the page when the advertisement submitted is rejected. Advertizers can click on this ‘Appeal’ bottom and resubmit the same advertisement again, and can probably get accepted this time, or got rejected for the second time.
Questions:
- how will you define the impact brought by this feature? (revenue)
A: Business metric: Revenue rescued, Revenue rescued/per customer
Guardrail metric: legal ads approved/per reviewer/per day - how will you measure the impact of this feature?
A: Run A/B testing on the randomization unit (i.e. advertiser) on the control group and treatment group. - Some advertisers that got rejected may not use that bottom; they remake an advertisement and resubmit it. How will you consider those revenues?
A: We can try to find those ‘duplicate’ advertisement record and remove the first one that was rejected and replace it with the successful one followed. Since the successful contribution of the second time transaction is not because of the new feature, we could count them in. - what is the control group and treatment group?
A: Firstly, randomly assign the treatment to advertisers who got rejected. And then to get the final control and treatment groups by conducting pair-wise matching for those advertisers in both groups based on their similar attributes. - what is the bias you should be looking for? will these biases lead to overestimation or underestimation?
A: There might be potential interference between treatment group and control group such that advertisers in the control group may be told by other advertisers in the treatment group that they can try to remake and resubmit their advisement again. This can cause the test being underestimated. - say there is only 30% of advertisers in the treatment group click the ‘Appeal’ bottom, how will you compare the control and treatment group?
A: ? 【我疑惑这个问题的问点是什么，而且感觉这个题和treatment和control group的定义还有关系】 - what would you do to increase the advertisers use the ‘Appeal’ bottom?
A: Pop-up notifications to advertisers who got rejected with notes reminder that people who really think they don’t break any rules can for sure resubmit their application to specialist for review.
以上是我的答案， 请问大家第六题是怎么考虑的，以及对于3-7题大家是否有不同的看法，欢迎一起讨论和分享思路。感谢！