Predicting Counterfactuals from Large Historical Data and Small Randomized Trials - Rosenfeld, Nir; Mansour, Yishay; Yom-Tov, Elad (Proceedings of the 26th International Conference on World Wide Web Companion, 2017)
When a new treatment is considered for use, whether a pharmaceutical drug or a search engine ranking algorithm, a typical question that arises is, will its performance exceed that of the current treatment? The conventional way to answer this counterfactual question is to estimate the eﬀect of the new treatment in comparison to that of the conventional treatment by running a controlled, randomized experiment. While this approach theoretically ensures an unbiased estimator, it suﬀers from several drawbacks, including the diﬃculty in ﬁnding representative experimental populations as well as the cost of running randomized trials. Moreover, such trials neglect the huge quantities of available controlcondition data, which in principle can be utilized for the harder task of predicting individualized eﬀects. In this paper we propose a discriminative framework for predicting the outcomes of a new treatment from a large dataset of the control condition and data from a small (and possibly unrepresentative) randomized trial comparing new and old treatments. Our learning objective, which requires minimal assumptions on the treatments, models the relation between the outcomes of the diﬀerent conditions. This allows us to not only estimate mean eﬀects but also to generate individual predictions for examples outside the small randomized sample. We demonstrate the utility of our approach through experiments in three areas: search engine operation, treatments to diabetes patients, and market value estimation of houses. Our results demonstrate that our approach can reduce the number and size of the currently performed randomized controlled experiments, thus saving signiﬁcant time, money and eﬀort on the part of practitioners.