Downloaded 10 times
Uploaded byDomino Data Lab
Supporting innovation in insurance with randomized experimentation
The document discusses the importance of randomized experimentation in the insurance sector, highlighting its effectiveness for assessing the impact of innovations on customer experience. It outlines essential considerations for experimental design, such as statistical power, cognitive biases, and the need for rapid iteration during the testing process. The conclusion emphasizes that despite operational challenges, wider adoption of this methodology can create significant value within insurance firms.
More Related Content
Similar to Supporting innovation in insurance with randomized experimentation
Supporting innovation in insurance with randomized experimentation
- 1. Supporting Innovation inInsurance with Randomized Experimentation Matt Best Senior Data Scientist Allstate Insurance Company DOMINO DATA SCIENCE POP-UP CHICAGO
- 2. Examples from RonnyKohavi’s “Introduction to A/B Testing” at KDD ’17 and Blake and Coey 2014: Why Marketplace Experimentation is Harder than it Seems
- 3. But, it’s difficultto accurately forecast the impact of an innovation on customer experience
- 4. Randomized experiments arean effective method to learn the impact of innovation
- 5. Randomized experiments arean effective method to learn the impact of innovation Random sampling
- 6. Randomized experiments arean effective method to learn the impact of innovation • Randomized experiments are also sometimes referred to as randomized controlled trials, A/B/n tests, bucket tests, and field experiments depending upon discipline Treatment Group Control Group Random sampling Random assignment
- 7. CUSTOMERS CUSTOMERS Web MobileService Rep AgencyWeb Mobile
- 8. Before an experiment,consider the importance of statistical power How much data is needed to assess an innovation’s impact? How large does the impact need to be for it to be detectable with a fixed quantity of data? Ideally, we’d ask … In practice, operational constraints often shift the question to …
- 9. Note: All axesunits are arbitrary to keep proprietary information confidential
- 10. Implied MDE Fixed samplesize Note: All axes units are arbitrary to keep proprietary information confidential
- 11. Optimistic estimate of impact Note:All axes units are arbitrary to keep proprietary information confidential Implied MDE Fixed sample size
- 12. Optimistic estimate of impact Newsample size Note: All axes units are arbitrary to keep proprietary information confidential Implied MDE Fixed sample size
- 13. A power analysissaved us from running a test with almost no chance of success! Optimistic estimate of impact New sample size Note: All axes units are arbitrary to keep proprietary information confidential Implied MDE Fixed sample size
- 14. Key takeaways beforethe experiment begins • Lessons learned: • Need to be able to rapidly iterate on power/sample analysis and experimental design as operational constraints are identified • Observations are rarely independent and identically distributed; be explicit about sources of variability • Technological solutions: • Using a knowledge management platform has enabled us to track the evolution of assumptions through the design process • Developed python package to verbosely describe and simulate progress through process flows
- 15. After an experiment,consider how cognitive biases influence decision making Treatment Group Control Group
- 16. Treatment Group Control Group Confirmationbias We look for and more strongly weigh information that confirms what we already believe Look again…my hypothesis must be true!
- 17. Treatment Group Control Group Hindsightbias After we see results, we tend to overestimate how well we would have predicted (or did predict) those results all along That result was obvious! Why run a test?
- 18. How to benefitfrom hindsight, prospectively? • Pre-mortem: “Imagine your experiment has spectacularly failed – write the story of that failure.” • Pre-register: “What would you do if we observe a {positive|negative|null} result?” • Good decision ‘hygiene’ helps reveal critical risks, assumptions, and disagreements early on… while we can still do something about it!
- 19. Summary and ClosingThoughts Randomized experimentation is a powerful tool data scientists may leverage to create value. Though challenging, insurance firms may benefit from wider adoption of the methodology, even in situations where it’s operationally challenging. Data scientists can enable experimentation by driving forward both technological and cultural solutions.
- 20. Thanks for yourattention! XD Team • Anthony Pham • Andrew Mehrmann • Matthew McAuley • Melissa Alvarado • Nicholas Syring (intern) BehavioralSight • Linnea Gandhi Allstate - D3 • Xiaoyan Anderson • Neal Coleman • Tony Eberle • Florent Buisson • Jason Khan Domino • Anna Anisin • Jeremy Mason
- 21. Data and Analyticsat Allstate: Our Centralized Organization Managing and governing data Developing analytics solutions Effectively delivering solutions through technology 250 data and analytics experts Who We Are We have experts across five locations: Silicon Valley, CA; Seattle, WA; Northbrook, IL; Chicago, IL; Belfast, Northern Ireland Data and analytics is embedded in everything we do. Each day, Allstate uses analytic models to create millions of targeted digital media impressions, process tens of thousands of claims, produce tens of thousands of quotes, and predict thousands of decision making actions across the corporation. Where We Work What We Do
- 22. Join Us fora Tour of the Allstate Office Sign up before noon at: Registration desk or Allstate booth Tuesday, November 14 3:30 - 4:00 pm Meet at the Allstate booth