Towards enterprise-ready AI deployments: Minimizing the risk of consuming AI models in business applications

Towards enterprise-ready AI deployments: Minimizing the risk of consuming AI models in business applications

• 1.
  Towards enterprise-ready AIdeployments: Minimizing the risk of consuming AI models in business applications Aleksander Slominski*, Vinod Muthusamy*, Vatche Ishakian+ *IBM Research, + Bentley University Artificial Intelligence for Industries (ai4i 2018) Sept 28, 2018
• 2.
  AI can reducecost or provide better customer experience: • “Healthcare, automotive and financial services are the sectors with the greatest potential for product enhancement and disruption due to AI” • $15.7 trillion “potential contribution to the global economy by 2030” • A. S. Rao and G. Verweij, “Sizing the prize,” PwC, 2017 AI in business applications
• 3.
  AI for frauddetection in banking ● Danske Bank ● Using a combination of rules, machine learning and deep learning models ● 60 percent reduction in false positives ○ “Danske Bank fights fraud with deep learning and AI - Case Study”
• 4.
  Maging Risk ofusing AI in business processes • Models can “fail” in corner cases and these can be very costly • “A framework was created to manage and track the models in the production system, and to make sure the models could be trusted.” • “Danske Bank fights fraud with deep learning and AI - Case Study” https://www.teradata.com/Resources/Case-Studies/Danske-Bank-Fight-Fraud -With-Deep-Learning-and-AI
• 5.
  https://www.mckinsey.com/business-functio ns/mckinsey-analytics/our-insights/what-ai-c an-and-cant-do-yet-for-your-business
• 6.
  Business Users needto learn to use AI • “A 2017 survey of 1,850 business leaders on the state of work showed that by the year 2020, 86 percent of companies say they will require intelligent automation to keep up with the pace of work.” • Example: “For a product manager, using AI to collect, analyze and act on massive amounts of data can result in forecasts that exceed 99 percent accuracy. As recently as ten years ago, the marketing profession was often relegated to an “art,” but now with advancements in gleaning insights from big data, it is already deep into its transformation by embracing statistics and science.” • https://www-935.ibm.com/services/us/gbs/thoughtleadership/superpo wer/
• 7.
  AI models introducerisk to business applications Business application Human decision Business application Model score risk Replacing a human task with a model
• 8.
  AI models introducerisk to business applications Business application Model v1 score Business application Model v2 score risk New version of a model is available
• 9.
  Model microservice • Commoninterface for models • Can introspect scoring inputs and results
• 10.
  We need aprincipled approach to consuming AI models in business applications •Generate usage dashboards •AI-aware staged deployment • Including continuous training •Correlations to business KPIs •ModelOps pipelines configure model deployment policies and manage lifecycle
• 11.
  We need aprincipled approach to consuming AI models in business applications • Generate usage dashboards • Correlations to business KPIs • AI-aware staged deployment • Including continuous training • A/B testing of models’ effect on KPIs • Drift detection • Fallback to safe/interpretable model or case worker • Active learning • Engage a human to make predictions where model is weak Business application Model v1 Model v2 App health scoring request primary canary Model proxy Human fallback App logs Model logs Model health Policy response Business application Model scoring request response
• 12.
  Business Users Dashboard
• 13.
  Helping business userscompare models Good outcome: 70% Model v1 Model v2 Bad outcome: 30% Good outcome: 80% Bad outcome: 20% ● Direct feedback on model predictions (supervised learning) ● Indirect feedback on application outcomes (reinforcement learning) ○ Function of KPI metrics ○ Evaluate model performance in terms of application performance  (better)  (worse)  (no big changes) 
• 14.
  Conclusions • The effectof AI models on business applications can be unpredictable • We propose a data-driven approach to apply AI techniques • Provide a high-level view of the risk of using AI in an enterprise application • Understand how models are being used • Understand the effect of models on business KPIs. • Business owners are able to make decisions about deploying AI • Track its performance over time • Get early warning about unexpected behavior • Roll back to previous versions. • Ultimately reduce the risk of AI models adversely affecting a business.
• 15.
  Questions
• 16.
  Feedback •Direct feedback • →supervised learning •Indirect feedback (e.g., KPI) • → reinforcement learning