Model drift is the degradation of analytics model performance due to changes in data and relationships between variables.
• Data Drift: When characteristics of the independent, feature, or predictor variables change
• Concept Drift: When the characteristics of the dependent, label, or target variables change
• Algorithms Drift: When algorithms, including assumptions, lose relevance due to changes in business needs
What are the root causes of these kinds of model drift? The primary reason for model drift is a change in business fundamentals. Business strategies and objectives change due to mergers, acquisitions and divestitures (MAD), new product introductions, new laws and regulations, entry into new markets, and more. Today more than ever, a business is a constantly evolving entity. All these disruptions will change the way original data analytics models are used by the business. Knowing the sources of model drift will help you identify the right remediation measures you will need to get the model back to an acceptable or desired level of performance.
Why does model drift matter? What is the business impact of model drift? Today, data analytics models are increasingly becoming major drivers of business decisions and performance. This trend will continue at a much faster pace, given the rate at which data is captured and the increasing maturity of machine learning (ML) technology. In this reality, managing model drift is critical to ensuring the accuracy of insights and predictions. Fundamentally, reducing or eliminating model drift will enhance trust in data & analytics, thereby promoting adoption across your organization.
So, how can you reduce or eliminate model drift? At its core, model drift is not a technology management problem; it is a change management problem. This change in the context of data and analytics can be effectively managed by implementing the following three strategies.
First, data is a reflection of reality. Often, the degradation of data results in the degradation of model and business performance. You need to manage data drift with effective data governance practices. We all know the fundamental principle of data processing is “garbage in, garbage out.” So, identify the variables in your hypothesis, define your data quality KPIs, set targets and thresholds, and track these metrics continuously to stay up to date with changes in data quality.Secondly, assess the fundamental dynamics of your business model and constantly review the relevance of existing data analytics models with your stakeholders. While talking to your stakeholders, ask these questions:
- Why do you want to have insights? How much do you want to know? What is the value of knowing and not knowing these insights? What outcomes will they enable?
- Who owns the insights coming out of our models? Who is accountable when it comes to transforming insights into decisions and actions?
- What are the relevant data attributes required for the model to derive accurate and timely insights?
Lastly, integrate ModelOps and DataOps practices to enable the quick and ethical replacement of the deployed analytics model with a revamped version if business circumstances change. Data is the fuel on which models run; without data, models have practically no business utility. Basically, the sound integration of ModelOps and DataOps practices helps to quickly progress analytics models from the lab to production.
Overall, the best way to manage model drift is by continuously governing and monitoring your model performance with the right KPIs. While deploying data analytics models is important, what really matters is the consumption of these models by business stakeholders to improve business performance. As they say, change is the only constant in life, and businesses are certainly no exception. Involving business stakeholders early on, reviewing any and all changes in metrics and data, and continuously adjusting for improvements is critical to successfully managing model drift.
About the Authors
Dr. Prashanth Southekal is managing principal of DBP Institute (www.dbp-institute.com), a data and analytics consulting, research, and education firm. He is also an advisor at Astral Insights. He is a Consultant, Author, and Professor. He has consulted for over 80 organizations including P&G, GE, Shell, Apple, and SAP. Dr. Southekal is the author of two books — “Data for Business Performance” and “Analytics Best Practices” — and writes regularly on data, analytics, and machine learning in Forbes.com, FP&A Trends, and CFO.University. His second book, ANALYTICS BEST PRACTICES was ranked the #1 analytics book of all time in May 2022 by BookAuthority. Apart from his consulting pursuits, he has trained over 3,000 professionals worldwide in Data and Analytics. Dr. Southekal is also an Adjunct Professor of Data and Analytics at IE Business School (Madrid, Spain). CDO Magazine included him in the top 75 global academic data leaders of 2022. He holds a PhD. from ESC Lille (FR) and an MBA from Kellogg School of Management (U.S.). He lives in Calgary, Canada with his wife, two children, and a high-energy Goldendoodle dog. Outside work, he loves juggling and cricket.
Christopher Andrassy is an entrepreneur and managing partner at Astral Insights, focused on transforming data into sustainable business value on a global scale. He began his career at PwC in New York City, supporting the digital transformation of mature organizations struggling to innovate in a hyper-competitive world. After experiencing the shortcomings of traditional analytics practices, he decided to begin a new chapter alongside colleagues and industry veterans. His departure from New York marked the inception of Astral Insights, a Raleigh-based decision intelligence firm helping mid-market and enterprise clients transform data into profit. Chris is also an investor focused on innovative technologies including synthetic biology, sustainable energy, and artificial intelligence. Outside of work, he is an avid musician, skier, traveler, and fitness enthusiast.
