Visibility and monitoring in deployed machine learning systems
Machine learning allows us to build systems of unprecedented capability, enabling everything from self-driving cars to the synthesis of speech indistinguishable from a human voice. This sophistication comes at a cost, however, making it harder to understand and monitor the behaviour of live ML systems.
In this talk, I discuss lessons learned from building tools and workflows to monitor machine learning systems, including:
- why ML systems require us to reconsider how we monitor software;
- what to monitor and how to monitor it, from population drift or domain shift to historical backtests;
- the importance of integrating machine learning engineers into the monitoring process.