Visibility and monitoring in deployed machine learning systems

Machine learning allows us to build systems of unprecedented capability, enabling everything from self-driving cars to the synthesis of speech indistinguishable from a human voice. This sophistication comes at a cost, however, making it harder to understand and monitor the behaviour of live ML systems.

In this talk, I discuss lessons learned from building tools and workflows to monitor machine learning systems, including:

• why ML systems require us to reconsider how we monitor software;
• what to monitor and how to monitor it, from population drift or domain shift to historical backtests;
• the importance of integrating machine learning engineers into the monitoring process.