The First Steps Towards a More Structured ML Development: Experiment Tracking

Why Your Machine Learning Team Should Track Experiments

Developing a machine learning model is an iterative process that involves testing and refining the model until it reaches a level of performance suitable for deployment. The large amount of “trial-and-error” runs are the main challenge in the very research heavy process. Each iteration represents a small experiment, testing different hypotheses and making adjustments to model parameters or training data, allowing developers to fine-tune and optimize the model accordingly. That is why this phase is also called the “experimentation phase”.

Experimentation is crucial during machine learning model development because even minor changes in model parameters or training data can significantly impact the model's performance and outcome.

However, without a systematic approach to track and manage experiments, the development process can become chaotic, making it challenging to maintain an overview of past experiments or onboard new colleagues to the project. In a large project, the number of experiments can easily exceed 1,000 runs. You could take an example from the world of research and keep logbooks, either on paper, in a project management tool or on a spreadsheet. Alternatively, you could also build your own automation and store important metadata in a database. Or follow our recommendation and make use of one of the many experiment tracking tools available, where you can even start for free when choosing open-source.

What is Experiment Tracking in Machine Learning?

Experiment trackers are offering a solution to organize, log, and analyze the outcomes of experiments in a structured and accessible manner. They do so by enabling developers to save crucial metadata associated with each experiment, such as the model configuration and evaluation metrics. Additionally, some experiment trackers also allow capturing data and code versions.

The goal of experiment trackers is to provide developers with a comprehensive overview of the experiments conducted throughout the development process. The key advantages of utilizing experiment tracking tools are:

1. Reproducibility: By accurately logging the details of each experiment, experiment tracking facilitates reproducibility. Data scientists can revisit past experiments, reconstruct the conditions, and reiterate from there. This capability is essential for building trust in the results and enabling further experimentation. Adding a tool for data versioning to an experiment tracker enables comprehensive reproducibility.
2. Comparison and Evaluation: Experiment tracking tools allow for easy comparison and evaluation of different experiments or model versions. By analyzing the recorded metrics and outcomes, data scientists can identify the most efficient configurations and approaches. This empowers them to make the right decisions when selecting the best-performing model for deployment or deciding on further optimizations. We recommend selecting a tool that offers intuitive visualizations, making the information easier and quicker to comprehend.
3. Collaboration: Experiment tracking tools enhance collaboration in the dev team by offering a more centralized platform for storing and accessing experiment metadata. This ensures dev team members have access to the same information, facilitating effective communication. It also allows new team members to understand the work that has been done, reducing the time required for onboarding.

Overview of Experiment Tracking Tools

We summarized the most common experiment trackers used in machine learning development below. All of these tools differentiate in their features, for instance if they are hosted or deployed-on-premise, in their searching & organization functions or in the comparisons of metadata, while for many teams open-source plays an important role.

Choose a Holistic AI Management Approach

While experiment tracking is a crucial part of managing AI development, there are additional aspects to consider for a more holistic approach to AI project management which are naturally out of scope of experiment trackers. Focusing on these areas can significantly boost the success of your AI project:

1. Understandability for All Stakeholders: The development of machine learning models often involves collaboration with various stakeholders (e.g. managers or domain experts), some of whom may not be deeply involved in the technical aspects or in daily operations. These stakeholders need high-level information and easy-to-understand visualizations. If there is a deficiency in this area, communicating the results and implications of experiments becomes challenging. Providing intuitive visualizations and explanations can facilitate collaboration and decision-making across diverse teams.
2. Documentation of Model Development: While experiment trackers excel in capturing and organizing information about individual experiments, they may not be able to document the development process of the model. This is usually still done in other tools, such as Confluence or similar. Documentation is essential for maintaining a comprehensive record of the model's evolution, including iterations, refinements, and - most importantly - the rationale behind various design choices. The importance of documentation also increases in light of upcoming regulatory requirements in AI development, hand-overs to colleagues and the reproducibility of experiments. Incorporating features that facilitate documentation of the development process can help enhance traceability and provide a more holistic view of the model's journey.
3. Reporting Progress and Business Impact: Businesses require project progress reporting that includes key performance indicators (KPIs) aligned with their specific goals and objectives. The ability to generate reports showcasing the development progress, business KPIs, and providing explanations for decision-making can be crucial in demonstrating the value and impact of the AI project to stakeholders and managers.
4. Compliance Checks: Depending on the industry or application, there may be specific (upcoming) regulations and guidelines that need to be followed during the experimentation phase. Experiment trackers may not always have built-in mechanisms to ensure compliance with regulatory requirements.
5. Data-Centric Approach: Tracking metrics and parameters is a good first step, but to properly manage your development process, it's also important to include as much information about the data as possible. This could involve using other tools for data versioning, such as DVC, or conducting a thorough data analysis with every run to understand what data you’ve trained on.

Bridging the gap between technical and non-technical stakeholders by providing clear visualizations, intuitive explanations, and user-friendly interfaces is essential for effective collaboration and communication throughout the development process.

We at trail want you to fully understand the whole development process, regardless of your (non-)technical background. Our AI management platform complements the capabilities of experiment trackers by preparing metrics & data in a way that is suitable for any stakeholder, creating reports on KPIs and automating the documentation during development, which also supports audit-readiness.

Conclusion

Experimentation plays a vital role in machine learning model development, allowing ML developers to optimize performance as well as outcomes. However, managing and tracking experiments can be challenging without proper tools and practices in place. Experiment tracking provides a structured approach to organize, log, and analyze experiment outcomes, enabling reproducibility, comparison, evaluation and better collaboration in the dev team.

By leveraging experiment tracking tools, data scientists and project leads can streamline the development process, increase efficiency, and make more informed decisions about model deployment. Nevertheless, comprehensive AI development management involves more areas, including stakeholder understandability, documentation, progress and business impact reporting and a data-centric approach.

To boost the success and increase transparency of your AI project, trail builds upon the solid foundation provided by experiment trackers, by adding another layer to create a holistic management solution for ML development. Our platform provides insights into MLOps data and makes it accessible to various stakeholders. Collaboration between business and tech was never that easy - take a look yourself.