Towards Zero-Cost Proxies for Performance Prediction beyond CIFAR-10

Date: 11.09.2024; 15:30-17:00

Room: Auditorium

Speakers

• Gabriela Kadlecová
• Jovita Lukasik

Motivation

Performance prediction is an important part of the **Neural Architecture Search** process, as it allows avoiding resource-intensive network training. Recently, zero-cost proxies have gained in popularity, as they do not require any network training and provide a fast estimate of the performance. A drawback of zero-cost proxies is the lack of a solid theoretical understanding of these proxies. It is not clear why they correlate with the validation accuracy, and additionally show inconsistent results across diverse search spaces and tasks. In addition, some zero-cost proxies even tend to be biased to some network properties, for example, the number of operations in the graph.

In this talk, we will present the advantages and also disadvantages of zero-cost proxies. We will provide a systematic overview of performance predictors and zero-cost proxies across search spaces (including newer ones like transformer spaces). Eventually, we will outline the different use cases, including zero-cost proxies as network encodings for tabular predictors, their usage in multi-objective settings, and the combination with graph neural predictors.

In the hands-on part of the tutorial, we aim to showcase the importance of an analysis of the proxies beyond a simple correlation with the CIFAR-10 validation accuracy. We will furthermore introduce the recently developed [GRAF](https://openreview.net/forum?id=EhPpZV6KLk) predictor – inspired by the zero-cost proxy biases, it uses neural graph features like operation count or path lengths for interpretable prediction. The participants will learn how zero-cost proxies and network properties impact the performance prediction, and how these proxies can be used to speed-up their NAS methods.

Outline

Part 1: Presentation on Performance Predictors in NAS

The first part of this tutorial will be a 45 minutes presentation, which will provide the participants an overview of performance prediction methods with a special focus on zero-cost proxies.

Presentation outline:

1. Short overview of different performance prediction methods
2. How predictors are used in different NAS settings
3. Zero-cost proxies – basics, examples across search spaces (beyond CIFAR-10 and convnets)
4. Integrating zero-cost proxies into tabular predictors and graph neural networks + multi-objective prediction
5. Zero-cost proxy biases (continued in hands-on)

Part 2: Hands-on Tutorial

In the second part, we will have a 30 minutes hands-on tutorial, where the participants learn to analyze zero-cost proxies, use them for prediction, and combine them with other features of the neural graph.

Hands-on outline:

1. Visualize zero-cost proxy biases.
2. Use zero-cost proxies and GRAF in prediction on search spaces available in NASLib.
3. Explore the interpretability of the GRAF predictor via SHAP ( Lundberg and Lee, 2017) – which proxies and features influence the prediction?
4. (advanced) How to implement novel zero-cost proxies and graph features, and integrate them to GRAF and NASLib.

Speakers