{"id":1634,"date":"2024-07-02T12:43:54","date_gmt":"2024-07-02T12:43:54","guid":{"rendered":"https:\/\/2024.automl.cc\/?page_id=1634"},"modified":"2024-09-09T12:21:11","modified_gmt":"2024-09-09T12:21:11","slug":"towards-zero-cost-proxies-for-performance-predictionbeyond-cifar-10","status":"publish","type":"page","link":"https:\/\/2024.automl.cc\/?page_id=1634","title":{"rendered":"Towards Zero-Cost Proxies for Performance Prediction beyond CIFAR-10"},"content":{"rendered":"<div data-colibri-id=\"1634-c1\" class=\"style-727 style-local-1634-c1 position-relative\">\n  <!---->\n  <div data-colibri-component=\"section\" data-colibri-id=\"1634-c2\" id=\"custom\" class=\"h-section h-section-global-spacing d-flex align-items-lg-center align-items-md-center align-items-center style-732 style-local-1634-c2 position-relative\">\n    <!---->\n    <!---->\n    <div class=\"h-section-grid-container h-section-boxed-container\">\n      <!---->\n      <div data-colibri-id=\"1634-c3\" class=\"h-row-container gutters-row-lg-2 gutters-row-md-2 gutters-row-0 gutters-row-v-lg-2 gutters-row-v-md-2 gutters-row-v-2 style-733 style-local-1634-c3 position-relative\">\n        <!---->\n        <div class=\"h-row justify-content-lg-center justify-content-md-center justify-content-center align-items-lg-stretch align-items-md-stretch align-items-stretch gutters-col-lg-2 gutters-col-md-2 gutters-col-0 gutters-col-v-lg-2 gutters-col-v-md-2 gutters-col-v-2\">\n          <!---->\n          <div class=\"h-column h-column-container d-flex h-col-lg-auto h-col-md-auto h-col-auto style-734-outer style-local-1634-c4-outer\">\n            <div data-colibri-id=\"1634-c4\" class=\"d-flex h-flex-basis h-column__inner h-px-lg-2 h-px-md-2 h-px-2 v-inner-lg-2 v-inner-md-2 v-inner-2 style-734 style-local-1634-c4 position-relative\">\n              <!---->\n              <!---->\n              <div class=\"w-100 h-y-container h-column__content h-column__v-align flex-basis-100 align-self-lg-start align-self-md-start align-self-start\">\n                <!---->\n                <div data-colibri-id=\"1634-c5\" class=\"h-text h-text-component style-735 style-local-1634-c5 position-relative h-element\">\n                  <!---->\n                  <!---->\n                  <div class=\"\">\n                    <p>Date: 11.09.2024; 15:30-17:00<\/p>\n                    <p>Room: Auditorium<\/p>\n                    <h2>Speakers<\/h2>\n                    <ul>\n                      <li>\n                        <a href=\"https:\/\/gabikadlecova.github.io\" style=\"color: rgb(3, 169, 244); font-size: 1em; font-weight: 400; font-family: &quot;Open Sans&quot;;\" class=\"customize-unpreviewable\">Gabriela Kadlecov\u00e1<\/a>\n                      <\/li>\n                      <li>\n                        <a href=\"https:\/\/jovitalukasik.github.io\" style=\"color: rgb(3, 169, 244); font-size: 1em; font-weight: 400; font-family: &quot;Open Sans&quot;;\" class=\"customize-unpreviewable\">Jovita Lukasik<\/a>\n                      <\/li>\n                    <\/ul>\n                    <h2>Motivation<\/h2>\n                    <p>Performance prediction is an important part of the **Neural Architecture Search** process, as it allows avoiding resource-intensive network training. Recently, zero-cost proxies have gained in popularity, as they do not require any\n                      network training and provide a fast estimate of the performance. A drawback of zero-cost proxies is the lack of a solid theoretical understanding of these proxies. It is not clear why they correlate with the validation accuracy,\n                      and additionally show inconsistent results across diverse search spaces and tasks. In addition, some zero-cost proxies even tend to be biased to some network properties, for example, the number of operations in the graph.<\/p>\n                    <p>In this talk, we will present the advantages and also disadvantages of zero-cost proxies. We will provide a systematic overview of performance predictors and zero-cost proxies across search spaces (including newer ones like transformer\n                      spaces). Eventually, we will outline the different use cases, including zero-cost proxies as network encodings for tabular predictors, their usage in multi-objective settings, and the combination with graph neural predictors.<\/p>\n                    <p>In the hands-on part of the tutorial, we aim to showcase the importance of an analysis of the proxies beyond a simple correlation with the CIFAR-10 validation accuracy. We will furthermore introduce the recently developed [GRAF](https:\/\/openreview.net\/forum?id=EhPpZV6KLk)\n                      predictor \u2013 inspired by the zero-cost proxy biases, it uses neural graph features like operation count or path lengths for interpretable prediction. The participants will learn how zero-cost proxies and network properties impact\n                      the performance prediction, and how these proxies can be used to speed-up their NAS methods.<\/p>\n                    <h2>Outline<\/h2>\n                    <h3>Part 1: Presentation on Performance Predictors in NAS<\/h3>\n                    <p>The first part of this tutorial will be a 45 minutes presentation, which will provide the participants an overview of performance prediction methods with a special focus on zero-cost proxies.<\/p>\n                    <h4>Presentation outline:<\/h4>\n                    <ol>\n                      <li>Short overview of different performance prediction methods<\/li>\n                      <li>How predictors are used in different NAS settings<\/li>\n                      <li>Zero-cost proxies \u2013 basics, examples across search spaces (beyond CIFAR-10 and convnets)<\/li>\n                      <li>Integrating zero-cost proxies into tabular predictors and graph neural networks + multi-objective prediction<\/li>\n                      <li>Zero-cost proxy biases (continued in hands-on)<\/li>\n                    <\/ol>\n                    <h3>Part 2: Hands-on Tutorial<\/h3>\n                    <p>In the second part, we will have a 30 minutes hands-on tutorial, where the participants learn to analyze zero-cost proxies, use them for prediction, and combine them with other features of the neural graph.<\/p>\n                    <h4>Hands-on outline:<\/h4>\n                    <ol>\n                      <li>Visualize zero-cost proxy biases.<\/li>\n                      <li>Use zero-cost proxies and\n                        <a href=\"https:\/\/openreview.net\/forum?id=EhPpZV6KLk\" class=\"customize-unpreviewable\">GRAF<\/a> in prediction on search spaces available in NASLib.<\/li>\n                      <li>Explore the interpretability of the GRAF predictor via SHAP (\n                        <a href=\"https:\/\/papers.nips.cc\/paper_files\/paper\/2017\/hash\/8a20a8621978632d76c43dfd28b67767-Abstract.html\" class=\"customize-unpreviewable\">Lundberg and Lee, 2017<\/a>) \u2013 which proxies and features influence the prediction?<\/li>\n                      <li>(advanced) How to implement novel zero-cost proxies and graph features, and integrate them to GRAF and NASLib.<\/li>\n                    <\/ol>\n                    <h2>Speakers<\/h2>\n                    <p>\n                      <br>\n                    <\/p>\n                  <\/div>\n                <\/div>\n              <\/div>\n            <\/div>\n          <\/div>\n        <\/div>\n      <\/div>\n    <\/div>\n  <\/div>\n  <div data-colibri-component=\"section\" data-colibri-id=\"1634-c6\" id=\"overlappable\" class=\"h-section h-section-global-spacing d-flex align-items-lg-center align-items-md-center align-items-center style-745 style-local-1634-c6 position-relative\">\n    <!---->\n    <!---->\n    <div class=\"h-section-grid-container h-section-boxed-container\">\n      <!---->\n      <div data-colibri-id=\"1634-c7\" class=\"h-row-container gutters-row-lg-0 gutters-row-md-0 gutters-row-0 gutters-row-v-lg-0 gutters-row-v-md-0 gutters-row-v-0 style-746 style-local-1634-c7 position-relative\">\n        <!---->\n        <div class=\"h-row justify-content-lg-center justify-content-md-center justify-content-center align-items-lg-stretch align-items-md-stretch align-items-stretch gutters-col-lg-0 gutters-col-md-0 gutters-col-0 gutters-col-v-lg-0 gutters-col-v-md-0 gutters-col-v-0\">\n          <!---->\n          <div class=\"h-column h-column-container d-flex h-col-lg-auto h-col-md-auto h-col-auto style-747-outer style-local-1634-c8-outer\">\n            <div data-colibri-id=\"1634-c8\" class=\"d-flex h-flex-basis h-column__inner h-ui-empty-state-container h-px-lg-0 h-px-md-0 h-px-0 v-inner-lg-0 v-inner-md-0 v-inner-0 style-747 style-local-1634-c8 position-relative\">\n              <!---->\n              <!---->\n              <div class=\"w-100 h-y-container h-column__content h-column__v-align flex-basis-100\">\n                <!---->\n              <\/div>\n            <\/div>\n          <\/div>\n          <div class=\"h-column h-column-container d-flex h-col-lg-auto h-col-md-auto h-col-auto style-748-outer style-local-1634-c9-outer\">\n            <div data-colibri-id=\"1634-c9\" class=\"d-flex h-flex-basis h-column__inner h-px-lg-2 h-px-md-2 h-px-2 v-inner-lg-2 v-inner-md-2 v-inner-2 style-748 style-local-1634-c9 position-relative\">\n              <!---->\n              <!---->\n              <div class=\"w-100 h-y-container h-column__content h-column__v-align flex-basis-100 align-self-lg-center align-self-md-center align-self-center\">\n                <!---->\n                <div data-colibri-id=\"1634-c10\" class=\"h-global-transition-all h-heading style-749 style-local-1634-c10 position-relative h-element\">\n                  <!---->\n                  <div class=\"h-heading__outer style-749 style-local-1634-c10\">\n                    <!---->\n                    <!---->\n                    <h4 class=\"\">Gabi Kadlecov\u00e1<\/h4>\n                  <\/div>\n                <\/div>\n                <div data-colibri-id=\"1634-c11\" class=\"h-text h-text-component style-750 style-local-1634-c11 position-relative h-element\">\n                  <!---->\n                  <!---->\n                  <div class=\"\">\n                    <p>\n                      <a href=\"https:\/\/gabikadlecova.github.io\" style=\"color: rgb(3, 169, 244); font-size: 1em; font-weight: 400; font-family: &quot;Open Sans&quot;;\" class=\"customize-unpreviewable\">Gabi Kadlecov\u00e1<\/a>&nbsp;is a PhD student at Charles University in Prague. She also works at the Institute of Computer Science of the Czech Academy of Sciences, where she is supervised by Roman Neruda. Her focus is Neural Architecture\n                      Search, specifically performance prediction and architecture embedding using graph neural networks. She presented the tutorial on performance prediction in NAS at the AutoML Fall School 2023. She also presented a tutorial on survival\n                      analysis at a statistics workshop. Her teaching experience includes labs on nature-inspired algorithms (2024, 2 labs), and introductory Python programming labs (2021).<\/p>\n                  <\/div>\n                <\/div>\n              <\/div>\n            <\/div>\n          <\/div>\n          <div class=\"h-column h-column-container d-flex h-col-lg-auto h-col-md-auto h-col-auto style-748-outer style-local-1634-c12-outer\">\n            <div data-colibri-id=\"1634-c12\" class=\"d-flex h-flex-basis h-column__inner h-px-lg-2 h-px-md-2 h-px-2 v-inner-lg-2 v-inner-md-2 v-inner-2 style-748 style-local-1634-c12 position-relative\">\n              <!---->\n              <!---->\n              <div class=\"w-100 h-y-container h-column__content h-column__v-align flex-basis-100 align-self-lg-center align-self-md-center align-self-center\">\n                <!---->\n                <div data-colibri-id=\"1634-c13\" class=\"h-global-transition-all h-heading style-749 style-local-1634-c13 position-relative h-element\">\n                  <!---->\n                  <div class=\"h-heading__outer style-749 style-local-1634-c13\">\n                    <!---->\n                    <!---->\n                    <h4 class=\"\">Jovita Lukasik<\/h4>\n                  <\/div>\n                <\/div>\n                <div data-colibri-id=\"1634-c14\" class=\"h-text h-text-component style-750 style-local-1634-c14 position-relative h-element\">\n                  <!---->\n                  <!---->\n                  <div class=\"\">\n                    <p>\n                      <a href=\"https:\/\/jovitalukasik.github.io\" style=\"font-family: &quot;Open Sans&quot;; font-weight: 400; font-size: 1em; color: rgb(3, 169, 244);\" class=\"customize-unpreviewable\">Jovita Lukasik<\/a>&nbsp;is a postdoctoral researcher at the University of Siegen, specialized in the research for architectural representations for efficient neural architecture search using performance prediction methods. Her current\n                      NAS research focus is in the area of multi-objectiveness for computer vision using prediction methods, especially with in the context of neural network robustness . In addition to her research, she also co-organizes virtual seminars\n                      on Automated Machine Learning and serves as a publication chair for the European Conference on Computer Vision 2024. She also co-organized the NAS@ICLR Workshop in 2021. Furthermore, she was a panelist at the \u201cAutoML, Trustworthiness\n                      and Alignment\u201d panel at the AutoML conference.<\/p>\n                  <\/div>\n                <\/div>\n              <\/div>\n            <\/div>\n          <\/div>\n          <div class=\"h-column h-column-container d-flex h-col-lg-auto h-col-md-auto h-col-auto style-753-outer style-local-1634-c15-outer\">\n            <div data-colibri-id=\"1634-c15\" class=\"d-flex h-flex-basis h-column__inner h-ui-empty-state-container h-px-lg-0 h-px-md-0 h-px-0 v-inner-lg-0 v-inner-md-0 v-inner-0 style-753 style-local-1634-c15 position-relative\">\n              <!---->\n              <!---->\n              <div class=\"w-100 h-y-container h-column__content h-column__v-align flex-basis-100\">\n                <!---->\n              <\/div>\n            <\/div>\n          <\/div>\n        <\/div>\n      <\/div>\n    <\/div>\n  <\/div>\n<\/div>","protected":false},"excerpt":{"rendered":"<p>Date: 11.09.2024; 15:30-17:00 Room: Auditorium Speakers Gabriela Kadlecov\u00e1 Jovita Lukasik Motivation Performance prediction is an important part of the **Neural Architecture Search** process, as it allows avoiding resource-intensive network training. Recently, zero-cost proxies have gained in popularity, as they do not require any network training and provide a fast estimate of the performance. A drawback [&hellip;]<\/p>\n","protected":false},"author":9,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"page-templates\/full-width-page.php","meta":{"footnotes":""},"class_list":["post-1634","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/2024.automl.cc\/index.php?rest_route=\/wp\/v2\/pages\/1634","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/2024.automl.cc\/index.php?rest_route=\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/2024.automl.cc\/index.php?rest_route=\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/2024.automl.cc\/index.php?rest_route=\/wp\/v2\/users\/9"}],"replies":[{"embeddable":true,"href":"https:\/\/2024.automl.cc\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1634"}],"version-history":[{"count":9,"href":"https:\/\/2024.automl.cc\/index.php?rest_route=\/wp\/v2\/pages\/1634\/revisions"}],"predecessor-version":[{"id":2605,"href":"https:\/\/2024.automl.cc\/index.php?rest_route=\/wp\/v2\/pages\/1634\/revisions\/2605"}],"wp:attachment":[{"href":"https:\/\/2024.automl.cc\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1634"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}