Qualified Investigation of ML Models for Forecasting SPEC Benchmark Performance

Abstract

Performance prediction via simulations is laborious and tedious. To avoid this problem, one way is to use supervised learning to forecast how well a system will do on SPEC benchmarks. This year's SPEC CPU includes a publicly available collection of results from 43 standardized performance tests divided into 4 suites and run on a variety of hardware setups. In this study, we will examine the dataset and try to find the answers to these questions: Can we reliably forecast the SPEC outcomes from the dataset's setups, without actually running the benchmarks? Secondly, which software and hardware aspects are most crucial? On the third point, in regard to forecast time and inaccuracy, which hyperparameters and models work best? thirdly, is it possible to use historical data to foretell how future systems will be performing? Preparing data, choosing features, this talk covers a wide range of topics, including hyperparameter tuning, employing decision trees, random forests, multi-layer perceptrons, and multi-task elastic-net neural networks to evaluate regression models, and more. There are three stages to feature selection: deleting features with zero variance, removing features with strong correlation, and finally, using Functional Recursion Using permutation importance, elastic-net coefficients, or importance metrics depending on trees to filter out candidates. Searching the hyperparameter space with a grid, we select the best models. Afterwards, we compare and evaluate their performance. We prove that using the initial set of 29 features in tree-based models yields 4% or better accuracy in predictions. With 10 characteristics, both the Random Forest and Quick Decision Tree models keep their average errors at 5% and 6%, respectively.