Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-4286

Add additional peak performance benchmark metrics alongside STARS

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • SRCnet
    • Hide

      The SRC workloads Gitlab repository hosts workloads that are representative of what users will be running at SRC sites. The STARS script collects runtimes for all of the tasks in an automated and concise way, taking an average over many runs. Runtime is only one part of understanding the impact of compute hardware on scientific pipelines. This feature looks at including metrics that track the use of the CPU, GPU, RAM, disk and power. Various software can achieve this as described below.

      We need to benchmark CPUs using standard tests that measure the raw performance of single and multithreaded performance, without the overhead of additional IO or complex workloads that combine steps using single and multithreaded applications. 

      We need disk IO performance measured with sustained loads outside of science tasks.

      We need to track power use of systems to understand our carbon footprint if possible (this won't be a complete picture, just a start).

      Weights and Biases will help track % use of single/multicore CPU, RAM/Disk use in terms of IO and capacity. This helps us understand bottle necks in tasks, and work out which tasks are dependent on specific hardware. 

       

      Show
      The SRC workloads Gitlab repository hosts workloads that are representative of what users will be running at SRC sites. The STARS script collects runtimes for all of the tasks in an automated and concise way, taking an average over many runs. Runtime is only one part of understanding the impact of compute hardware on scientific pipelines. This feature looks at including metrics that track the use of the CPU, GPU, RAM, disk and power. Various software can achieve this as described below. We need to benchmark CPUs using standard tests that measure the raw performance of single and multithreaded performance, without the overhead of additional IO or complex workloads that combine steps using single and multithreaded applications.  We need disk IO performance measured with sustained loads outside of science tasks. We need to track power use of systems to understand our carbon footprint if possible (this won't be a complete picture, just a start). Weights and Biases will help track % use of single/multicore CPU, RAM/Disk use in terms of IO and capacity. This helps us understand bottle necks in tasks, and work out which tasks are dependent on specific hardware.   
    • Hide
      • CPU benchmark software included in the STARS yaml or separate folder /bench/other
      • Disk IO bench included
      • Track CPU/GPU, RAM, Disk and power use inside a Weights and Biases dashboard.
      • Compare to Nvidia insight or other metric tracking software (possibly not needed/too big a scope)
      • Merge request and video demo
      Show
      CPU benchmark software included in the STARS yaml or separate folder /bench/other Disk IO bench included Track CPU/GPU, RAM, Disk and power use inside a Weights and Biases dashboard. Compare to Nvidia insight or other metric tracking software (possibly not needed/too big a scope) Merge request and video demo
    • 1
    • 1
    • 0
    • PI23 - UNCOVERED

    • SRC23-PB compute-tests example-workflows-and-benchmarks tests-compilation

    Description

      Alongside the science workloads, it is important to measure the performance of a system with a standard benchmarks. It would be useful to include this in the suit of tests run by STARS so they can be automatically run.

      Geekbench6 is a cross-platform CPU benchmarking software, reporting single and multithreaded scores: https://www.geekbench.com/index.html

      Cinebench is an industry standard CPU benchmarking software but only for windows? https://www.maxon.net/en/cinebench

      Weights and Biases is a way to track a range of metrics from CPU, GPU, RAM, disk, power and more. It also enables the creation of dashboards to show these metrics. This will be useful in preparation for defining our own custom dashboards: https://wandb.ai/site

       

      Attachments

        Issue Links

          Structure

            Activity

              People

                Unassigned Unassigned
                A.Clarke Clarke, Alex
                Votes:
                0 Vote for this issue
                Watchers:
                0 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (0%)

                  Feature Estimate: 1.0

                  IssuesStory Points
                  To Do00.0
                  In Progress   00.0
                  Complete00.0
                  Total00.0

                  Dates

                    Created:
                    Updated:

                    Structure Helper Panel