Skip to main content

Select efficient accelerators and instance types for AI workloads

Applicable Role: Provider and Consumer

Description​

AI workloads such as training, fine-tuning, and inference require significant compute resources. The type of hardware used, including CPUs, GPUs, TPUs, and specialized accelerators, has a direct impact on energy efficiency and performance.

Different hardware options vary in their ability to execute AI workloads efficiently. Selecting appropriate hardware and compute resources improves utilization, reduces execution time, and lowers overall energy consumption.

Solution​

  • Choose hardware that is optimized for the specific workload, such as GPUs or TPUs for parallel processing tasks
  • Use specialized accelerators where available to improve efficiency
  • Right-size compute resources to match workload requirements and avoid over-provisioning
  • Monitor utilization and adjust resource allocation to improve efficiency
  • Evaluate performance-per-watt benchmarks when selecting hardware and instance types

SCI Impact​

SCI = (E × I) + M per R

E (Energy): Efficient hardware reduces compute time and energy consumption for AI workloads.

M (Embodied Carbon): Better utilization can reduce the number of required machines and associated embodied emissions.

Cost Impact​

  • Hardware costs: Instance type choice affects hourly compute rates significantly
  • Utilization efficiency: Better hardware-workload fit reduces per-inference cost
  • Reserved instance savings: Efficient hardware selection enables better RI negotiation
  • Power and cooling costs: Specialized accelerators may have lower operational energy costs
  • Trade-off: Premium accelerators (H100, TPU) cost more upfront but may have better cost-per-inference

Assumptions​

  • Suitable hardware options are available for the workload
  • Performance benchmarks reflect real-world usage

Considerations​

  • Specialized hardware may not be available in all regions
  • Costs may vary across hardware options
  • Migration to different hardware may require changes in software or frameworks
  • Underutilized hardware can negate efficiency gains

References​