Data Engineer
Structures data systems and storage solutions to minimize energy consumption while maintaining performance and accessibility.
6 patternsArchitecture
Large-scale AI/ML models require significant storage space and take more resources to run as compared to optimized models.
Use efficient file formats for AI/ML developmentEfficient storage of the model becomes extremely important to manage the data used for ML model development.
Development
Storing uncompressed data wastes bandwidth and increases storage capacity requirements; applying appropriate compression reduces both storage consumption and the energy needed to read and write it.
Leverage pre-trained models and transfer learning for AI/ML developmentAs part of your AI/ML process, you should evaluate using a pre-trained model and use transfer learning to avoid training a new model from scratch.
Operations
It's better to maximise storage utilisation so the storage layer is optimised for the task, not only in terms of energy proportionality but also in terms of embodied carbon. Two storage units running at low utilization rates will consume more energy than one running at a high utilization rate. In addition, the unused capacity on the underutilised storage unit could be more efficiently used for another task or process.
Set storage retention policiesFrom an embodied carbon perspective, it's better to have an automated mechanism to delete unused storage resources so we are efficient with hardware and so that the storage layer is optimised for the task.