Data Engineer

Persona

Structures data systems and storage solutions to minimize energy consumption while maintaining performance and accessibility.

6 patterns

Architecture

Optimize the size of AI/ML models

Large-scale AI/ML models require significant storage space and take more resources to run as compared to optimized models.

Use efficient file formats for AI/ML development

Efficient storage of the model becomes extremely important to manage the data used for ML model development.

Development

Compress stored data

Storing uncompressed data wastes bandwidth and increases storage capacity requirements; applying appropriate compression reduces both storage consumption and the energy needed to read and write it.

Leverage pre-trained models and transfer learning for AI/ML development

As part of your AI/ML process, you should evaluate using a pre-trained model and use transfer learning to avoid training a new model from scratch.

Operations

Optimise storage utilization

It's better to maximise storage utilisation so the storage layer is optimised for the task, not only in terms of energy proportionality but also in terms of embodied carbon. Two storage units running at low utilization rates will consume more energy than one running at a high utilization rate. In addition, the unused capacity on the underutilised storage unit could be more efficiently used for another task or process.

Set storage retention policies

From an embodied carbon perspective, it's better to have an automated mechanism to delete unused storage resources so we are efficient with hardware and so that the storage layer is optimised for the task.