Data Engineer

Persona

Structures data systems and storage solutions to minimize energy consumption while maintaining performance and accessibility.

5 patterns

Development

Compress stored data

Storing uncompressed data wastes bandwidth and increases storage capacity requirements; applying appropriate compression reduces both storage consumption and the energy needed to read and write it.

• cloud
• size:medium

Leverage pre-trained models and transfer learning

Fine-tune existing pre-trained models instead of training from scratch to dramatically reduce the compute, energy, and time required for model development.

• ai
• compute
• machine-learning
• size:medium

Optimize data storage formats for AI training and inference

Use efficient storage formats, compression, and indexing strategies for AI datasets and embeddings to reduce storage footprint, data transfer, and retrieval compute.

• ai
• machine-learning
• size:medium
• storage

Operations

Optimise storage utilization

It's better to maximise storage utilisation so the storage layer is optimised for the task, not only in terms of energy proportionality but also in terms of embodied carbon. Two storage units running at low utilization rates will consume more energy than one running at a high utilization rate. In addition, the unused capacity on the underutilised storage unit could be more efficiently used for another task or process.

• size:small
• storage

Set storage retention policies

From an embodied carbon perspective, it's better to have an automated mechanism to delete unused storage resources so we are efficient with hardware and so that the storage layer is optimised for the task.

• size:small
• storage