6 docs tagged with "persona:data-engineer"

Storing uncompressed data wastes bandwidth and increases storage capacity requirements; applying appropriate compression reduces both storage consumption and the energy needed to read and write it.

As part of your AI/ML process, you should evaluate using a pre-trained model and use transfer learning to avoid training a new model from scratch.

It's better to maximise storage utilisation so the storage layer is optimised for the task, not only in terms of energy proportionality but also in terms of embodied carbon. Two storage units running at low utilization rates will consume more energy than one running at a high utilization rate. In addition, the unused capacity on the underutilised storage unit could be more efficiently used for another task or process.

Large-scale AI/ML models require significant storage space and take more resources to run as compared to optimized models.

From an embodied carbon perspective, it's better to have an automated mechanism to delete unused storage resources so we are efficient with hardware and so that the storage layer is optimised for the task.

Efficient storage of the model becomes extremely important to manage the data used for ML model development.