In AI model development, compute power is the "engine," but data and storage are the "fuel." Whether you are training large-scale models, building image generation systems, or deploying real-time inference services, selecting the right storage architecture is critical for performance, stability, and cost optimization.

Storage comparison

General Selection Logic:

1. 1. Assess Your Workload's Needs: If you are only running inference without needing to download data or save results, additional storage may not be required. Model training or fine-tuning typically requires a combination, such as local and object storage. Distributed tasks demand shared storage.
2. 2. Use Local Storage for High-Frequency, Low-Latency Access: Prioritize local storage for the primary data stream during the training phase to leverage its extremely low latency and high throughput.
3. 3. Use Shared Storage for Mid-Frequency, Collaborative Access: Choose shared storage to ensure synchronous data access across multiple processes or nodes, ideal for collaborative tasks and shared datasets.
4. 4. Use Object Storage for Low-Frequency Access and Archiving: Use object or block storage for long-term preservation of model versions, training logs, and other assets. It is cost-effective and suited for "write-once, read-many" scenarios.

1. Using Scene Differences

• Shared Storage is mounted and accessed as a standard file system (NAS), allowing all GPU nodes to read and write to the same directory concurrently. Canopy Wave's shared storage is built on CephFS, a scalable, POSIX-compliant distributed file system designed for high availability and performance. It is ideal for high-frequency access to shared datasets, logs, and intermediate models during distributed training.
• Object Storage: Network transmission through the S3 protocol does not occupy GPU resources, and is suitable for efficient processing and transmission of large-scale data. It is commonly used to store raw datasets, model archiving, and upload training data from the public network or locally, and can also be used for model deployment, archiving, and migration after training is completed. In distributed training, object storage is often used in conjunction with other types of storage to balance performance and flexibility.

2. Comparison of technical dimensions

A Combined Approach for Distributed Training:

1. 1. Before Training: Datasets, pre-trained weights, and configuration files are stored in object storage (S3).
2. 2. During Training: The processed data is pulled from object storage to high-performance shared storage. All GPU nodes mount this shared directory for real-time, concurrent access. Intermediate checkpoints and logs are also written to shared storage due to its low latency.
3. 3. After Training: The final model, logs, and metric reports are pushed back from shared storage to object storage for long-term archiving, versioning, and downstream use (e.g., deployment, evaluation).

This combination is a best practice, leveraging the strengths of each system for maximum efficiency.

In a GPU cluster, local storage is the directly attached storage (DAS) inside each server. At Canopy Wave, we use high-performance NVMe drives.

Advantages:

1. 1. Extreme Performance: Connected via the PCIe bus, local NVMe storage offers ultra-low latency and high throughput (3-7 GB/s), bypassing network bottlenecks entirely. It is ideal for data-intensive tasks like loading large media files or caching frequently accessed training data.
2. 2. Resource Isolation: Each server's local storage is exclusive, eliminating the "noisy neighbor" problem of I/O congestion or lock contention found in shared systems.
3. 3. Architectural Simplicity: It requires no complex configuration of shared file systems, making it suitable for rapid deployment and agile development scenarios.
4. 4. Cost-Effectiveness: The cost per gigabyte for NVMe is often significantly lower than that of enterprise-grade shared storage systems.

Disadvantages:

1. 1. No Data Sharing: Data on a local drive is isolated to that server, making it unsuitable as a primary data source for distributed training where all nodes need a unified view of the data.
2. 2. Lack of Data Redundancy: Local storage typically lacks built-in redundancy. A server failure or instance termination can lead to permanent data loss, making it unsuitable for long-term assets.
3. 3. Limited & Inelastic Capacity: Local disk capacity is fixed and cannot be dynamically expanded like object or shared storage.
4. 4. Scheduling Complexity: Schedulers must account for data locality, which can lead to inefficient resource utilization and frequent data transfers if not managed properly.

Conclusion: Local storage excels as a high-speed cache, a temporary working directory for data preprocessing, or for low-latency inference tasks. However, it should not be the sole storage solution. In robust AI pipelines, it is best used as a first-level cache in a tiered architecture, combined with shared and object storage to create a stable and efficient system.