Testing data transfer speed with Warp

​

Prerequisites

1. Install Warp:
   wget https://dl.min.io/aistor/warp/release/linux-amd64/warp
   chmod +x warp
   
2. Create an Object Storage bucket.
3. Make sure that the project has enough Object Storage quota for about 20 TiB of test data.
4. Create an access key for a service account that can read and upload objects in the bucket. For example, assign the service account the storage.editor role for the bucket or its parent project. You need its AWS-like ID and secret key to generate the test dataset. When you create the access key by using the CLI, the AWS-like ID and secret key are specified in the .status.aws_access_key_id and .status.secret parameters of the output. To create an access key, make sure you are in a group that has the admin role.

​

How to prepare a test dataset

1. Set the bucket endpoint, name and credentials as environment variables, and specify the ID of the region that contains the bucket:
   export S3_ENDPOINT="storage.<region_ID>.nebius.cloud:443"
   export S3_BUCKET="<bucket_name>"
   export S3_ACCESS_KEY="<AWS-like_ID>"
   export S3_SECRET_KEY="<secret_key>"
   
2. Generate the test dataset. Use the --noclear parameter to keep the generated objects in the bucket after the test:
   ./warp get \
      --host "$S3_ENDPOINT" \
      --tls \
      --bucket "$S3_BUCKET" \
      --access-key "$S3_ACCESS_KEY" \
      --secret-key "$S3_SECRET_KEY" \
      --objects 20000 \
      --obj.size 1GiB \
      --duration 1m \
      --storage-class ENHANCED_THROUGHPUT \
      --concurrent 32 \
      --noclear
   
   This command generates a test dataset of 20,000 objects, each 1 GiB in size, and keeps it in the bucket specified in the environment variables. The Enhanced Throughput storage class provides the best transfer speed.

​

How to test data transfer speed

1. Create and launch a one-time data transfer from the test bucket.
2. While the data transfer is running, monitor the test bucket. Use the Traffic metric, measured in bytes per second, to observe the transfer speed.