: Data Migration Tool Osstransfer

Function description

Osstransfer tool can migrate data stored locally and data stored by other objects to OSS, with the following characteristics:

• Support rich data sources:
  • Local Data: Migrate data stored locally to OSS;
  • Other Object Storage Service: Currently support AWS S3, migrate Alibaba Cloud OSS, Tencent Cloud OSS, Baidu OSS, Huawei OBS and Qiniu Cloud Storage to JD Cloud OSS, which will be expanded continuously;
  • URL List: Download according to URL download list according to designated URL download list, migrate it to JD Cloud OSS;
  • Mutual Bucket Replication: Bucket data of JD Cloud OSS are replicated mutually to support data replication across regions, across accounts and in the same region.
• Support breakpoint upload;
• Support traffic control;
• Support migrating the file with specific prefix;
• Support parallel data download and upload;
• Migration verification: verification after object migration.

Use Environment

System Environment

• linux Environment

Software Dependence

• jdk1.8

Use Method

1. Get tool

Download link: transfer-tools

2. Get Configuration File

Create or download configuration file under the same directory application.yml Download link: [Example yml File](https://downloads.oss.cn-north-1.jcloudcs.com/application.yml

3. Modify application.yml configuration file

Before performing migration start script, you are required to modify application.yml configuration file according to your own demands.

3.1 The fields descriptions of Osstransfer configuration file are shown as below:

3.2 application.yml example

3.3.1 get file list (jobType: listObject)

3.3.1.1 listS3, get AWS S3, Tencent Cloud COS, Baidu BOS, Huawei OBS, JD Cloud OSS

jobType: listObject
sourceType: s3file
src.access.id : XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
src.secret.key: YYYYYYYYYYYYYYYYYYYYYYYYYYYYY
src.endpoint : https://s3.cn-north-1.jdcloud-oss.com
src.bucket : yourbucket
src.prefix :

jobType: listObject
sourceType: aliyunfile
src.access.id : AAAAAAAAAAAAAAAAAAAAAAAAA
src.secret.key: BBBBBBBBBBBBBBBBBBBBBBBBB
src.endpoint : https://oss-cn-beijing.aliyuncs.com
src.bucket : yourbucket
src.prefix :

jobType: listObject
sourceType: diskfile
filePath: /yourpath

3.3.2.1 Migrate from AWS S3, Tencent Cloud COS, Baidu BOS, Huawei OBS to JD Cloud OSS

jobType: transfer
sourceType: s3file

src.access.id : XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
src.secret.key: YYYYYYYYYYYYYYYYYYYYYYYYYYYYY
src.endpoint : https://s3.cn-north-1.jdcloud-oss.com
src.bucket : yourbucket
src.prefix :

des.access.id : AAAAAAAAAAAAAAAAAAAAAAAAAAAA
des.secret.key: BBBBBBBBBBBBBBBBBBBBBBBBBBBB
des.endpoint : https://s3.cn-north-1.jdcloud-oss.com
des.bucket : yourbucket
des.prefix:

#Optional Field

task.limit.threadCount: 20
task.limit.qps: 50

transfer.coverFile: true
transfer.put.maxsize: 33554432
transfer.multipart.partsize: 33554432
transfer.multipart.threads: 5

jobType: transfer
sourceType: aliyunfile

src.access.id : XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
src.secret.key: YYYYYYYYYYYYYYYYYYYYYYYYYYYYY
src.endpoint : https://oss-cn-beijing.aliyuncs.com
src.bucket : yourbucket
src.prefix :

des.access.id : AAAAAAAAAAAAAAAAAAAAAAAAAAA
des.secret.key: BBBBBBBBBBBBBBBBBBBBBBBBBBB
des.endpoint : https://s3.cn-north-1.jdcloud-oss.com
des.bucket : yourbucket
des.prefix:

#Optional Field

task.limit.threadCount: 20
task.limit.qps: 50

transfer.coverFile: true
transfer.put.maxsize: 33554432
transfer.multipart.partsize: 33554432
transfer.multipart.threads: 5

3.3.2.3 Migrate locally to JD Cloud OSS

jobType: transfer
sourceType: diskfile

filePath: /yourpath

 

des.access.id : AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
des.secret.key: BBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
des.endpoint :https://s3.cn-north-1.jdcloud-oss.com
des.bucket : yourbucket
des.prefix:

urlFilePrefix: 1

 #urlFilePrefix is set as 1, because if file system key is started with "/", then JD Cloud OSS does not support

jobType: transfer
sourceType: urlfile
filePath: /path/onlyurl.txt
urlType: onlyUrl
urlFilePrefix: 35

des.access.id : AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
des.secret.key: BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
des.endpoint : https://s3.cn-north-1.jdcloud-oss.com
des.bucket : yourbucket

3.3.2.5 Configure JD Cloud Bucket Mutual Replication

If it migrates from one designated JD Cloud OSS to another Bucket, the designated value of sourceType is s3file.

jobType: transfer
sourceType: s3file

src.access.id : XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
src.secret.key: YYYYYYYYYYYYYYYYYYYYYYYYYYYYY
src.endpoint : https://s3.cn-north-1.jdcloud-oss.com
src.bucket : yourbucket
src.prefix :

 

des.access.id : AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
des.secret.key: BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
des.endpoint : https://s3.cn-north-1.jdcloud-oss.com
des.bucket : yourbucket
des.prefix:

4. Running Migration Tool

Linux

java -jar transfer-tools-java-1.0.0.jar --Dspring.config.location=application.yml

1. Osstransfer Migration Tool is generally divided into three types of jobs, including listObject, transfer, respectively. During migration, each file put uploaded and multipart uploaded is subject to md5 comparison.

2. The single file in file migration is 19.5T at largest.

3. When implementing Data Migration Service from the cloud vendor, only the permission of public reading of the source Bucket is supported; otherwise, it will result in migration failure.

4. The maximum length of JD Cloud OSS is 1022 bytes. Therefore, files larger than 1022 bytes cannot be migrated.

5. AWS S3 endpoint only supports https.

6. object key includes char(10) line feed, char(13) enter does not migrate.

7. The maximum of JD Cloud OSS Content-Disposition is 100 bytes, it is recommended to set ContentDispositionTooLongContinue configuration item as true when using Osstransfer, In this way, when Content-Disposition exceeds 100 bytes, it is guaranteed that migration of files is successful.

Migration Principles and Process

Principle Description

Osstransfer Migration Tool uses sdk first to implement list for each data source to get the list of object. The goal is that if there are object changes during the migration, the migration tool will not be affected.

Migration Process

1. During the migration process, the migration log will be printed to the./log directory by default.

All migrated files will be printed to audit-0.log, the files that migrated successfully will be printed to audit.success log, if the files that fail in migration need to be screened out, please use the command:

grep "1$" audit-0.log*

2. Audit Log Description