Object Storage Service

Products

Object Storage Service

Managing JD Cloud OSS via S3cmd

Introduction

S3cmd is a free command line tool client, used for uploading, retrieving and managing data on Amazon S3 and other cloud storage service providers (as JD Cloud OSS) using the S3 protocol. It is suitable to the senior users familiar with the command line program. It is also the ideal choice for batch processing of scripts and S3 automatic backup, and is triggered by cron, etc.

Customer Scenario: API-level development and debugging

Environment Requirements

S3cmd environment requirement: S3cmd can run on Linux (Fedora, RedHat, SuSe, Ubuntu, Debian, etc.) and Apple MAC. Python 2.6 or higher revision is required. S3cmd version 2 also is compatible to Python 3.x.

Python revision view: Input python in the Linux shell environment and enter to view the Python version. The version is shown as follows:

Python 2.7.12 (default, Dec  4 2017, 14:50:18) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

The python is successfully installed when the above indications are given. If the Python is not installed, it may obtain the Python installation package from the official website of python. Specific installation instructions on how to install and use Python are provided by the web service.

Installation and Usage

Download S3cmd

git clone https://github.com/jdcloud-cmw/s3cmd.git

S3cmd Installation

sudo cp -rf s3cmd/ /usr/local/
sudo ln -s /usr/local/s3cmd/s3cmd /usr/bin/s3cmd

Configuration File

The configuration file is ~/.s3cfg and the content of used configuration file is as follows:

[default]
access_key = [you access key]
secret_key = [you secret key]
access_token = 
add_encoding_exts = 
add_headers = 
bucket_location = cn
ca_certs_file = 
cache_file = 
check_ssl_certificate = True
check_ssl_hostname = True
cloudfront_host = cloudfront.amazonaws.com
default_mime_type = binary/octet-stream
delay_updates = False
delete_after = False
delete_after_fetch = False
delete_removed = False
dry_run = False
enable_multipart = True
encrypt = False
expiry_date = 
expiry_days = 
expiry_prefix = 
follow_symlinks = False
force = False
get_continue = False
gpg_command = /usr/bin/gpg
gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_passphrase = 
guess_mime_type = True
host_base = s3.cn-north-1.jcloudcs.com
host_bucket = %(bucket)s.s3.cn-north-1.jcloudcs.com
human_readable_sizes = False
invalidate_default_index_on_cf = False
invalidate_default_index_root_on_cf = True
invalidate_on_cf = False
kms_key = 
limit = -1
limitrate = 0
list_md5 = False
log_target_prefix = /home/eric/jcloud/s3.log
long_listing = False
max_delete = -1
mime_type = 
multipart_chunk_size_mb = 15
multipart_max_chunks = 10000
preserve_attrs = True
progress_meter = True
proxy_host = 
proxy_port = 0
put_continue = False
recursive = False
recv_chunk = 65536
reduced_redundancy = False
requester_pays = False
restore_days = 1
restore_priority = Standard
send_chunk = 65536
server_side_encryption = False
signature_v2 = False
simpledb_host = sdb.amazonaws.com
skip_existing = False
socket_timeout = 300
stats = False
stop_on_error = False
storage_class = 
urlencoding_mode = normal
use_http_expect = False
use_https = True
use_mime_magic = True
verbosity = WARNING
website_endpoint = http://%(bucket)s.s3.%(location)s.amazonaws.com/
website_error = 
website_index = index.html

Check if S3cmd is available for use

Input s3cmd and enter under the Linux shell environment and the use is available when the following indication is given:

Taking Make bucket as example, test whether S3cmd is available:

Supported Functions

Commands supported by S3cmd to JD Cloud OSS are listed as follows:

Commands:

Function Type Function Command
Manager Bucket Make bucket
s3cmd mb s3://BUCKET
Remove bucket
s3cmd rb s3://BUCKET
List objects or buckets
s3cmd ls [s3://BUCKET[/PREFIX]]
Disk usage by buckets
s3cmd du [s3://BUCKET[/PREFIX]]
Modify Access control list for Bucket or Files
s3cmd setacl s3://BUCKET[/OBJECT]
Modify Bucket Policy
s3cmd setpolicy FILE s3://BUCKET
Delete Bucket Policy
s3cmd delpolicy s3://BUCKET
Modify Bucket CORS
s3cmd setcors FILE s3://BUCKET
Delete Bucket CORS
s3cmd delcors s3://BUCKET
Create Website from bucket
s3cmd ws-create s3://BUCKET
Delete Website
s3cmd ws-delete s3://BUCKET
Info about Website
s3cmd ws-info s3://BUCKET
Manager Object Put file into bucket
s3cmd put FILE [FILE...]s3://BUCKET[/PREFIX]
Get file from bucket
s3cmd get s3://BUCKET/OBJECT LOCAL_FILE
Delete file from bucket
s3cmd del s3://BUCKET/OBJECT
Delete file from bucket (alias for del)
s3cmd rm s3://BUCKET/OBJECT
Show multipart uploads
s3cmd multipart s3://BUCKET [Id]
Abort a multipart upload
s3cmd abortmp s3://BUCKET/OBJECT Id
List parts of a multipart upload
s3cmd listmp s3://BUCKET/OBJECT Id
Other Functions Sign arbitrary string using the secret key
s3cmd sign STRING-TO-SIGN
Get various information about Buckets or Files
s3cmd info s3://BUCKET[/OBJECT]
Note: The bucket information display is not supported here and the displayed object information is incomplete.
Synchronize a directory tree to S3 (checks files freshness using size and md5 checksum, unless overridden by options, see below)
s3cmd sync LOCAL_DIR s3://BUCKET[/PREFIX] or s3://BUCKET[/PREFIX] LOCAL_DIR
Note: The automatic synchronization will not be made and please input this command to synchronize a catalog.
Sign an S3 URL to provide limited public access with expiry
s3cmd signurl s3://BUCKET/OBJECT < expiry_epoch|+expiry_offset >
Update Time:2019-05-28 16:47:19