Managing JD Cloud OSS via S3cmd
Introduction
S3cmd is a free command line tool client, used for uploading, retrieving and managing data on Amazon S3 and other cloud storage service providers (as JD Cloud OSS) using the S3 protocol. It is suitable to the senior users familiar with the command line program. It is also the ideal choice for batch processing of scripts and S3 automatic backup, and is triggered by cron, etc.
Customer Scenario: API-level development and debugging
Environment Requirements
S3cmd environment requirement: S3cmd can run on Linux (Fedora, RedHat, SuSe, Ubuntu, Debian, etc.) and Apple MAC. Python 2.6 or higher revision is required. S3cmd version 2 also is compatible to Python 3.x.
Python revision view: Input python in the Linux shell environment and enter to view the Python version. The version is shown as follows:
Python 2.7.12 (default, Dec 4 2017, 14:50:18) [GCC 5.4.0 20160609] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>>
The python is successfully installed when the above indications are given. If the Python is not installed, it may obtain the Python installation package from the official website of python. Specific installation instructions on how to install and use Python are provided by the web service.
Installation and Usage
Download S3cmd
git clone https://github.com/jdcloud-cmw/s3cmd.git
S3cmd Installation
sudo cp -rf s3cmd/ /usr/local/ sudo ln -s /usr/local/s3cmd/s3cmd /usr/bin/s3cmd
Configuration File
The configuration file is ~/.s3cfg and the content of used configuration file is as follows:
[default] access_key = [you access key] secret_key = [you secret key] access_token = add_encoding_exts = add_headers = bucket_location = cn ca_certs_file = cache_file = check_ssl_certificate = True check_ssl_hostname = True cloudfront_host = cloudfront.amazonaws.com default_mime_type = binary/octet-stream delay_updates = False delete_after = False delete_after_fetch = False delete_removed = False dry_run = False enable_multipart = True encrypt = False expiry_date = expiry_days = expiry_prefix = follow_symlinks = False force = False get_continue = False gpg_command = /usr/bin/gpg gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s gpg_passphrase = guess_mime_type = True host_base = s3.cn-north-1.jcloudcs.com host_bucket = %(bucket)s.s3.cn-north-1.jcloudcs.com human_readable_sizes = False invalidate_default_index_on_cf = False invalidate_default_index_root_on_cf = True invalidate_on_cf = False kms_key = limit = -1 limitrate = 0 list_md5 = False log_target_prefix = /home/eric/jcloud/s3.log long_listing = False max_delete = -1 mime_type = multipart_chunk_size_mb = 15 multipart_max_chunks = 10000 preserve_attrs = True progress_meter = True proxy_host = proxy_port = 0 put_continue = False recursive = False recv_chunk = 65536 reduced_redundancy = False requester_pays = False restore_days = 1 restore_priority = Standard send_chunk = 65536 server_side_encryption = False signature_v2 = False simpledb_host = sdb.amazonaws.com skip_existing = False socket_timeout = 300 stats = False stop_on_error = False storage_class = urlencoding_mode = normal use_http_expect = False use_https = True use_mime_magic = True verbosity = WARNING website_endpoint = http://%(bucket)s.s3.%(location)s.amazonaws.com/ website_error = website_index = index.html
Check if S3cmd is available for use
Input s3cmd and enter under the Linux shell environment and the use is available when the following indication is given:
Taking Make bucket as example, test whether S3cmd is available:
Supported Functions
Commands supported by S3cmd to JD Cloud OSS are listed as follows:
Commands：
| Function Type | Function Command |
| Manager Bucket | Make bucket s3cmd mb s3://BUCKET |
| Remove bucket s3cmd rb s3://BUCKET |
|
| List objects or buckets s3cmd ls [s3://BUCKET[/PREFIX]] |
|
| Disk usage by buckets s3cmd du [s3://BUCKET[/PREFIX]] |
|
| Modify Access control list for Bucket or Files s3cmd setacl s3://BUCKET[/OBJECT] |
|
| Modify Bucket Policy s3cmd setpolicy FILE s3://BUCKET |
|
| Delete Bucket Policy s3cmd delpolicy s3://BUCKET |
|
| Modify Bucket CORS s3cmd setcors FILE s3://BUCKET |
|
| Delete Bucket CORS s3cmd delcors s3://BUCKET |
|
| Create Website from bucket s3cmd ws-create s3://BUCKET |
|
| Delete Website s3cmd ws-delete s3://BUCKET |
|
| Info about Website s3cmd ws-info s3://BUCKET |
|
| Manager Object | Put file into bucket s3cmd put FILE [FILE...]s3://BUCKET[/PREFIX] |
| Get file from bucket s3cmd get s3://BUCKET/OBJECT LOCAL_FILE |
|
| Delete file from bucket s3cmd del s3://BUCKET/OBJECT |
|
| Delete file from bucket (alias for del) s3cmd rm s3://BUCKET/OBJECT |
|
| Show multipart uploads s3cmd multipart s3://BUCKET [Id] |
|
| Abort a multipart upload s3cmd abortmp s3://BUCKET/OBJECT Id |
|
| List parts of a multipart upload s3cmd listmp s3://BUCKET/OBJECT Id |
|
| Other Functions | Sign arbitrary string using the secret key s3cmd sign STRING-TO-SIGN |
| Get various information about Buckets or Files s3cmd info s3://BUCKET[/OBJECT] Note: The bucket information display is not supported here and the displayed object information is incomplete. |
|
| Synchronize a directory tree to S3 (checks files freshness using size and md5 checksum, unless overridden by options, see below) s3cmd sync LOCAL_DIR s3://BUCKET[/PREFIX] or s3://BUCKET[/PREFIX] LOCAL_DIR Note: The automatic synchronization will not be made and please input this command to synchronize a catalog. |
|
| Sign an S3 URL to provide limited public access with expiry s3cmd signurl s3://BUCKET/OBJECT < expiry_epoch|+expiry_offset > |