locopy.s3 module¶

S3 Module Module to wrap the boto3 api usage and provide functionality to manage multipart upload to S3 buckets

class locopy.s3.S3(profile=None, kms_key=None, **kwargs)[source]¶

Bases: object

S3 wrapper class which utilizes the boto3 library to push files to an S3 bucket.

Parameters:

profile (str, optional) – The name of the AWS profile to use which is typical stored in the credentials file. You can also set environment variable AWS_DEFAULT_PROFILE which would be used instead.
kms_key (str, optional) – The KMS key to use for encryption If kms_key Defaults to None then the AES256 ServerSideEncryption will be used.
**kwargs – Optional keyword arguments.

profile¶

String representing the AWS profile for authentication

Type:: str

kms_key¶

String representing the s3 kms key

Type:: str

session¶

Hold the AWS session credentials / info

Type:: boto3.Session

s3¶

Hold the S3 client object which is used to upload/delete files to S3

Type:: botocore.client.S3

Raises:

S3Error – Error initializing AWS Session (ex: invalid profile)
S3CredentialsError – Issue with AWS credentials
S3InitializationError – Issue initializing S3 session

delete_from_s3(bucket, key)[source]¶

Delete a file from an S3 bucket.

Parameters:

bucket (str) – The AWS S3 bucket from which you are deleting the file.
key (str) – The name of the S3 object.

Raises:

S3DeletionError – If there is a issue deleting from the S3 bucket

delete_list_from_s3(s3_list)[source]¶

Delete a list of files from an S3 bucket.

Parameters:: s3_list (list) – List of strings with the s3 paths of the files to delete. The strings should not include the s3:// scheme.

download_from_s3(bucket, key, local)[source]¶

Download a file from a S3 bucket.

Parameters:

bucket (str) – The AWS S3 bucket which you are copying the local file to.
key (str) – The key to name the S3 object.
local (str) – The local file which you wish to copy to.

Raises:

S3DownloadError – If there is a issue downloading to the S3 bucket

download_list_from_s3(s3_list, local_path=None)[source]¶

Download a list of files from s3.

Parameters:

s3_list (list) – List of strings with the s3 paths of the files to download
local_path (str, optional) – The local path where the files will be copied to. Defualts to the current working directory (os.getcwd())

Returns:

Returns a list of strings of the local file names

Return type:

list

parse_s3_url(s3_url)[source]¶

Parse a string of the s3 url to extract the bucket and key. scheme or not.

Parameters:

s3_url (str) – s3 url. The string can include the s3:// scheme (which is disgarded)

Returns:

bucket (str) – s3 bucket
key (str) – s3 key

upload_list_to_s3(local_list, bucket, folder=None)[source]¶

Upload a list of files to a S3 bucket.

Parameters:

local_list (list) – List of strings with the file paths of the files to upload
bucket (str) – The AWS S3 bucket which you are copying the local file to.
folder (str, optional) – The AWS S3 folder of the bucket which you are copying the local files to. Defaults to None. Please note that you must follow the / convention when using subfolders.

Returns:

Returns a list of the generated S3 bucket and keys of the files which were uploaded. The S3:// part is NOT include. The output would look like the following: ["my-bucket/key1", "my-bucket/key2", ...]

Return type:

list

Notes

There is a assumption that if you are loading multiple files (via splits) it follows a structure such as file_name.extension.# (# splits). It allows for the COPY statement to use the key prefix vs specificing an exact file name. The returned list helps with this process downstream.

upload_to_s3(local, bucket, key)[source]¶

Upload a file to a S3 bucket.

Parameters:

local (str) – The local file which you wish to copy.
bucket (str) – The AWS S3 bucket which you are copying the local file to.
key (str) – The key to name the S3 object.

Raises:

S3UploadError – If there is a issue uploading to the S3 bucket