s3fs: 'Amazon Web Service S3' File System

Description

Access 'Amazon Web Service Simple Storage Service' ('S3') https://aws.amazon.com/s3/ as if it were a file system. Interface based on the R package 'fs'.

Author(s)

Maintainer: Dyfan Jones dyfan.r.jones@gmail.com

See Also

Useful links:

• https://github.com/DyfanJones/s3fs

• Report bugs at https://github.com/DyfanJones/s3fs/issues

Access AWS S3 as if it were a file system.

Description

This creates a file system "like" API based off fs (e.g. dir_ls, file_copy, etc.) for AWS S3 storage.

Public fields

Active bindings

Methods

Public methods

• S3FileSystem$new()

• S3FileSystem$file_chmod()

• S3FileSystem$file_copy()

• S3FileSystem$file_create()

• S3FileSystem$file_delete()

• S3FileSystem$file_download()

• S3FileSystem$file_exists()

• S3FileSystem$file_info()

• S3FileSystem$file_move()

• S3FileSystem$file_size()

• S3FileSystem$file_stream_in()

• S3FileSystem$file_stream_out()

• S3FileSystem$file_temp()

• S3FileSystem$file_tag_delete()

• S3FileSystem$file_tag_info()

• S3FileSystem$file_tag_update()

• S3FileSystem$file_touch()

• S3FileSystem$file_upload()

• S3FileSystem$file_url()

• S3FileSystem$file_version_info()

• S3FileSystem$is_file()

• S3FileSystem$is_dir()

• S3FileSystem$is_bucket()

• S3FileSystem$is_file_empty()

• S3FileSystem$bucket_chmod()

• S3FileSystem$bucket_create()

• S3FileSystem$bucket_delete()

• S3FileSystem$dir_copy()

• S3FileSystem$dir_create()

• S3FileSystem$dir_delete()

• S3FileSystem$dir_exists()

• S3FileSystem$dir_download()

• S3FileSystem$dir_info()

• S3FileSystem$dir_ls()

• S3FileSystem$dir_ls_url()

• S3FileSystem$dir_tree()

• S3FileSystem$dir_upload()

• S3FileSystem$path()

• S3FileSystem$path_dir()

• S3FileSystem$path_ext()

• S3FileSystem$path_ext_remove()

• S3FileSystem$path_ext_set()

• S3FileSystem$path_file()

• S3FileSystem$path_join()

• S3FileSystem$path_split()

• S3FileSystem$clear_cache()

• S3FileSystem$clone()

Method new()

Initialize S3FileSystem class

Usage

S3FileSystem$new(
  aws_access_key_id = NULL,
  aws_secret_access_key = NULL,
  aws_session_token = NULL,
  region_name = NULL,
  profile_name = NULL,
  endpoint = NULL,
  disable_ssl = FALSE,
  multipart_threshold = fs_bytes("2GB"),
  request_payer = FALSE,
  anonymous = FALSE,
  ...
)

Arguments

Method file_chmod()

Change file permissions

Usage

S3FileSystem$file_chmod(
  path,
  mode = c("private", "public-read", "public-read-write", "authenticated-read",
    "aws-exec-read", "bucket-owner-read", "bucket-owner-full-control")
)

Arguments

Returns

character vector of s3 uri paths

Method file_copy()

copy files

Usage

S3FileSystem$file_copy(
  path,
  new_path,
  max_batch = fs_bytes("100MB"),
  overwrite = FALSE,
  ...
)

Arguments

Returns

character vector of s3 uri paths

Method file_create()

Create file on AWS S3, if file already exists it will be left unchanged.

Usage

S3FileSystem$file_create(path, overwrite = FALSE, ...)

Arguments

Returns

character vector of s3 uri paths

Method file_delete()

Delete files in AWS S3

Usage

S3FileSystem$file_delete(path, ...)

Arguments

Returns

character vector of s3 uri paths

Method file_download()

Downloads AWS S3 files to local

Usage

S3FileSystem$file_download(path, new_path, overwrite = FALSE, ...)

Arguments

Returns

character vector of s3 uri paths

Method file_exists()

Check if file exists in AWS S3

Usage

S3FileSystem$file_exists(path)

Arguments

Returns

logical vector if file exists

Method file_info()

Returns file information within AWS S3 directory

Usage

S3FileSystem$file_info(path)

Arguments

Returns

A data.table with metadata for each file. Columns returned are as follows.

• bucket_name (character): AWS S3 bucket of file

• key (character): AWS S3 path key of file

• uri (character): S3 uri of file

• size (numeric): file size in bytes

• type (character): file type (file or directory)

• etag (character): An entity tag is an opague identifier

• last_modified (POSIXct): Created date of file.

• delete_marker (logical): Specifies retrieved a logical marker

• accept_ranges (character): Indicates that a range of bytes was specified.

• expiration (character): File expiration

• restore (character): If file is archived

• archive_status (character): Archive status

• missing_meta (integer): Number of metadata entries not returned in "x-amz-meta" headers

• version_id (character): version id of file

• cache_control (character): caching behaviour for the request/reply chain

• content_disposition (character): presentational information of file

• content_encoding (character): file content encodings

• content_language (character): what language the content is in

• content_type (character): file MIME type

• expires (POSIXct): date and time the file is no longer cacheable

• website_redirect_location (character): redirects request for file to another

• server_side_encryption (character): File server side encryption

• metadata (list): metadata of file

• sse_customer_algorithm (character): server-side encryption with a customer-provided encryption key

• sse_customer_key_md5 (character): server-side encryption with a customer-provided encryption key

• ssekms_key_id (character): ID of the Amazon Web Services Key Management Service

• bucket_key_enabled (logical): s3 bucket key for server-side encryption with

• storage_class (character): file storage class information

• request_charged (character): indicates successfully charged for request

• replication_status (character): return specific header if request involves a bucket that is either a source or a destination in a replication rule https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.head_object

• parts_count (integer): number of count parts the file has

• object_lock_mode (character): the file lock mode

• object_lock_retain_until_date (POSIXct): date and time of when object_lock_mode expires

• object_lock_legal_hold_status (character): file legal holding

Method file_move()

Move files to another location on AWS S3

Usage

S3FileSystem$file_move(
  path,
  new_path,
  max_batch = fs_bytes("100MB"),
  overwrite = FALSE,
  ...
)

Arguments

Returns

character vector of s3 uri paths

Method file_size()

Return file size in bytes

Usage

S3FileSystem$file_size(path)

Arguments

Method file_stream_in()

Streams in AWS S3 file as a raw vector

Usage

S3FileSystem$file_stream_in(path, ...)

Arguments

Returns

list of raw vectors containing the contents of the file

Method file_stream_out()

Streams out raw vector to AWS S3 file

Usage

S3FileSystem$file_stream_out(
  obj,
  path,
  max_batch = fs_bytes("100MB"),
  overwrite = FALSE,
  ...
)

Arguments

Returns

character vector of s3 uri paths

Method file_temp()

return the name which can be used as a temporary file

Usage

S3FileSystem$file_temp(pattern = "file", tmp_dir = "", ext = "")

Arguments

Returns

character vector of s3 uri paths

Method file_tag_delete()

Delete file tags

Usage

S3FileSystem$file_tag_delete(path)

Arguments

Returns

character vector of s3 uri paths

Method file_tag_info()

Get file tags

Usage

S3FileSystem$file_tag_info(path)

Arguments

Returns

data.table of file version metadata

• bucket_name (character): AWS S3 bucket of file

• key (character): AWS S3 path key of file

• uri (character): S3 uri of file

• size (numeric): file size in bytes

• version_id (character): version id of file

• tag_key (character): name of tag

• tag_value (character): tag value

Method file_tag_update()

Update file tags

Usage

S3FileSystem$file_tag_update(path, tags, overwrite = FALSE)

Arguments

Returns

character vector of s3 uri paths

Method file_touch()

Similar to fs::file_touch this does not create the file if it does not exist. Use s3fs$file_create() to do this if needed.

Usage

S3FileSystem$file_touch(path, ...)

Arguments

Returns

character vector of s3 uri paths

Method file_upload()

Uploads files to AWS S3

Usage

S3FileSystem$file_upload(
  path,
  new_path,
  max_batch = fs_bytes("100MB"),
  overwrite = FALSE,
  ...
)

Arguments

Returns

character vector of s3 uri paths

Method file_url()

Generate presigned url for S3 object

Usage

S3FileSystem$file_url(path, expiration = 3600L, ...)

Arguments

Returns

return character of urls

Method file_version_info()

Get file versions

Usage

S3FileSystem$file_version_info(path, ...)

Arguments

Returns

return data.table with file version info, columns below:

• bucket_name (character): AWS S3 bucket of file

• key (character): AWS S3 path key of file

• uri (character): S3 uri of file

• size (numeric): file size in bytes

• version_id (character): version id of file

• owner (character): file owner

• etag (character): An entity tag is an opague identifier

• last_modified (POSIXct): Created date of file.

Method is_file()

Test for file types

Usage

S3FileSystem$is_file(path)

Arguments

Returns

logical vector if object is a file

Method is_dir()

Test for file types

Usage

S3FileSystem$is_dir(path)

Arguments

Returns

logical vector if object is a directory

Method is_bucket()

Test for file types

Usage

S3FileSystem$is_bucket(path, ...)

Arguments

Returns

logical vector if object is a ⁠AWS S3⁠ bucket

Method is_file_empty()

Test for file types

Usage

S3FileSystem$is_file_empty(path)

Arguments

Returns

logical vector if file is empty

Method bucket_chmod()

Change bucket permissions

Usage

S3FileSystem$bucket_chmod(
  path,
  mode = c("private", "public-read", "public-read-write", "authenticated-read")
)

Arguments

Returns

character vector of s3 uri paths

Method bucket_create()

Create bucket

Usage

S3FileSystem$bucket_create(
  path,
  region_name = NULL,
  mode = c("private", "public-read", "public-read-write", "authenticated-read"),
  versioning = FALSE,
  ...
)

Arguments

Returns

character vector of s3 uri paths

Method bucket_delete()

Delete bucket

Usage

S3FileSystem$bucket_delete(path)

Arguments

Method dir_copy()

Copies the directory recursively to the new location.

Usage

S3FileSystem$dir_copy(
  path,
  new_path,
  max_batch = fs_bytes("100MB"),
  overwrite = FALSE,
  ...
)

Arguments

Returns

character vector of s3 uri paths

Method dir_create()

Create empty directory

Usage

S3FileSystem$dir_create(path, overwrite = FALSE, ...)

Arguments

Returns

character vector of s3 uri paths

Method dir_delete()

Delete contents and directory in AWS S3

Usage

S3FileSystem$dir_delete(path)

Arguments

Returns

character vector of s3 uri paths

Method dir_exists()

Check if path exists in AWS S3

Usage

S3FileSystem$dir_exists(path = ".")

Arguments

Returns

character vector of s3 uri paths

Method dir_download()

Downloads AWS S3 files to local

Usage

S3FileSystem$dir_download(path, new_path, overwrite = FALSE, ...)

Arguments

Returns

character vector of s3 uri paths

Method dir_info()

Returns file information within AWS S3 directory

Usage

S3FileSystem$dir_info(
  path = ".",
  type = c("any", "bucket", "directory", "file"),
  glob = NULL,
  regexp = NULL,
  invert = FALSE,
  recurse = FALSE,
  refresh = FALSE,
  ...
)

Arguments

Returns

data.table with directory metadata

• bucket_name (character): AWS S3 bucket of file

• key (character): AWS S3 path key of file

• uri (character): S3 uri of file

• size (numeric): file size in bytes

• version_id (character): version id of file

• etag (character): An entity tag is an opague identifier

• last_modified (POSIXct): Created date of file

Method dir_ls()

Returns file name within AWS S3 directory

Usage

S3FileSystem$dir_ls(
  path = ".",
  type = c("any", "bucket", "directory", "file"),
  glob = NULL,
  regexp = NULL,
  invert = FALSE,
  recurse = FALSE,
  refresh = FALSE,
  ...
)

Arguments

Returns

character vector of s3 uri paths

Method dir_ls_url()

Generate presigned url to list S3 directories

Usage

S3FileSystem$dir_ls_url(path, expiration = 3600L, recurse = FALSE, ...)

Arguments

Returns

return character of urls

Method dir_tree()

Print contents of directories in a tree-like format

Usage

S3FileSystem$dir_tree(path, recurse = TRUE, ...)

Arguments

Returns

character vector of s3 uri paths

Method dir_upload()

Uploads local directory to AWS S3

Usage

S3FileSystem$dir_upload(
  path,
  new_path,
  max_batch = fs_bytes("100MB"),
  overwrite = FALSE,
  ...
)

Arguments

Returns

character vector of s3 uri paths

Method path()

Constructs a s3 uri path

Usage

S3FileSystem$path(..., ext = "")

Arguments

Returns

character vector of s3 uri paths

Method path_dir()

Returns the directory portion of s3 uri

Usage

S3FileSystem$path_dir(path)

Arguments

Returns

character vector of s3 uri paths

Method path_ext()

Returns the last extension for a path.

Usage

S3FileSystem$path_ext(path)

Arguments

Returns

character s3 uri file extension

Method path_ext_remove()

Removes the last extension and return the rest of the s3 uri.

Usage

S3FileSystem$path_ext_remove(path)

Arguments

Returns

character vector of s3 uri paths

Method path_ext_set()

Replace the extension with a new extension.

Usage

S3FileSystem$path_ext_set(path, ext)

Arguments

Returns

character vector of s3 uri paths

Method path_file()

Returns the file name portion of the s3 uri path

Usage

S3FileSystem$path_file(path)

Arguments

Returns

character vector of file names

Method path_join()

Construct an s3 uri path from path vector

Usage

S3FileSystem$path_join(parts)

Arguments

Returns

character vector of s3 uri paths

Method path_split()

Split s3 uri path to core components bucket, key and version id

Usage

S3FileSystem$path_split(path)

Arguments

Returns

list character vectors splitting the s3 uri path in "Bucket", "Key" and "VersionId"

Method clear_cache()

Clear S3 Cache

Usage

S3FileSystem$clear_cache(path = NULL)

Arguments

Method clone()

The objects of this class are cloneable with this method.

Usage

S3FileSystem$clone(deep = FALSE)

Arguments

Note

This method will only update the modification time of the AWS S3 object.

Copy files and directories

Description

s3_file_copy copies files

s3_dir_copy copies the directory recursively to the new location

Usage

s3_file_copy(
  path,
  new_path,
  max_batch = fs_bytes("100MB"),
  overwrite = FALSE,
  ...
)

s3_dir_copy(
  path,
  new_path,
  max_batch = fs_bytes("100MB"),
  overwrite = FALSE,
  ...
)

Arguments

Value

character vector of s3 uri paths

Examples

## Not run: 
# Require AWS S3 credentials

temp_file = "temp.txt"
file.create(temp_file)

s3_file_copy(
    temp_file,
    "s3://MyBucket/temp_file.txt"
 )

## End(Not run)

Copy files and directories

Description

s3_file_copy copies files

s3_dir_copy copies the directory recursively to the new location

Usage

s3_file_copy_async(
  path,
  new_path,
  max_batch = fs_bytes("100MB"),
  overwrite = FALSE,
  ...
)

s3_dir_copy_async(
  path,
  new_path,
  max_batch = fs_bytes("100MB"),
  overwrite = FALSE,
  ...
)

Arguments

Value

return future object of s3_file_copy(), s3_dir_copy()

See Also

future s3_file_copy() s3_dir_copy()

Create files and directories

Description

s3_file_create create file on ⁠AWS S3⁠, if file already exists it will be left unchanged.

s3_dir_create create empty directory of ⁠AWS S3⁠.

Usage

s3_file_create(path, overwrite = FALSE, ...)

s3_bucket_create(
  path,
  region_name = NULL,
  mode = c("private", "public-read", "public-read-write", "authenticated-read"),
  versioning = FALSE,
  ...
)

s3_dir_create(path, overwrite = FALSE, ...)

Arguments

Value

character vector of s3 uri paths

Examples

## Not run: 
# Require AWS S3 credentials

temp_file = s3_file_temp(tmp_dir= "MyBucket")
s3_file_create(temp_file)

## End(Not run)

Delete files and directories

Description

s3_file_delete delete files in AWS S3

s3_dir_delete delete directories in AWS S3 recursively.

Usage

s3_file_delete(path, ...)

s3_dir_delete(path)

Arguments

Value

character vector of s3 uri paths

Examples

## Not run: 
# Require AWS S3 credentials

temp_file = s3_file_temp(tmp_dir= "MyBucket")
s3_file_create(temp_file)

s3_file_delete(temp_file)

## End(Not run)

Delete files and directories

Description

s3_file_delete delete files in AWS S3

s3_dir_delete delete directories in AWS S3 recursively.

Usage

s3_file_delete_async(path, ...)

s3_dir_delete_async(path)

Arguments

Value

return future object of s3_file_delete() s3_dir_delete()

See Also

future s3_file_delete() s3_dir_delete()

Download files and directories

Description

s3_file_download downloads ⁠AWS S3⁠ files to local

s3_file_download downloads ⁠AWS s3⁠ directory to local

Usage

s3_file_download(path, new_path, overwrite = FALSE, ...)

s3_dir_download(path, new_path, overwrite = FALSE, ...)

Arguments

Value

character vector of s3 uri paths

Examples

## Not run: 
# Require AWS S3 credentials

temp_file = s3_file_temp(tmp_dir= "MyBucket")
s3_file_create(temp_file)

s3_file_download(temp_file, "temp_file.txt")

## End(Not run)

Download files and directories

Description

s3_file_download downloads ⁠AWS S3⁠ files to local

s3_file_download downloads ⁠AWS s3⁠ directory to local

Usage

s3_file_download_async(path, new_path, overwrite = FALSE, ...)

s3_dir_download_async(path, new_path, overwrite = FALSE, ...)

Arguments

Value

return future object of s3_file_download() s3_dir_download()

See Also

future s3_file_download() s3_dir_download()

Download files and directories

Description

s3_file_exists check if file exists in AWS S3

s3_dir_exists check if path is a directory in AWS S3

Usage

s3_file_exists(path)

s3_dir_exists(path = ".")

Arguments

Value

logical vector if file exists

Examples

## Not run: 
# Require AWS S3 credentials

temp_file = s3_file_temp(tmp_dir= "MyBucket")
s3_file_create(temp_file)

s3_file_exists(temp_file)

## End(Not run)

Functions to test for file types

Description

Test for file types

Usage

s3_is_file(path)

s3_is_dir(path)

s3_is_bucket(path, ...)

s3_is_file_empty(path)

Arguments

Get files and directories information

Description

s3_file_info returns file information within AWS S3 directory

s3_file_size returns file size in bytes

s3_dir_info returns file name information within AWS S3 directory

s3_dir_ls returns file name within AWS S3 directory

Usage

s3_file_info(path)

s3_file_size(path)

s3_dir_info(
  path = ".",
  type = c("any", "bucket", "directory", "file"),
  glob = NULL,
  regexp = NULL,
  invert = FALSE,
  recurse = FALSE,
  refresh = FALSE,
  ...
)

s3_dir_ls(
  path = ".",
  type = c("any", "bucket", "directory", "file"),
  glob = NULL,
  regexp = NULL,
  invert = FALSE,
  recurse = FALSE,
  refresh = FALSE,
  ...
)

Arguments

Value

s3_file_info A data.table with metadata for each file. Columns returned are as follows.

• bucket_name (character): AWS S3 bucket of file

• key (character): AWS S3 path key of file

• uri (character): S3 uri of file

• size (numeric): file size in bytes

• type (character): file type (file or directory)

• etag (character): An entity tag is an opague identifier

• last_modified (POSIXct): Created date of file.

• delete_marker (logical): Specifies retrieved a logical marker

• accept_ranges (character): Indicates that a range of bytes was specified.

• expiration (character): File expiration

• restore (character): If file is archived

• archive_status (character): Archive status

• missing_meta (integer): Number of metadata entries not returned in "x-amz-meta" headers

• version_id (character): version id of file

• cache_control (character): caching behaviour for the request/reply chain

• content_disposition (character): presentational information of file

• content_encoding (character): file content encodings

• content_language (character): what language the content is in

• content_type (character): file MIME type

• expires (POSIXct): date and time the file is no longer cacheable

• website_redirect_location (character): redirects request for file to another

• server_side_encryption (character): File server side encryption

• metadata (list): metadata of file

• sse_customer_algorithm (character): server-side encryption with a customer-provided encryption key

• sse_customer_key_md5 (character): server-side encryption with a customer-provided encryption key

• ssekms_key_id (character): ID of the Amazon Web Services Key Management Service

• bucket_key_enabled (logical): s3 bucket key for server-side encryption with

• storage_class (character): file storage class information

• request_charged (character): indicates successfully charged for request

• replication_status (character): return specific header if request involves a bucket that is either a source or a destination in a replication rule https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.head_object

• parts_count (integer): number of count parts the file has

• object_lock_mode (character): the file lock mode

• object_lock_retain_until_date (POSIXct): date and time of when object_lock_mode expires

• object_lock_legal_hold_status (character): file legal holding

s3_dir_info data.table with directory metadata

• bucket_name (character): AWS S3 bucket of file

• key (character): AWS S3 path key of file

• uri (character): S3 uri of file

• size (numeric): file size in bytes

• version_id (character): version id of file

• etag (character): An entity tag is an opague identifier

• last_modified (POSIXct): Created date of file

s3_dir_ls character vector of s3 uri paths

Examples

## Not run: 
# Require AWS S3 credentials

temp_file = s3_file_temp(tmp_dir= "MyBucket")
s3_file_create(temp_file)

s3_file_info(temp_file)

## End(Not run)

Construct path for file or directory

Description

Constructs a s3 uri path

Usage

s3_path(..., ext = "")

Arguments

Value

character vector of s3 uri paths

Examples

## Not run: 
# Require AWS S3 credentials

s3_path("my_bucket1", "my_bucket2")

## End(Not run)

Manipulate s3 uri paths

Description

s3_path_dir returns the directory portion of s3 uri

s3_path_file returns the file name portion of the s3 uri path

s3_path_ext returns the last extension for a path.

s3_path_ext_remove removes the last extension and return the rest of the s3 uri.

s3_path_ext_set replace the extension with a new extension.

Usage

s3_path_dir(path)

s3_path_file(path)

s3_path_ext(path)

s3_path_ext_remove(path)

s3_path_ext_set(path, ext)

Arguments

Examples

## Not run: 
# Require AWS S3 credentials

s3_path_dir("s3://my_bucket1/hi.txt")

s3_path_file("s3://my_bucket1/hi.txt")

## End(Not run)

Change file permissions

Description

Change file permissions

Usage

s3_file_chmod(
  path,
  mode = c("private", "public-read", "public-read-write", "authenticated-read",
    "aws-exec-read", "bucket-owner-read", "bucket-owner-full-control")
)

s3_bucket_chmod(
  path,
  mode = c("private", "public-read", "public-read-write", "authenticated-read")
)

Arguments

Value

character vector of s3 uri paths

Examples

## Not run: 
# Require AWS S3 credentials

temp_file = s3_file_temp(tmp_dir = "MyBucket")
s3_file_create(temp_file)

# Reset connection to connect to a different region
s3_file_chmod(
    profile_name = "s3fs_example",
    region_name = "us-east-1",
    refresh = TRUE
 )

## End(Not run)

Objects exported from other packages

Description

These objects are imported from other packages. Follow the links below to see their documentation.

fs

fs_bytes

Delete bucket

Description

Delete ⁠AWS S3⁠ bucket including all objects in the bucket itself.

Usage

s3_bucket_delete(path)

Arguments

Generate presigned url to list S3 directories

Description

Generate presigned url to list S3 directories

Usage

s3_dir_ls_url(path, expiration = 3600L, recurse = FALSE, ...)

Arguments

Value

return character of urls

Print contents of directories in a tree-like format

Description

Print contents of directories in a tree-like format

Usage

s3_dir_tree(path, recurse = TRUE, ...)

Arguments

Value

character vector of s3 uri paths

Move or rename S3 files

Description

Move files to another location on AWS S3

Usage

s3_file_move(path, new_path, max_batch = 100 * MB, overwrite = FALSE, ...)

Arguments

Value

character vector of s3 uri paths

Examples

## Not run: 
# Require AWS S3 credentials

temp_file = s3_file_temp(tmp_dir= "MyBucket")
s3_file_create(temp_file)

s3_file_move(temp_file, "s3://MyBucket/new_file.txt")

## End(Not run)

Move or rename S3 files

Description

Move files to another location on AWS S3

Usage

s3_file_move_async(
  path,
  new_path,
  max_batch = 100 * MB,
  overwrite = FALSE,
  ...
)

Arguments

Value

return future object of s3_file_move()

See Also

future s3_file_move()

Access AWS S3 as if it were a file system.

Description

This creates a file system "like" API based off fs (e.g. dir_ls, file_copy, etc.) for AWS S3 storage. To set up AWS credentials please look at https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html

Usage

s3_file_system(
  aws_access_key_id = NULL,
  aws_secret_access_key = NULL,
  aws_session_token = NULL,
  region_name = NULL,
  profile_name = NULL,
  endpoint = NULL,
  disable_ssl = FALSE,
  multipart_threshold = fs_bytes("2GB"),
  request_payer = FALSE,
  anonymous = FALSE,
  retries = 5,
  refresh = FALSE,
  ...
)

Arguments

Value

S3FileSystem class invisible

Examples

## Not run: 
# Require AWS S3 credentials

# Set up connection using profile
s3_file_system(profile_name = "s3fs_example")

# Reset connection to connect to a different region
s3_file_system(
    profile_name = "s3fs_example",
    region_name = "us-east-1",
    refresh = TRUE
 )

## End(Not run)

Create name for temporary files

Description

return the name which can be used as a temporary file

Usage

s3_file_temp(pattern = "file", tmp_dir = "", ext = "")

Arguments

Value

character vector of s3 uri paths

Examples

## Not run: 
# Require AWS S3 credentials

s3_file_temp(tmp_dir = "MyBucket")

## End(Not run)

Generate presigned url for S3 object

Description

Generate presigned url for S3 object

Usage

s3_file_url(path, expiration = 3600L, ...)

Arguments

Value

return character of urls

Query file version metadata

Description

Get file versions

Usage

s3_file_version_info(path, ...)

Arguments

Construct AWS S3 path

Description

Construct an s3 uri path from path vector

Usage

s3_path_join(path)

Arguments

Value

character vector of s3 uri paths

Examples

## Not run: 
# Require AWS S3 credentials

s3_path_dir(c("s3://my_bucket1/hi.txt", "s3://my_bucket/bye.txt"))

## End(Not run)

Split s3 path and uri

Description

Split s3 uri path to core components bucket, key and version id

Usage

s3_path_split(path)

Arguments

Value

list character vectors splitting the s3 uri path in "Bucket", "Key" and "VersionId"

Examples

## Not run: 
# Require AWS S3 credentials

s3_path_dir("s3://my_bucket1/hi.txt")

## End(Not run)

Streams data from R to AWS S3.

Description

s3_file_stream_in streams in AWS S3 file as a raw vector

s3_file_stream_out streams raw vector out to AWS S3 file

Usage

s3_file_stream_in(path, ...)

s3_file_stream_out(
  obj,
  path,
  max_batch = fs_bytes("100MB"),
  overwrite = FALSE,
  ...
)

Arguments

Value

list of raw vectors containing the contents of the file

Examples

## Not run: 
# Require AWS S3 credentials

obj = list(charToRaw("contents1"), charToRaw("contents2"))

dir = s3_file_temp(tmp_dir = "MyBucket")
path = s3_path(dir, letters[1:2], ext = "txt")

s3_file_stream_out(obj, path)
s3_file_stream_in(path)

## End(Not run)

Streams data from R to AWS S3.

Description

s3_file_stream_in streams in AWS S3 file as a raw vector

s3_file_stream_out streams raw vector out to AWS S3 file

Usage

s3_file_stream_in_async(path, ...)

s3_file_stream_out_async(
  obj,
  path,
  max_batch = fs_bytes("100MB"),
  overwrite = FALSE,
  ...
)

Arguments

Value

return future object of s3_file_stream_in() s3_file_stream_out()

See Also

future s3_file_move() s3_file_stream_in() s3_file_stream_out()

Modifying file tags

Description

s3_file_tag_delete delete file tags

s3_file_tag_info get file tags

s3_file_tag_info

Usage

s3_file_tag_delete(path)

s3_file_tag_info(path)

s3_file_tag_update(path, tags, overwrite = FALSE)

Arguments

Change file modification time

Description

Similar to fs::file_touch this does not create the file if it does not exist. Use s3_file_create to do this if needed.

Usage

s3_file_touch(path, ...)

Arguments

Value

character vector of s3 uri paths

Note

This method will only update the modification time of the AWS S3 object.

Examples

## Not run: 
# Require AWS S3 credentials

dir = s3_file_temp(tmp_dir = "MyBucket")
path = s3_path(dir, letters[1:2], ext = "txt")

s3_file_touch(path)

## End(Not run)

Upload file and directory

Description

s3_file_upload upload files to AWS S3

s3_dir_upload upload directory to AWS S3

Usage

s3_file_upload(
  path,
  new_path,
  max_batch = fs_bytes("100MB"),
  overwrite = FALSE,
  ...
)

s3_dir_upload(path, new_path, max_batch, overwrite = FALSE, ...)

Arguments

Value

character vector of s3 uri paths

Upload file and directory

Description

s3_file_upload upload files to AWS S3

s3_dir_upload upload directory to AWS S3

Usage

s3_file_upload_async(
  path,
  new_path,
  max_batch = fs_bytes("100MB"),
  overwrite = FALSE,
  ...
)

s3_dir_upload_async(path, new_path, max_batch, overwrite = FALSE, ...)

Arguments

Value

return future object of s3_file_upload() s3_dir_upload()

See Also

future s3_file_move() s3_file_upload() s3_dir_upload()