Script to access Amazon S3 files

35 views (last 30 days)
André Bibaud
André Bibaud on 25 Sep 2017
Commented: Stephane on 23 Dec 2022
I setup a cluster on AWS EC2 to run my scripts faster. Right now my script load input variables locally (on my C: drive) and write simulation results also locally. I setup an S3 bucket with a sub-directory for my input variables and a sub-directory for the results. How do I change the load and write function to point to the S3 bucket instead of my c: drive?

Answers (3)

Jhon Wine
Jhon Wine on 24 Jan 2018
I found datastore to be very easy to use. Run the code below in each worker.
%Set S3 Credentials
setenv('AWS_ACCESS_KEY_ID', 'ID');
setenv('AWS_REGION', 'us-west-2');
%Load Data
fp= 's3://mybucket/data/';
Also define this function
function data= AWSRead(fileName)
fid = fopen(fileName);
data= fread(fid,inf,'short');
Make sure the EC2 cloud is on the same site as S3 to avoid data transfer bottlenecks

David Fink
David Fink on 28 Sep 2017
Since S3 uses a RESTful architecture, use 'webread' and 'webwrite' to access the S3 bucket.
  1 Comment
Jhon Wine
Jhon Wine on 17 Nov 2018
Is there an example, how to use webread and webwrite with S3? There is no good tutorial..

Sign in to comment.

Harish Sundaresh
Harish Sundaresh on 26 Jun 2021
can MATLAB read .json files from aws s3? I get these errors
%% webread
Error using webread (line 119)
The 's3' protocol specified in URL, 'url', is not supported. Specify the URL with the protocol 'http://' or 'https://'.
%% jsondecode
Error using jsondecode
JSON syntax error: expected value but found end of text.
Stephane on 23 Dec 2022
Thanks for sharing this Dmitry

Sign in to comment.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!