tall array and data store problem

I have problem with runnig an exmple from Matlab for tall arrays.

My question: I want to save a big cell array (12GB,20*35*5). I wanted to make a datastore (first tall array) to work efficiently in future but I am getting an error.

Do you have a suggestion to save this big array properly? I would appreciate if there is a script.

t = tall(rand(500,1))
location = 'C:\ExampleData';
write(location,t);

Error I get:>>>>

Writing tall data to folder C:\ExampleData

Evaluating tall expression using the Parallel Pool 'local':

Evaluation 8% complete

Error using parallel.internal.bigdata.MatArrayShuffleSortStrategy/createReduceInputSource (line 83)

Internal problem while evaluating tall expression. The problem was:

Error: File: C:\Program Files\MATLAB-R2020a\toolbox\matlab\datastoreio\+matlab\+io\+datastore\@TallDatastore\TallDatastore.m Line: 501 Column: 24

The import statement 'import matlab.io.internal.cellArrayDisp' cannot be found or cannot be imported. Imported names must end with '.*' or be fully qualified.

Error in parallel.internal.bigdata.ParallelPoolExecutor>@(partition)shuffleSortStrategy.createReduceInputSource(partition.PartitionIndex) (line 233)

readerFactory = @(partition) shuffleSortStrategy.createReduceInputSource(partition.PartitionIndex);

Error in matlab.bigdata.internal.executor.convertToIndependentTasks>@(partition)LocalReadProcessor(readerFactory(partition),numOutputs) (line 337)

readerExecution = CompositeDataProcessorBuilder([], @(partition) LocalReadProcessor(readerFactory(partition), numOutputs));

Error in ...

Error in tall/write (line 248)

iDoWrite(location, ta, writeFunction);

Caused by:

Error using matlab.io.datastore.TallDatastore

Error: File: C:\Program Files\MATLAB-R2020a\toolbox\matlab\datastoreio\+matlab\+io\+datastore\@TallDatastore\TallDatastore.m Line: 501 Column: 24

The import statement 'import matlab.io.internal.cellArrayDisp' cannot be found or cannot be imported. Imported names must end with '.*' or be fully

qualified.

Answer 1

[ag]

Hi Boby,

I understand that you are facing problem while using the "tall" function in MATLAB.

As I haven't been able to reproduce this issue, and the code works fine for me, I think this might be caused because of “shadowing”.

This problem arises if you have third-party toolboxes on your MATLAB path and they are earlier on the path than the built-in folders.

To resolve this, you can use the below code:

restoredefaultpath 
rehash toolboxcache 
savepath 

The "savepath" command will remove the the third-party tools from the path. If you still use them, do not use the “savepath” command, instead try moving the third-party tools to the end of the path (“pathtool” is very useful for this purpose).

For more details on “pathtool” please refer to the following documentation:

Open Set Path dialog box to view and change search path - MATLAB pathtool - MathWorks India

I hope this helps!

Best Regards,

Aryan Gupta.