kmeans for dataset that's too big for memory

1 view (last 30 days)
Jeff
Jeff on 7 Oct 2013
Commented: Jeff on 8 Oct 2013
Hi,
I'd like to do kmeans for a dataset that is too big to be loaded in memory. What options do I have? I can load a part of the dataset to workspace by using MATFILE but general kmeans function requires loading entire dataset, which is not possible for my computer. Please help.

Answers (1)

Jan
Jan on 7 Oct 2013
Edited: Jan on 7 Oct 2013
The obvious strategy is to install more RAM.
If you start with the claim, that the function you want to use requires the complete data set to be loaded at once, there is no way to escape from the need to load the complete data set at once. If this is not possible for your computer, upgrade it or run the code on a more powerful computer.
Sorry, I know that this answer is trivial. But it is the only possible answer matching your question exactly. Of course you could apply a data compression in the RAM or condense the data at first. But this is not exactly what you are asking for, and in addition, it would be much more expensive most likely.
  3 Comments
Jan
Jan on 7 Oct 2013
And does "Big data" mean that it is more than matches into the RAM? How big are your data?
Jeff
Jeff on 8 Oct 2013
Yes, big data here refers to dataset that's too large for the memory. The dataset I'm using is about 90GB in size.

Sign in to comment.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!