Matlab and Hadoop integration
5 views (last 30 days)
Show older comments
we made a cluster of 3 systems, having 4 workers each, altogether 12 workers.
After scheduling a job in MATLAB on that cluster, it got stuck in 'Starting Parallel Pool' Phase. We are attaching some screenshots for your reference.
0 Comments
Accepted Answer
Kojiro Saito
on 14 Jan 2019
Without your whole code (apart1.m), it would be difficult to investigate why the error occurs, but as far as I guess, it might be missing to set mapreducer to use parallel.cluster.Hadoop.
% Specify HADOOP_HOME in MATLAB
setenv('HADOOP_HOME', '/path/to/hadoop/install')
% This will run mapreduce on Hadoop
cluster = parallel.cluster.Hadoop;
% If you want to change properties of parallel.cluster.Hadoop,
% please see https://www.mathworks.com/help/distcomp/parallel.cluster.hadoop_function.html#bujzb46-2
% for example, if the installation path of MATLAB Distributed Computing Server on Hadoop clusters
% is different from that of MATLAB Desktop on Hadoop node, you need to change ClusterMatlabRoot property.
% cluster.ClusterMatlabRoot = '/path/to/MDCS/install';
mr = mapreducer(cluster);
After this mapreducer setting, mapreduce will be run on Hadoop.
More Answers (0)
See Also
Categories
Find more on Cluster Configuration in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!