Decreasing Computational Time with Parfor and variable slicing
1 view (last 30 days)
I am fairly new to parallel computing and so far I feel like I have been successful. However I have written a code in parallel with parfor and I am using a HUGE data set on the magnitude of 34000 x n. I was wondering if there is a way to make my computations even more efficient. I also have a message saying that variable is indexed but not sliced in a parfor loop. This might result in unnecessary communication overhead. Here is a copy of my code
softTFIDFMat = zeros(n,n);
temp = zeros(1,n);
score = tfidfn(i,:)'*tfidfn(j,:).*jMat;
score = sum(score(:));
temp(j) = score;
softTFIDFMat(i,:) = temp;
tfidfn is a sparse matrix that is 34303 x n, where n is generally > 2000 and jMat is also a sparse double. Any help would be appreciated. Computational time is a little under 24 hours as of now.
More Answers (2)
Edric Ellis on 17 Apr 2013
It looks as though each iteration of your PARFOR loop accesses every element of "tfidfn", so you cannot slice it. Even if "tfidfn" were dense, it's still "only" about 0.5GB, and so the transfer time for that to each worker is very likely to be completely insignificant compared to 24 hours for the complete computation.