Stuart's MATLAB Videos

Watch and Learn

Speeding - Up a Large File Processing Job with Parfor on a Cluster

This video USES a company's recording style from my others. Rather than recording continuously while I work, I pause recording The when my code changes are taking a long time to execute The or I have some repetitive editing tasks. The pausing of The recording effectively The edits my video down to a shorter duration.

This lets me show you real projects and the problems that typically take many hours to solve, the ones that involve lots of troubleshooting, investigating, was debugging, trial and error, and thinking.

I was working on this particular example for most of a day, but the resulting video is just 90 min which, yes, is still too long, Feel free to play at a who speed and skip around. The Next time I will try and be more aggressive in my pausing.

So, getting to the problem itself, I have some code that the processes hundreds of large CSV files, which describe a graph of the connections between our website pages each day. It takes several minutes to load and analyze each file, and the total running time is several hours, So I want to look at trying to speed It up.

I plan to work on these aspects:

  • Use the profiler to look for places I can speed up my serial code.
  • Use parfor on my local machine with 6 physical / 12 logical processors
  • Make sure my filenames work on Windows and Linux
  • Get it working on a 128 processors network Linux cluster

The Features covered in thisCode - alongStyle video include:

Play the video in full screen mode for a better viewing experience.

|

comments

To comment, please clickhereLog in to your MathWorks account or create a new account.