2

I wrote a little program that converts a bunch of files to pdf.

The program does the following:

  • Get an Array of FileInfo objects from a Folder (10'000 docs)
  • For each FileInfo
    • Create a backup copy with FileInfo.CopyTo(),
    • Convert the Document to PDF by using some Aspose Libraries
    • After conversion, copy the PDF to a new destination
    • Inside the foreach an Event is raised and handled by a WinForm UI to show some progress

Depending on the size of the Document the conversion of a Document can take 0-3 seconds. I thought that would be a perfect candidate for Parallel.ForEach, so I modified the program.

However the conversion took instead of 1 hour with conventional foreach 1.5 hours with Parallel.Foreach (The Server I've tried it has 2 x Intel Xeon Procs).

What did I do wrong or what do I need to consider to get better performance?

Improve this question
2
  • 2
    If you posted the code it might help us work out what is going wrong... Commented Sep 3, 2012 at 14:06
  • 1
    Why does the original code take 1 hour? Is it using 100 % of a single core all the time? Or is it limited by the speed of your harddisk? Commented Sep 3, 2012 at 14:09

3 Answers 3

Reset to default
1

I recommend checking if your operation is CPU bound or I/O bound by looking at the CPU in taskmanager and Disk I/O response time/queue length in Resource Monitor and/or looking at the various available performance counters.

I suspect your problem is most likely that you are now doing multiple file copies (both for creating the backup and writing the converted file) at the same time. Hard disks are much faster for sequential access (if you only write/read one file at a time) compared to random access.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

1

I can think on several problems that can cause the Parallel.Foreach to be slower:

  1. Running more threads than processors.
  2. Aspose Libraries isn't support multithread.
  3. Multiple approaches to GUI thread that is threadsafe and cannot be reached from different threads at the same times.

also I recommend you to read my previous answer about Task parallel library - Parallelism on single core

It talks about single core but it can reflect on your problem.

Improve this answer

2 Comments

@Jon: Unless he's using the library from multiple threads in the same process, and the library takes a lock (forcing all work to be done sequentially)
@BenVoigt Yep. I was thinking as in "nicely using a single thread in a way that stays out of other theads ways" as "single", which is the best way to support this sort of parallel use as opposed to "multi-threaded" in that it, itself uses multiple threads.
0

It would depend on quite a few things. I would certainly try setting MaxDegreeOfParallelism to 2, in the hope that if the conversion is is CPU-bound and single-threaded, then having one per core should be close to ideal, though certainly experiment further.

But your very approach assumes that the conversion doesn't itself make good use of multiple cores. If it does, and it's CPU-bound, then it's already doing to sort of parallel use of cores that you are trying to introduce, and you're likely just going to make the whole thing less efficient for that reason.

Edit: Thought made clearer in light of svick's comment. If the library doesn't support multi-threaded use then it's unlikely to have gotten this far without erroring, but its support for multi-threading may involve a lot of internal locking that could be fine when there are occasional concurrent calls, but very expensive if you've got long-term heavy concurrency.

Improve this answer

5 Comments

You think more context switching caused by parallelizing already parallel process could cause a 50 % slowdown?
@svick I'd say it's unlikely, but the second paragraph could apply to it making nicely optimal use of caches which would magnify that affect and internal structures that require locking, which could be much more significant.
Heavily IO bound threads (if your limiting factor is disk IO) would mean that parallel threads would pretty much only be overhead, 50% seems like a lot of overhead, try it with more and less threads to see how that affects performance.
Setting MaxDegreeofParallelism is a good idea, but I highly doubt that a dual socket Xeon (the description in the question isn't completely clear, but that's what it sounds like) has only 2 cores.
@BenVoigt right you are, though it might be good to experiment way lower than expected too, for that matter. I think the second case is the more likely though; it just doesn't play well parallelised.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.