Profiling issues

Profiling issues

[MR ZenWiz]

I have been attempting to profile a large-scale application my company
builds in order to try and find out what some of the bottlenecks in
performance are.  To do this, I added -pg to all the compile and link
time options in the app.

This seems to work some of the time, in that when the app runs to
completion and terminates, the gmon.out files have been quite useful
(I have other issues with these I won't go into in this thread).

The app is generally built on SLES 11 SP1 to run on SLES and RHEL6 systems.

The first problem we saw was that on RHEL6, the app will finish its
run but never terminates.  We know this because the informational
messages it puts on the console indicate that it is done and wrapping
up execution, but then the app hangs for hours.  In one instance it
was still hung 24 hours after it announced it was done and we
eventually killed it.

We tried using a SLES system next, and for one set of test runs it worked fine.

Subsequent attempts to run the profiling version on SLES have failed
in the same manner as those on RHEL6 - long hang after execution is
done.  With a small dataset, one such run actually completed - one
hour after it said it was done.  (The delay between announced
completion and return to the command line is usually a few seconds.)

We do not have gdb on these systems at all, though I can press to have
it installed if that will help.

Is there any history of this sort of problem with which anyone here is
familiar?  I checked the archives and saw only one hit for profiling,
and that was not the same (though somewhat helpful).

The app is an ~800,000 line C++ program for backing up and restoring data.

Any assistance would be most appreciated.

Thanks.
MR