Re: Splitting up 27_io/basic_istream/ignore/wchar_t/94749.cc (takes too long)

[Mike Stump <mikestump@comcast.net>]

On Jun 9, 2023, at 9:20 AM, Hans-Peter Nilsson via Gcc-patches <gcc-patches@gcc.gnu.org> wrote:
> 
> The test 27_io/basic_istream/ignore/wchar_t/94749.cc takes
> about 10 minutes to run for cris-elf in the "gdb simulator"

I'd let the libstdc++ people comment on specific things.  I'll comment on general things.  We could let line count (or word count or character count) scale the timeout in part, we could record times in a db and put an expected run time into test cases or in an along side db. We could have factors for slow systems, slow simulators. A 5 GHz x86_64 will likely be faster that a 40 year old pdp11. We can have these scale factors trigger off OS, cpu statically, and/or we can do a quick bogomips calculation and let that scale it and record that scaling factor in the build tree.

A wealth of possibilities. Solutions that require maintenance or test case modification are annoying. Solutions that need port work are annoying. I'd be tempted to say bogomips into the build (test) tree.  There are two parts, time to compile test cases and time to run them.  I'd be fine with a half solution that only does what you need.  The other part can be done by someone that has a need.

I'd invite comments by others on other solutions or commentary on downsides.  For example, having a 208 thread count machine that takes 2-3 minutes to run the full testsuite is nice. A problem arises when 4-10 test cases suddenly start timing out.  You then go to around 10-53 minutes to test, which  is annoying. Anything that boosts the timeouts can hinder early port bring up, which, we'd like to avoid. I mention it without much a solution other than a db approach in the test tree that records each test case and can identify test cases that timeout and trim the timeout for them to something nicer like base + 50% once they timeout with a larger allotment of time.

We could entertain wild thoughts. For example, make a run cache that caches run results given an object. See an object in the future, just look it up in a hash cache for the object and return those results instead of running it.  This can give you a large speedup in testing and would simultaneously advantage all slow simulation ports.  Maybe a 20-100x speedup? If you want to go this way I'd say do it in python at the bottom as it would be nice to switch over to python in the next 5-20 years and away from tcl.

A object cache in python, should be fairly small wether it is used for remembering run times from previous runs and setting a timeout based upon it, or as a does it run  and pass or run and fail cache.  The caches are likely only part of the problem, one still needs to have a timeout when no cache entry it present.  They can speed testing for the day to day grind of people that run 1-200 times a week.