Is test case with 700k lines of code a valid test case?

Is test case with 700k lines of code a valid test case?

[Andrey Tarasevich]

Hi,

I have a source file with 700k lines of code 99% of which are printf() statements. Compiling this test case crashes GCC 5.3.0 with segmentation fault. 
Can such test case be considered valid or source files of size 35 MB are too much for a C compiler and it should crash? It crashes on Ubuntu 14.04 64bit with 16GB of RAM. 

Cheers,
Andrey

Re: Is test case with 700k lines of code a valid test case?

[C Bergström]

On Mon, Mar 14, 2016 at 11:31 PM, Andrey Tarasevich
<tarasevich@cs.uni-saarland.de> wrote:
> Hi,
>
> I have a source file with 700k lines of code 99% of which are printf() statements. Compiling this test case crashes GCC 5.3.0 with segmentation fault.
> Can such test case be considered valid or source files of size 35 MB are too much for a C compiler and it should crash? It crashes on Ubuntu 14.04 64bit with 16GB of RAM.

Sorry I can't help, but troll reply..
------------
If you're intentionally combining everything into a single source for
optimization reasons - you can expect it to really stress and hit many
problems in the compiler.

If your code is just extremely poorly written or not organized well -
I'd encourage you to both file bug reports (of a more reduced nature
than 700ksloc) and refactor things so that it's better designed.

Re: Is test case with 700k lines of code a valid test case?

[Paul_Koning]

> On Mar 14, 2016, at 11:31 AM, Andrey Tarasevich <tarasevich@cs.uni-saarland.de> wrote:
> 
> Hi,
> 
> I have a source file with 700k lines of code 99% of which are printf() statements. Compiling this test case crashes GCC 5.3.0 with segmentation fault. 
> Can such test case be considered valid or source files of size 35 MB are too much for a C compiler and it should crash? It crashes on Ubuntu 14.04 64bit with 16GB of RAM. 

I have a rather simple view of this sort of thing.  If I feed input to a program and the program crashes, that's always a bug.  This is true even if the input was "invalid" in some way.

That said, 700k lines in a single source file is a bit extravagant, but I see no way that such a thing could be legitimately called "invalid".  If it's all one function (or worse yet, one basic block), I would not be at all surprised if it exceeds a resource limit on how big a function can be, but if so, the expected output would be an error message, not a crash.

	paul

Re: Is test case with 700k lines of code a valid test case?

[Andrey Tarasevich]

I cross-checked with Clang 3.7.1. It eats up 4gb of RAM and compiles it under 40sec

> On 14 Mar 2016, at 16:44, Dennis Luehring <dl.soluz@gmx.net> wrote:
> 
> also cross-checked with clang 3.8?
> 
> Am 14.03.2016 um 16:31 schrieb Andrey Tarasevich:
>> Hi,
>> 
>> I have a source file with 700k lines of code 99% of which are printf() statements. Compiling this test case crashes GCC 5.3.0 with segmentation fault.
>> Can such test case be considered valid or source files of size 35 MB are too much for a C compiler and it should crash? It crashes on Ubuntu 14.04 64bit with 16GB of RAM.
>> 
>> Cheers,
>> Andrey
> 

Re: Is test case with 700k lines of code a valid test case?

[Andrey Tarasevich]

> On 14 Mar 2016, at 16:39, C Bergström <cbergstrom@pathscale.com> wrote:
> 
> On Mon, Mar 14, 2016 at 11:31 PM, Andrey Tarasevich
> <tarasevich@cs.uni-saarland.de> wrote:
>> Hi,
>> 
>> I have a source file with 700k lines of code 99% of which are printf() statements. Compiling this test case crashes GCC 5.3.0 with segmentation fault.
>> Can such test case be considered valid or source files of size 35 MB are too much for a C compiler and it should crash? It crashes on Ubuntu 14.04 64bit with 16GB of RAM.
> 
> Sorry I can't help, but troll reply..
> ------------
> If you're intentionally combining everything into a single source for
> optimization reasons - you can expect it to really stress and hit many
> problems in the compiler.
> 
> If your code is just extremely poorly written or not organized well -
> I'd encourage you to both file bug reports (of a more reduced nature
> than 700ksloc) and refactor things so that it's better designed.

Not that I intentionally written such code, but it was automatically generated by test tool.

Re: Is test case with 700k lines of code a valid test case?

[Ewart Timothée]

99% of 700k, 693k of printf(), is it ascii art ?
++t

> Le 14 Mar 2016 à 16:42, Paul_Koning@Dell.com a écrit :
> 
> 
>> On Mar 14, 2016, at 11:31 AM, Andrey Tarasevich <tarasevich@cs.uni-saarland.de> wrote:
>> 
>> Hi,
>> 
>> I have a source file with 700k lines of code 99% of which are printf() statements. Compiling this test case crashes GCC 5.3.0 with segmentation fault. 
>> Can such test case be considered valid or source files of size 35 MB are too much for a C compiler and it should crash? It crashes on Ubuntu 14.04 64bit with 16GB of RAM. 
> 
> I have a rather simple view of this sort of thing.  If I feed input to a program and the program crashes, that's always a bug.  This is true even if the input was "invalid" in some way.
> 
> That said, 700k lines in a single source file is a bit extravagant, but I see no way that such a thing could be legitimately called "invalid".  If it's all one function (or worse yet, one basic block), I would not be at all surprised if it exceeds a resource limit on how big a function can be, but if so, the expected output would be an error message, not a crash.
> 
> 	paul
> 

Re: Is test case with 700k lines of code a valid test case?

[C Bergström]

I don't speak with any community authority - I think your test tool is
misconfigured then. I don't see any pragmatic reason to generate such
a test. It's unlikely to mirror any real world code and artificial
test cases like this, at best only serve as some arbitrary data point.

On Mon, Mar 14, 2016 at 11:52 PM, Andrey Tarasevich
<tarasevich@cs.uni-saarland.de> wrote:
>
>> On 14 Mar 2016, at 16:39, C Bergström <cbergstrom@pathscale.com> wrote:
>>
>> On Mon, Mar 14, 2016 at 11:31 PM, Andrey Tarasevich
>> <tarasevich@cs.uni-saarland.de> wrote:
>>> Hi,
>>>
>>> I have a source file with 700k lines of code 99% of which are printf() statements. Compiling this test case crashes GCC 5.3.0 with segmentation fault.
>>> Can such test case be considered valid or source files of size 35 MB are too much for a C compiler and it should crash? It crashes on Ubuntu 14.04 64bit with 16GB of RAM.
>>
>> Sorry I can't help, but troll reply..
>> ------------
>> If you're intentionally combining everything into a single source for
>> optimization reasons - you can expect it to really stress and hit many
>> problems in the compiler.
>>
>> If your code is just extremely poorly written or not organized well -
>> I'd encourage you to both file bug reports (of a more reduced nature
>> than 700ksloc) and refactor things so that it's better designed.
>
> Not that I intentionally written such code, but it was automatically generated by test tool.

Re: Is test case with 700k lines of code a valid test case?

[Andrey Tarasevich]

Yeah, I guess attempt to commit a C file with 700K lines of code won’t end well for a developer.

Thanks for clarifying things!

> On 14 Mar 2016, at 17:05, C Bergström <cbergstrom@pathscale.com> wrote:
> 
> I don't speak with any community authority - I think your test tool is
> misconfigured then. I don't see any pragmatic reason to generate such
> a test. It's unlikely to mirror any real world code and artificial
> test cases like this, at best only serve as some arbitrary data point.
> 
> On Mon, Mar 14, 2016 at 11:52 PM, Andrey Tarasevich
> <tarasevich@cs.uni-saarland.de> wrote:
>> 
>>> On 14 Mar 2016, at 16:39, C Bergström <cbergstrom@pathscale.com> wrote:
>>> 
>>> On Mon, Mar 14, 2016 at 11:31 PM, Andrey Tarasevich
>>> <tarasevich@cs.uni-saarland.de> wrote:
>>>> Hi,
>>>> 
>>>> I have a source file with 700k lines of code 99% of which are printf() statements. Compiling this test case crashes GCC 5.3.0 with segmentation fault.
>>>> Can such test case be considered valid or source files of size 35 MB are too much for a C compiler and it should crash? It crashes on Ubuntu 14.04 64bit with 16GB of RAM.
>>> 
>>> Sorry I can't help, but troll reply..
>>> ------------
>>> If you're intentionally combining everything into a single source for
>>> optimization reasons - you can expect it to really stress and hit many
>>> problems in the compiler.
>>> 
>>> If your code is just extremely poorly written or not organized well -
>>> I'd encourage you to both file bug reports (of a more reduced nature
>>> than 700ksloc) and refactor things so that it's better designed.
>> 
>> Not that I intentionally written such code, but it was automatically generated by test tool.

Re: Is test case with 700k lines of code a valid test case?

[David Malcolm]

On Mon, 2016-03-14 at 16:31 +0100, Andrey Tarasevich wrote:
> Hi,
> 
> I have a source file with 700k lines of code 99% of which are
> printf() statements. Compiling this test case crashes GCC 5.3.0 with
> segmentation fault. 
> Can such test case be considered valid

Yes.

> or source files of size 35 MB are too much for a C compiler and it
> should crash? It crashes on Ubuntu 14.04 64bit with 16GB of RAM. 

Please file a bug about this.

Re: Is test case with 700k lines of code a valid test case?

[Paul_Koning]

> On Mar 14, 2016, at 12:05 PM, C Bergström <cbergstrom@pathscale.com> wrote:
> 
> I don't speak with any community authority - I think your test tool is
> misconfigured then. I don't see any pragmatic reason to generate such
> a test. It's unlikely to mirror any real world code and artificial
> test cases like this, at best only serve as some arbitrary data point.

I don't agree.  Generated code often produces very large files.  Also, proper handling of resource issues is important, because where those issues hit depends on the environment, and for some users might be in perfectly plausible code.  An error message that says "I can't handle this" is fine.  But a crash gives the user no information what is wrong, no suggestions on how to avoid the problem.

	paul

Re: Is test case with 700k lines of code a valid test case?

[Richard Biener]

On March 14, 2016 4:31:57 PM GMT+01:00, Andrey Tarasevich <tarasevich@cs.uni-saarland.de> wrote:
>Hi,
>
>I have a source file with 700k lines of code 99% of which are printf()
>statements. Compiling this test case crashes GCC 5.3.0 with
>segmentation fault. 
>Can such test case be considered valid or source files of size 35 MB
>are too much for a C compiler and it should crash? It crashes on Ubuntu
>14.04 64bit with 16GB of RAM. 

It is supposed to compile fine with -O1 at least.

Richard.

>Cheers,
>Andrey

Re: Is test case with 700k lines of code a valid test case?

[Paulo Matos]

On 14/03/16 16:31, Andrey Tarasevich wrote:
> Hi,
> 
> I have a source file with 700k lines of code 99% of which are printf() statements. Compiling this test case crashes GCC 5.3.0 with segmentation fault. 
> Can such test case be considered valid or source files of size 35 MB are too much for a C compiler and it should crash? It crashes on Ubuntu 14.04 64bit with 16GB of RAM. 
> 
> Cheers,
> Andrey
> 

I would think it's useful but a reduced version would be great.
Can you reduce the test? If you need a hand, I can help. Contact me
directly and I will give it a try.

Cheers,
-- 
Paulo Matos

Re: Is test case with 700k lines of code a valid test case?

[Jonathan Wakely]

On 18 March 2016 at 12:45, Paulo Matos wrote:
>
>
> On 14/03/16 16:31, Andrey Tarasevich wrote:
>> Hi,
>>
>> I have a source file with 700k lines of code 99% of which are printf() statements. Compiling this test case crashes GCC 5.3.0 with segmentation fault.
>> Can such test case be considered valid or source files of size 35 MB are too much for a C compiler and it should crash? It crashes on Ubuntu 14.04 64bit with 16GB of RAM.
>>
>> Cheers,
>> Andrey
>>
>
> I would think it's useful but a reduced version would be great.
> Can you reduce the test? If you need a hand, I can help. Contact me
> directly and I will give it a try.

It's probably crashing because it's too large, so if you reduce it
then it won't crash.

Re: Is test case with 700k lines of code a valid test case?

[Jakub Jelinek]

On Fri, Mar 18, 2016 at 02:02:48PM +0000, Jonathan Wakely wrote:
> On 18 March 2016 at 12:45, Paulo Matos wrote:
> >> I have a source file with 700k lines of code 99% of which are printf() statements. Compiling this test case crashes GCC 5.3.0 with segmentation fault.
> >> Can such test case be considered valid or source files of size 35 MB are too much for a C compiler and it should crash? It crashes on Ubuntu 14.04 64bit with 16GB of RAM.
> >>
> >> Cheers,
> >> Andrey
> >>
> >
> > I would think it's useful but a reduced version would be great.
> > Can you reduce the test? If you need a hand, I can help. Contact me
> > directly and I will give it a try.
> 
> It's probably crashing because it's too large, so if you reduce it
> then it won't crash.

But if most of the lines are pretty much the same or similar, it might be
worth trying to recreate it with preprocessor macros.  Just pick up a couple
of most common lines and duplicate them as many times as needed to try to
get the testcase into similar size.

	Jakub

Re: Is test case with 700k lines of code a valid test case?

[Paulo Matos]

[-- Attachment #1.1: Type: text/plain, Size: 277 bytes --]



On 18/03/16 15:02, Jonathan Wakely wrote:
> 
> It's probably crashing because it's too large, so if you reduce it
> then it won't crash.
> 

Would be curious to see what's the limit though, or if it depends on the
machine he's running GCC on.

-- 
Paulo Matos


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Is test case with 700k lines of code a valid test case?

[Paul_Koning]

> On Mar 18, 2016, at 12:53 PM, Paulo Matos <pmatos@linki.tools> wrote:
> 
> 
> 
> On 18/03/16 15:02, Jonathan Wakely wrote:
>> 
>> It's probably crashing because it's too large, so if you reduce it
>> then it won't crash.
>> 
> 
> Would be curious to see what's the limit though, or if it depends on the
> machine he's running GCC on.

It presumably depends on the machine, or rather the resource limits currently in effect (ulimit etc.)  But the expected outcome when a resource limit is exceeded is a clean error message saying so, not a crash.

	paul

Re: Is test case with 700k lines of code a valid test case?

[Jakub Jelinek]

On Fri, Mar 18, 2016 at 05:16:50PM +0000, Paul_Koning@Dell.com wrote:
> 
> > On Mar 18, 2016, at 12:53 PM, Paulo Matos <pmatos@linki.tools> wrote:
> > 
> > 
> > 
> > On 18/03/16 15:02, Jonathan Wakely wrote:
> >> 
> >> It's probably crashing because it's too large, so if you reduce it
> >> then it won't crash.
> >> 
> > 
> > Would be curious to see what's the limit though, or if it depends on the
> > machine he's running GCC on.
> 
> It presumably depends on the machine, or rather the resource limits currently in effect (ulimit etc.)  But the expected outcome when a resource limit is exceeded is a clean error message saying so, not a crash.

It depends.  If the problem is e.g. running into the ulimit -s limit, then
the compiler just crashes, all the driver can do is report that it crashed.
Slowing down the compiler by testing stack depth in every function just in
case over limit testcase is being compiled is undesirable.

	Jakub