Problems testing binutils with DejaGnu 1.6+

Problems testing binutils with DejaGnu 1.6+

[Andrey ``Bass'' Shcheglov]

[-- Attachment #1: Type: text/plain, Size: 2539 bytes --]

Hello,

I'm testing binutils (git master) with overridden RUNTESTFLAGS in order
to produce XML reports:

> $ make -k RUNTESTFLAGS='--all --xml' check

This ran fine with runtest 1.5.1, but produces the following error with
runtest 1.6:

> Making a new site.exp file...
> srcdir=`cd . && pwd`; export srcdir; \
> r=`pwd`; export r; \
> LC_ALL=C; export LC_ALL; \
> EXPECT=expect; export EXPECT; \
> runtest=runtest; \
> if /bin/bash -c "$runtest --version" > /dev/null 2>&1; then \
>   CC="gcc" CC_FOR_BUILD="gcc" \
>   CC_FOR_TARGET="gcc" CFLAGS_FOR_TARGET="-g -O2" \
>     $runtest --tool binutils --srcdir ${srcdir}/testsuite \
>         --all --xml; \
> else echo "WARNING: could not find \`runtest'" 1>&2; :;\
> fi
> WARNING: Couldn't find tool init file
> couldn't open "/home/nadezhda/TeamCity/buildAgent/work/412eaf977238ca9a/binutils/testsuite": illegal operation on a directory
> Makefile:1322: recipe for target 'check-DEJAGNU' failed
>     while executing
> "open [file join $outdir $xml_file_name] w"
>     (procedure "open_logs" line 19)
>     invoked from within
> "open_logs"
>     (file "/usr/share/dejagnu/runtest.exp" line 1223)
> make[5]: *** [check-DEJAGNU] Error 1
> make[4]: *** [check-am] Error 2
> make[3]: *** [check-recursive] Error 1
> make[2]: *** [check] Error 2
> make[1]: *** [check-binutils] Error 2
> make: *** [do-check] Error 2

The "check-DEJAGNU" target looks identical on systems with runtest 1.5
and runtest 1.6:

> 1321 check-DEJAGNU: site.exp
> 1322         srcdir=`cd $(srcdir) && pwd`; export srcdir; \
> 1323         r=`pwd`; export r; \
> 1324         LC_ALL=C; export LC_ALL; \
> 1325         EXPECT=$(EXPECT); export EXPECT; \
> 1326         runtest=$(RUNTEST); \
> 1327         if $(SHELL) -c "$$runtest --version" > /dev/null 2>&1; then \
> 1328           CC="$(CC)" CC_FOR_BUILD="$(CC_FOR_BUILD)" \
> 1329           CC_FOR_TARGET="$(CC_FOR_TARGET)" CFLAGS_FOR_TARGET="$(CFLAGS)" \
> 1330                 $$runtest --tool $(DEJATOOL) --srcdir $${srcdir}/testsuite \
> 1331                         $(RUNTESTFLAGS); \
> 1332         else echo "WARNING: could not find \`runtest'" 1>&2; :;\
> 1333         fi

In order to work around the issue, I have to add an explicit argument to
the --xml option of runtest 1.6 (ignored by runtest 1.5):

> $ make -k RUNTESTFLAGS='--all --xml=testsuite.xml' check

Is this a problem with binutils (Makefile.am/site.exp), or should I
report a bug to DejaGnu maintainers?

Regards,
Andrey.


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3996 bytes --]

Re: Problems testing binutils with DejaGnu 1.6+

[Nick Clifton]

Hi Andrey,

  Sorry for the delay in responding to this email.  I am not an expert
  on dejagnu and I was hoping that someone else would answer...

>> $ make -k RUNTESTFLAGS='--all --xml' check
> 
> This ran fine with runtest 1.5.1, but produces the following error with
> runtest 1.6:

Right - this happens for me too.

> Is this a problem with binutils (Makefile.am/site.exp), or should I
> report a bug to DejaGnu maintainers?

No, I think that it is a dejagnu problem.  I can reproduce the failure
without using the binutils make system at all:

  $ cd <bbinutils-build-dir>/binutils
  % runtest --tool binutils --srcdir <path-to-binutils-testsuite> --all --xml
  [...]
  couldn't open "<path-to-binutils-testsuite>": illegal operation on a directory
    while executing
  "open [file join $outdir $xml_file_name] w"

I think that the code to handle --xml without a filename argument is broken
and it is picking the first pathname like argument that it finds on the command
line.  (This is just a guess).

So - please do report the problem to DejaGnu.

Cheers
  Nick

Re: Problems testing binutils with DejaGnu 1.6+

[Andrey ``Bass'' Shcheglov]

Hi Nick,

Thank you for your response.

Indeed, further testing using the minimal testsuite revealed this was a
DejaGnu issue, and versions 1.5.3+ are affected.

I will report the issue to DejaGnu maintainers.

Regards,
Andrey.

On 10.08.2017 16:25, Nick Clifton wrote:
>   Sorry for the delay in responding to this email.  I am not an expert
>   on dejagnu and I was hoping that someone else would answer...
> 
>>> $ make -k RUNTESTFLAGS='--all --xml' check
>>
>> This ran fine with runtest 1.5.1, but produces the following error with
>> runtest 1.6:
> 
> Right - this happens for me too.
> 
>> Is this a problem with binutils (Makefile.am/site.exp), or should I
>> report a bug to DejaGnu maintainers?
> 
> No, I think that it is a dejagnu problem.  I can reproduce the failure
> without using the binutils make system at all:
> 
>   $ cd <bbinutils-build-dir>/binutils
>   % runtest --tool binutils --srcdir <path-to-binutils-testsuite> --all --xml
>   [...]
>   couldn't open "<path-to-binutils-testsuite>": illegal operation on a directory
>     while executing
>   "open [file join $outdir $xml_file_name] w"
> 
> I think that the code to handle --xml without a filename argument is broken
> and it is picking the first pathname like argument that it finds on the command
> line.  (This is just a guess).
> 
> So - please do report the problem to DejaGnu.