Bug in closed-source, proprietary software that I do not have access to

Bug in closed-source, proprietary software that I do not have access to

[Thomas Koenig]

Hi,

consider this: There is a bug, confirmed by several people. This occurs
in closed-source, proprietary software, and appears to be due to one
of my commits.

Despite considerable help from somebody who has access to the source,
and putting in quite a few (volunteer) hours myself, there is no
test case.

So, what to do? Close the PR as INVALID? This would be our standard
policy, correct?

FYI, the proprietary, closed-source software is SPEC, the corresponding
PR is 90539, and the friendly helper is Martin Liska.

Regards

	Thomas

Re: Bug in closed-source, proprietary software that I do not have access to

[Toon Moene]

On 5/25/19 2:52 PM, Thomas Koenig wrote:

> Hi,
> 
> consider this: There is a bug, confirmed by several people. This occurs
> in closed-source, proprietary software, and appears to be due to one
> of my commits.
> 
> Despite considerable help from somebody who has access to the source,
> and putting in quite a few (volunteer) hours myself, there is no
> test case.
> 
> So, what to do? Close the PR as INVALID? This would be our standard
> policy, correct?
> 
> FYI, the proprietary, closed-source software is SPEC, the corresponding
> PR is 90539, and the friendly helper is Martin Liska.

But the problem seems to be related to netcdf (your comment #22), which 
is freely available (don't know off-hand which license).

Does the problem trigger with netcdf's own test programs ?

That would open a way to investigate.

-- 
Toon Moene - e-mail: toon@moene.org - phone: +31 346 214290
Saturnushof 14, 3738 XG  Maartensdijk, The Netherlands
At home: http://moene.org/~toon/; weather: http://moene.org/~hirlam/
Progress of GNU Fortran: http://gcc.gnu.org/wiki/GFortran#news

Re: Bug in closed-source, proprietary software that I do not have access to

[Thomas Koenig]

Am 25.05.19 um 15:16 schrieb Toon Moene:
> On 5/25/19 2:52 PM, Thomas Koenig wrote:
> 
>> Hi,
>>
>> consider this: There is a bug, confirmed by several people. This occurs
>> in closed-source, proprietary software, and appears to be due to one
>> of my commits.
>>
>> Despite considerable help from somebody who has access to the source,
>> and putting in quite a few (volunteer) hours myself, there is no
>> test case.
>>
>> So, what to do? Close the PR as INVALID? This would be our standard
>> policy, correct?
>>
>> FYI, the proprietary, closed-source software is SPEC, the corresponding
>> PR is 90539, and the friendly helper is Martin Liska.
> 
> But the problem seems to be related to netcdf (your comment #22), which 
> is freely available (don't know off-hand which license).

I looked at the current netcdf code, and their C interface is different
from what SPEC seems to use. So, it is either a an old version,
or SPEC changed something for some reason, or both.

> Does the problem trigger with netcdf's own test programs ?

I have never built netcdf myself.  Would somebody who wants to
(possibly) generate a test case then have to build 30+ versions
on the off-chance of finding something?

Re: Bug in closed-source, proprietary software that I do not have access to

[Richard Biener]

On May 25, 2019 2:52:53 PM GMT+02:00, Thomas Koenig <tkoenig@netcologne.de> wrote:
>Hi,
>
>consider this: There is a bug, confirmed by several people. This occurs
>in closed-source, proprietary software, and appears to be due to one
>of my commits.
>
>Despite considerable help from somebody who has access to the source,
>and putting in quite a few (volunteer) hours myself, there is no
>test case.
>
>So, what to do? Close the PR as INVALID? This would be our standard
>policy, correct?

No, it would be WAITING since we're waiting for a testcase. 

>FYI, the proprietary, closed-source software is SPEC, the corresponding
>PR is 90539, and the friendly helper is Martin Liska.
>
>Regards
>
>	Thomas

Re: Bug in closed-source, proprietary software that I do not have access to

[gilles]

Thomas,

Several SPEC benchmarks are available at no cost for non commercial 
usage.

See https://www.spec.org/order.html, you may qualify

Cheers,

Gilles

----- Original Message -----
> Hi,
> 
> consider this: There is a bug, confirmed by several people. This 
occurs
> in closed-source, proprietary software, and appears to be due to one
> of my commits.
> 
> Despite considerable help from somebody who has access to the source,
> and putting in quite a few (volunteer) hours myself, there is no
> test case.
> 
> So, what to do? Close the PR as INVALID? This would be our standard
> policy, correct?
> 
> FYI, the proprietary, closed-source software is SPEC, the 
corresponding
> PR is 90539, and the friendly helper is Martin Liska.
> 
> Regards
> 
> 	Thomas
> 

Re: Bug in closed-source, proprietary software that I do not have access to

[Thomas Koenig]

Hi Gilles,

> Several SPEC benchmarks are available at no cost for non commercial
> usage.
> 
> See https://www.spec.org/order.html, you may qualify.

I checked, and I do not qualify - I am just a simple volunteer with
a day job which has nothing to do with maintaining gfortran, and
I do draw the line at paying for software in addition to the work
I donate to gcc.

It appears that wanting to supporting a closed-source,
commercial benchmark like SPEC and supporting a part of gcc
needed for that with mostly volunteers is not ideal.

Regards

	Thomas

Re: Bug in closed-source, proprietary software that I do not have access to

[Steve Kargl]

On Sat, May 25, 2019 at 02:52:53PM +0200, Thomas Koenig wrote:
> 
> consider this: There is a bug, confirmed by several people. This occurs
> in closed-source, proprietary software, and appears to be due to one
> of my commits.
> 
> Despite considerable help from somebody who has access to the source,
> and putting in quite a few (volunteer) hours myself, there is no
> test case.
> 
> So, what to do? Close the PR as INVALID? This would be our standard
> policy, correct?
> 
> FYI, the proprietary, closed-source software is SPEC, the corresponding
> PR is 90539, and the friendly helper is Martin Liska.
> 

Put the PR in WAITING as Richard suggested with a note that
SPEC is proprietary and without a testcase, it is not possible
in investigate further. 

For WRF, I suppose you or Martin could be a good citizen and
contact the project to report a bug.

https://www.mmm.ucar.edu/wrf-user-support-contributor-information

For netcdf, I suppose you of Martin could report a bug in
their interface at

https://www.unidata.ucar.edu/software/netcdf/#help

Martin's comment #23 shows the interface as

And Fortran interface is defined in netcdf/include/netcdf.inc:

      integer         nf_put_vara_double
!                         (integer             ncid,
!                          integer             varid,
!                          integer             start(1),
!                          integer             count(1),
!                          doubleprecision     dvals(1))
      external        nf_put_vara_double

start(1), count(1), and dvals(1) are each an element of an array.
i.e, scalar things.  If one looks at the NetCDF sources, the code
in question (netcdf-fortran-4.4.5/fortran/nf_varaio.F90) is 

 Function nf_put_vara_double(ncid, varid, start, counts, dvals) &
                                RESULT(status)

! Write out real(RK8) variable to dataset for given start and count vectors

 USE netcdf_nc_interfaces

 Implicit NONE

 Integer,   Intent(IN) :: ncid, varid
 Integer,   Intent(IN) :: start(*), counts(*)
 Real(RK8), Intent(IN) :: dvals(*)

So, the comment #23 and the implementation have a mismatch. I'll note
that NetCDF has a module fortran/module_netcdf_nf_interfaces.F90, but
munging a direct call from C into the Fortran code could mess up the
interface.  Now, returning to Martin's comment #20 where I wrap the code


  A0 = nc_put_vara_double (*fncid, *fvarid + -1,
       (const size_t *) f2c_coords (*fncid, *fvarid + -1, (const int *) A3,
       (size_t *) &B3),
       (const size_t *) f2c_counts (*fncid, *fvarid + -1, (const int *) A4,
       (size_t *) &B4),
        A5);

Depending on the declaration of A5, that naked A5 which is dvals(*),
likely  to work.



-- 
Steve

Re: Bug in closed-source, proprietary software that I do not have access to

[Toon Moene]

On 5/25/19 7:01 PM, Steve Kargl wrote:

> For WRF, I suppose you or Martin could be a good citizen and
> contact the project to report a bug.

I have thought about this. As a person with experience building and 
running weather forecasting codes, I would be first in line to try this. 
But the SPEC code could be modified and not resemble a fresh WRF setup ...

-- 
Toon Moene - e-mail: toon@moene.org - phone: +31 346 214290
Saturnushof 14, 3738 XG  Maartensdijk, The Netherlands
At home: http://moene.org/~toon/; weather: http://moene.org/~hirlam/
Progress of GNU Fortran: http://gcc.gnu.org/wiki/GFortran#news

Re: Bug in closed-source, proprietary software that I do not have access to

[Thomas Koenig]

Hi Toon,

> On 5/25/19 7:01 PM, Steve Kargl wrote:
> 
>> For WRF, I suppose you or Martin could be a good citizen and
>> contact the project to report a bug.
> 
> I have thought about this. As a person with experience building and 
> running weather forecasting codes, I would be first in line to try this. 
> But the SPEC code could be modified and not resemble a fresh WRF setup ...

The current version of netcdf has, in module_netcdf_nf_interfaces.F90,

!--------------------------------- nf_put_vara_double ----------------------
Interface
  Function nf_put_vara_double(ncid, varid, start, counts, dvals) &
                                 RESULT(status)

  USE netcdf_nf_data, ONLY: RK8

  Integer,   Intent(IN) :: ncid, varid
  Integer,   Intent(IN) :: start(*), counts(*)
  Real(RK8), Intent(IN) :: dvals(*)
  Integer               :: status

  End Function nf_put_vara_double
End Interface

which looks good (well, it's not BIND(C)). In another file,
it has it as a simple INTEGER declaration with EXTERNAL.
I don't know which of the two is actually used.

Re: Bug in closed-source, proprietary software that I do not have access to

[Toon Moene]

On 5/25/19 7:31 PM, Thomas Koenig wrote:
> Hi Toon,
> 
>> On 5/25/19 7:01 PM, Steve Kargl wrote:
>>
>>> For WRF, I suppose you or Martin could be a good citizen and
>>> contact the project to report a bug.
>>
>> I have thought about this. As a person with experience building and 
>> running weather forecasting codes, I would be first in line to try 
>> this. But the SPEC code could be modified and not resemble a fresh WRF 
>> setup ...
> 
> The current version of netcdf has, in module_netcdf_nf_interfaces.F90,
> 
> !--------------------------------- nf_put_vara_double 
> ----------------------
> Interface
>   Function nf_put_vara_double(ncid, varid, start, counts, dvals) &
>                                  RESULT(status)
> 
>   USE netcdf_nf_data, ONLY: RK8
> 
>   Integer,   Intent(IN) :: ncid, varid
>   Integer,   Intent(IN) :: start(*), counts(*)
>   Real(RK8), Intent(IN) :: dvals(*)
>   Integer               :: status
> 
>   End Function nf_put_vara_double
> End Interface
> 
> which looks good (well, it's not BIND(C)). In another file,
> it has it as a simple INTEGER declaration with EXTERNAL.
> I don't know which of the two is actually used.
> 
OK.

One thing I can do (tomorrow) is to download WRF, build it and see how 
it uses netcdf.

The trunk compiler I have at hand is revision 271618, so it includes 
your update that's the subject of PR90539.

-- 
Toon Moene - e-mail: toon@moene.org - phone: +31 346 214290
Saturnushof 14, 3738 XG  Maartensdijk, The Netherlands
At home: http://moene.org/~toon/; weather: http://moene.org/~hirlam/
Progress of GNU Fortran: http://gcc.gnu.org/wiki/GFortran#news

Re: Bug in closed-source, proprietary software that I do not have access to

[Toon Moene]

On 5/25/19 7:43 PM, Toon Moene wrote:

> On 5/25/19 7:31 PM, Thomas Koenig wrote:
>> Hi Toon,
>>
>>> On 5/25/19 7:01 PM, Steve Kargl wrote:
>>>
>>>> For WRF, I suppose you or Martin could be a good citizen and
>>>> contact the project to report a bug.
>>>
>>> I have thought about this. As a person with experience building and 
>>> running weather forecasting codes, I would be first in line to try 
>>> this. But the SPEC code could be modified and not resemble a fresh 
>>> WRF setup ...
>>
>> The current version of netcdf has, in module_netcdf_nf_interfaces.F90,
>>
>> !--------------------------------- nf_put_vara_double 
>> ----------------------
>> Interface
>>   Function nf_put_vara_double(ncid, varid, start, counts, dvals) &
>>                                  RESULT(status)
>>
>>   USE netcdf_nf_data, ONLY: RK8
>>
>>   Integer,   Intent(IN) :: ncid, varid
>>   Integer,   Intent(IN) :: start(*), counts(*)
>>   Real(RK8), Intent(IN) :: dvals(*)
>>   Integer               :: status
>>
>>   End Function nf_put_vara_double
>> End Interface
>>
>> which looks good (well, it's not BIND(C)). In another file,
>> it has it as a simple INTEGER declaration with EXTERNAL.
>> I don't know which of the two is actually used.
>>
> OK.
> 
> One thing I can do (tomorrow) is to download WRF, build it and see how 
> it uses netcdf.
> 
> The trunk compiler I have at hand is revision 271618, so it includes 
> your update that's the subject of PR90539.
> 

What I started with is some modern versions of the libraries needed: 
hdf5-1.10.5, netcdf-c-4.6.3, and netcdf-fortran-4.4.5, built with trunk 
revision 271618.

netcdf-c-4.6.3 (the C netcdf library) passed all its own tests.

netcdf-fortran-4.4.5 is the library of Fortran "glue" routines to the C 
library.

These are the relevant C interoperability comments during configure, 
indicating that it will use BIND(C) declarations for interfacing with 
the C library:

checking for Fortran flag to compile .f90 files... none
checking fortran 90 modules inclusion flag... -I
checking if Fortran compiler supports Fortran 2003 ISO_C_BINDING... yes
checking if Fortran compiler supports Fortran 2008 ISO_FORTRAN_ENV 
additions... yes
checking if Fortran compiler supports TS29113 standard extension... yes
checking whether F03 native code is desired... yes

However, there *is* a Segmentation Fault when running one of its tests:

cat nf_test/tst_f90.log

Program received signal SIGSEGV: Segmentation fault - invalid memory 
reference.

Backtrace for this error:
#0  0x7f915ef0d8b0 in ???
#1  0x7f915ef0cae3 in ???
#2  0x7f915eb6483f in ???
#3  0x7f915fa151cd in __netcdf_MOD_nf90_put_var_1d_fourbytereal
	at /home/toon/netcdf/netcdf-fortran-4.4.5/fortran/netcdf_expanded.f90:940
#4  0x402cfc in netcdftest
	at /home/toon/netcdf/netcdf-fortran-4.4.5/nf_test/tst_f90.f90:95
#5  0x40226c in main
	at /home/toon/netcdf/netcdf-fortran-4.4.5/nf_test/tst_f90.f90:7
FAIL tst_f90 (exit status: 139)

This is the error location in 
/home/toon/netcdf/netcdf-fortran-4.4.5/fortran/netcdf_expanded.f90:

     939        nf90_put_var_1D_FourByteReal = &
     940           nf_put_vara_real(ncid, varid, localStart, localCount, 
values)
     941      end if
     942    end function nf90_put_var_1D_FourByteReal

So at least *a* Segmentation Fault is reproducible with these freely 
available libraries ....

Kind regards,

-- 
Toon Moene - e-mail: toon@moene.org - phone: +31 346 214290
Saturnushof 14, 3738 XG  Maartensdijk, The Netherlands
At home: http://moene.org/~toon/; weather: http://moene.org/~hirlam/
Progress of GNU Fortran: http://gcc.gnu.org/wiki/GFortran#news

Re: Bug in closed-source, proprietary software that I do not have access to

[Thomas Koenig]

Hi Toon,

> However, there *is* a Segmentation Fault when running one of its tests:
> 
> cat nf_test/tst_f90.log
> 
> Program received signal SIGSEGV: Segmentation fault - invalid memory 
> reference.

Thanks!  Now I finally have (open) source to work with...

I think I have an idea what's wrong now.

Regards

	Thomas

Re: Bug in closed-source, proprietary software that I do not have access to

[Martin Liška]

On 5/26/19 8:11 PM, Thomas Koenig wrote:
> Hi Toon,
> 
>> However, there *is* a Segmentation Fault when running one of its tests:
>>
>> cat nf_test/tst_f90.log
>>
>> Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Thank you very much Toon, the segmentation fault looks very similar to what we have in cam_4 and wrf.

> 
> Thanks!  Now I finally have (open) source to work with...
> 
> I think I have an idea what's wrong now.

That' great!

Thanks,
Martin

> 
> Regards
> 
>     Thomas