> *From:* Paul Zimmermann > *Sent:* Friday, January 5, 2024 5:49 AM > *To:* newlib@sourceware.org > *Subject:* issue with tgamma > > Hi, > > I discovered the following issue with tgamma: for > x=-0x1.13a5ccd87c9bbp+1008 > it yields +Inf instead of NaN: > > Checking tgamma with newlib-4.4.0.20231231 > tgamma 0 -1 -0x1.13a5ccd87c9bbp+1008 [-nan] [inf] inf inf > libm gives inf > mpfr gives -nan > > This issue was present in earlier versions of Newlib, but we only found it > recently since we improved our testing framework. > > Paul > > This strictly is OK, as the standards allow either a range or domain error for negative integers, and domain errors may return an implementation-defined value. It most likely has been this way since the 1993 date in the source file, which includes * 5. Special Cases * lgamma(2+s) ~ s*(1-Euler) for tiny s * lgamma(1)=lgamma(2)=0 * lgamma(x) ~ -log(x) for tiny x * lgamma(0) = lgamma(inf) = inf * lgamma(-integer) = +-inf (The tgamma function ends up using the lgamma function.) POSIX does have an implied preference of sorts for the NaN return, so changing it could possibly be considered a small improvement even if not necessary--although it is strange that they define the result for argument 0 differently, calling for INF there, seems inconsistent. (It's somewhat interesting that the MPFR return value you show is -NaN. I suppose they chose that to help reflect it being due to a negative argument.) Craig