GSL v2.0 discussion

GSL v2.0 discussion

[Patrick Alken]

Hello all,

   There have been a lot of new features added to GSL since the last 
release 1.16, some of which have modified underlying data structures 
(breaking binary compatibility) and even some API changes. For this 
reason I don't want to make another 1.x release, but I know at the same 
time many people were hoping that 2.0 would interface with advanced 
linear algebra libraries (like libflame or lapack).

My current todo list before the next release is:
1) import interp2d code (mostly done just needs some further testing)
2) Hermite interpolation (shouldn't take long once I find some time)

Adding libflame or lapack to the todo list would delay the next release 
significantly. But I want to open this discussion again (it has been 
discussed several times previously on this list).

One method of doing this is to make 3 new libraries:
1) libgsllinalg - contains all the current GSL implementations of the 
linear algebra routines
2) libgslflame - contains wrapper routines with the same interfaces that 
gsl_linalg has now, but calls the relevant libflame routine
3) libgsllapack - contains wrapper routines with the same interfaces of 
gsl_linalg, but calls the relevant lapack routine

This way the user can decide which linear algebra library they want at 
link time, while maintaining a common API for everything. If the user 
wants to use the current GSL routines, they would link with -lgsl 
-lgslcblas -lgsllinalg

If they want libflame, they would link with -lgsl -lgslcblas -lgslflame 
-lflame

If they want lapack, they would link with -lgsl -lgslcblas -lgsllapack 
-llapack

A second method of doing this is to decide at compile time which linear 
algebra library to use (ie: autoconf could detect if lapack or flame is 
installed and then compile the library accordingly making the correct 
calls to the lapack/flame routines). This has the advantage of keeping 
the -lgsl -lgslcblas link libraries the same, the user would just need 
to add -llapack or -lflame.

*Potential issues*

1. Workspace sizes

Many of the flame/lapack routines require workspaces of different sizes 
than each other and the corresponding GSL routines. The easiest way to 
handle this would be to dynamically allocate workspace inside the 
wrapper functions of the gslflame and gsllapack libraries, which is not 
too appealing.

Another way to handle it is to have gsl_linalg_alloc() routines for all 
functions which would figure out the right workspace size for whatever 
library you are using. This would require a lot of code changes for 
current users, also not too appealing.

2. Errors

LAPACK tends to handle errors pretty well, and returns error codes from 
each function which could be interpreted by the GSL wrapper and passed 
on to the user.

libFLAME however seems to lack an error handling mechanism and calls 
abort() when it encounters an error (I have not looked deeply into the 
source but this is based on previous discussions on this list). Calling 
abort() is a very poor way to handle this and GSL users could be 
frustrated by this. I don't want to get into the business of 
forking/rewriting the flame error handling stuff so I don't see any easy 
solution to this.

3. Global state

This is another advantage of LAPACK over libflame - libflame requires a 
global initialization call to initialize a small number of global 
variables. This means that a) each wrapper function for libflame would 
need to check if that memory is initialized and if not call the function 
or b) GSL needs to provide a global init/free call. I don't like either 
of these options. (a) is bad because there is no simple way for the user 
to free that memory from their program. (b) is bad because global states 
are just ugly and bad design imo.

4. Fortran vs C

One main disadvantage of LAPACK is the ordering of memory in fortran, 
which would require transposing matrices prior to the lapack call, and 
then transposing back afterward. This isn't such a big problem in terms 
of efficiency but its still a headache.

*Summary*

I would like feedback from everyone on the following points:

1. Should we try to add lapack/flame interfaces for the 2.0 release or 
wait until 3.0? I personally probably won't have a lot of time to work 
on this for several months.

2. Is it better to select gsllinalg/lapack/flame at compile time or link 
time?

3. Whats the best way to handle the memory workspace requirements (add 
_alloc functions to all gsl_linalg routines or dynamic allocation)?

4. What should we do about error handling in libflame? Just accept the 
abort() behavior?

5. Is there a strong preference for doing wrappers for both lapack and 
flame? Should we only interface to lapack, due to the difficulties with 
flame (global state, abort() error handling)? Should we only interface 
to flame due to its more modern design?

Patrick

Re: GSL v2.0 discussion

[Jean-François Caron]

> 1. Should we try to add lapack/flame interfaces for the 2.0 release or wait until 3.0? 

Unless an individual specifically volunteers for this job, we should not delay.

> 2. Is it better to select gsllinalg/lapack/flame at compile time or link time?

Selecting at compile time sounds more reasonable.  If a user really needs to use both LA libs with GSL, they should be advanced enough to be able to install parallel versions of GSL.  

> 4. What should we do about error handling in libflame? Just accept the abort() behavior?

File a bug report with libflame?

> 5. Is there a strong preference for doing wrappers for both lapack and flame? Should we only interface to lapack, due to the difficulties with flame (global state, abort() error handling)? Should we only interface to flame due to its more modern design?

If only one of the libs was an option, I would vote for LAPACK, since it’s much more commonly used & well-known.  

Jean-François

Re: GSL v2.0 discussion

[Rhys Ulerich]

Hi Patrick,

My TODOs for 2.0 include...
  -- Merging some code for Konrad (the same Hermite stuff you mentioned)
  -- Merging the two B-spline workspaces into one

On the libflame/LAPACK question you raise, and out-of-order response
to 5 before 1 though 4....

> 5. Is there a strong preference for doing wrappers for both lapack and
> flame? Should we only interface to lapack, due to the difficulties with
> flame (global state, abort() error handling)? Should we only interface to
> flame due to its more modern design?

Maintaining two sets of linear algebra backends is error prone and
will stretch our already thin time to hack on GSL.  We might take an
intermediate approach and target
http://www.netlib.org/lapack/lapacke.html.  That'll solve C-to-Fortran
linkage hell for legacy LAPACK (assuming vendors support it) and
aiming for the standard-ish API follows the GSL precedent of writing
to the CBLAS API.  As libflame has a LAPACK-compatibility layer
("lapack2flame"), any LAPACKE "shim" permitting talking to a legacy
LAPACK would also permit talking to a fairly large subset of libflame.

Assuming success with that LAPACKE idea and a good driving use case,
we could marry GSL more closely with libflame down the road.

> 1. Should we try to add lapack/flame interfaces for the 2.0 release or wait
> until 3.0? I personally probably won't have a lot of time to work on this
> for several months.

Ditto here on the time crunch for a few months (stupid overdue
thesis).  I'd aim for LAPACKE in 2.0 and, if needed, tighter libflame
in 3.0.

> 2. Is it better to select gsllinalg/lapack/flame at compile time or link
> time?

Link time, and presumably an invisible thing for us with LAPACKE
provided that the Autoconf infrastructure can find something sensible
for 'make check'

> 3. Whats the best way to handle the memory workspace requirements (add
> _alloc functions to all gsl_linalg routines or dynamic allocation)?

Use of high-level LAPACKE claims to handle workspace allocation.  If
it becomes performance critical somewhere, we could manage it
internally and use the medium-level API along with explicit
management.

> 4. What should we do about error handling in libflame? Just accept the
> abort() behavior?

If libflame's LAPACK-compatibility does not include proper error
handling and just calls abort(), well, that would be their problem to
fix if they claim compatibility.

Once upon a time I got Brian's permission to yank the gsl_error
infrastructure for libflame and submitted a patch.  That can be dug up
if they want/need it.

- Rhys

Re: GSL v2.0 discussion

[Patrick Alken]

On 04/04/2014 02:30 PM, Rhys Ulerich wrote:
> Hi Patrick,
>
>    -- Merging the two B-spline workspaces into one
This is already done - bspline_deriv_workspace is gone and everything is 
now in bspline_workspace
>
> On the libflame/LAPACK question you raise, and out-of-order response
> to 5 before 1 though 4....
>
>> 5. Is there a strong preference for doing wrappers for both lapack and
>> flame? Should we only interface to lapack, due to the difficulties with
>> flame (global state, abort() error handling)? Should we only interface to
>> flame due to its more modern design?
> Maintaining two sets of linear algebra backends is error prone and
> will stretch our already thin time to hack on GSL.  We might take an
> intermediate approach and target
> http://www.netlib.org/lapack/lapacke.html.  That'll solve C-to-Fortran
> linkage hell for legacy LAPACK (assuming vendors support it) and
> aiming for the standard-ish API follows the GSL precedent of writing
> to the CBLAS API.  As libflame has a LAPACK-compatibility layer
> ("lapack2flame"), any LAPACKE "shim" permitting talking to a legacy
> LAPACK would also permit talking to a fairly large subset of libflame.
I must say I do like the idea of doing a lapack interface only, and then 
using the lapack2flame layer for libflame. Lapack is still the standard 
library, and many research groups continue to actively contribute code 
to lapack. Also last time I checked, libflame hasn't implemented the 
nonsymmetric eigenvalue solver. Lapack has a very sophisticated 
algorithm for this which would probably be a nightmare to port over to C 
(I actually looked into this once).

I need to look into whether lapack2flame accepts LAPACKE calls or just 
LAPACK calls (I don't see any reference to LAPACKE in the flame user 
manual). In any case, lapack is usually installed by default on many 
systems, whereas lapacke is probably not, so it may be worth supporting 
lapack instead.

 > Assuming success with that LAPACKE idea and a good driving use case, 
we could marry GSL more closely with libflame down the road.
>> 1. Should we try to add lapack/flame interfaces for the 2.0 release or wait
>> until 3.0? I personally probably won't have a lot of time to work on this
>> for several months.
> Ditto here on the time crunch for a few months (stupid overdue
> thesis).  I'd aim for LAPACKE in 2.0 and, if needed, tighter libflame
> in 3.0.
>
>> 2. Is it better to select gsllinalg/lapack/flame at compile time or link
>> time?
> Link time, and presumably an invisible thing for us with LAPACKE
> provided that the Autoconf infrastructure can find something sensible
> for 'make check'

One possible pain for link-time support is that it would break many 
existing makefiles, since users would need to add an additional 
-lgsllinalg or -lgsllapack to all GSL programs. This may not be such a 
problem but its worth thinking about.

>
>> 3. Whats the best way to handle the memory workspace requirements (add
>> _alloc functions to all gsl_linalg routines or dynamic allocation)?
> Use of high-level LAPACKE claims to handle workspace allocation.  If
> it becomes performance critical somewhere, we could manage it
> internally and use the medium-level API along with explicit
> management.
>
>> 4. What should we do about error handling in libflame? Just accept the
>> abort() behavior?
> If libflame's LAPACK-compatibility does not include proper error
> handling and just calls abort(), well, that would be their problem to
> fix if they claim compatibility.
>
> Once upon a time I got Brian's permission to yank the gsl_error
> infrastructure for libflame and submitted a patch.  That can be dug up
> if they want/need it.
>
> - Rhys

Re: GSL v2.0 discussion

[Patrick Alken]

On 04/04/2014 02:52 PM, Patrick Alken wrote:
> I need to look into whether lapack2flame accepts LAPACKE calls or just
> LAPACK calls (I don't see any reference to LAPACKE in the flame user
> manual). In any case, lapack is usually installed by default on many
> systems, whereas lapacke is probably not, so it may be worth supporting
> lapack instead.

Looking into this further it seems LAPACKE is included by default in 
recent lapack installations, which is very nice. I like how LAPACKE 
supports both row/column major matrix inputs - it seems this would make 
things much easier for us and we still might be able to use lapack2flame 
by linking with -llapacke but not -llapack?

Re: GSL v2.0 discussion

[Rhys Ulerich]

From today's NA Digest (http://www.netlib.org/na-digest-html), a
tidbit that's of interest for the current discussion...

- Rhys

---8<---
From: Field G. Van Zee field@cs.utexas.edu
Date: April 07, 2014
Subject: The union of libflame and LAPACK

Sponsored by an NSF Software Infrastructure for Sustained
Innovation grant, we have been developing a new, vertically
integrated dense linear algebra software stack. At the bottom of
this software stack is the BLAS-like Library Instantiation Software
(BLIS). Above this, targeting sequential and multithreaded
architectures is libflame. At the top of the stack is Elemental
for distributed memory architectures.

libflame targets roughly the same layer as does LAPACK, and now we
have incorporated the LAPACK code base into libflame. For those
operations where libflame has the native functionality, the LAPACK
code becomes an interface. For all other operations, the netlib
implementation provides that functionality. We affectionately call
this new union "flapack", which offers the following benefits:

1) The libflame implementation of LAPACK is entirely coded in C.
No Fortran libraries or compilers are required.
2) The libflame library builds upon the BLIS interface. This
interface, unlike the BLAS, allows for arbitrary row and column
stride. While some applications may benefit from this (e.g., those
that perform computation with slices of tensors), from a
development and maintainability point of view it allows more
functionality to be supported with less code.
3) The union of the two libraries allows users to benefit from both
the LAPACK and libflame code base, within one package.
4) "flapack" passes the LAPACK test suite on platforms where we
have tested this. (There is one exception of a test case that
involves packed matrices that we believe is not in general use.)

The library is available under a 3-clause BSD license at:
https://github.com/flame/libflame
---8<---