About coordinated efforts on scientific software.

About coordinated efforts on scientific software.

[Alan Aspuru-Guzik]

This e-mail is a reply to Cristoph Siopi's email.

Some points:

There is a relevant coordinated effort by the Department of Energy for
creating a common base for scientific software. I actually went to a
workshop for the ACTS Toolkit:
http://acts.nersc.gov/

Which is a collection of software that is released using the GPL licencse
for most cases, and that is well suited for some of the tasks that people
need.  For example, if one needs parallel Matrix and Vector operations, I
recommend to take a look at PetSc. It looks really good.

I have been using GSL, and doing my vector MPI distribution by hand, just
because I did not know that such tools were in such advanced state, and
that they were actually written in C, etc.

Another very, VERY interesting effort, similar to what the GNOME people
wanted to do using CORBA as a glue for their component architecture,
Bonobo, is the Common Components Architecture:
http://www.cca-forum.org/

During the same workshop, I attended a one day tutorial on it, and it
seems very promising. When it becomes more mature, it would be very
interesting for one of us (maybe me) to CCA-ize the GSL, in such a way
that it can be used in conjuction with the tools from ACTS, that are
already starting to be CCAized (like TAO, the Toolkit for Advanced
Optimization).

So there are some people that are indeed looking at the big picture. Maybe
what we need in the volunteer arena is leadership and organization like
the one that started umbrella projects, such as GNOME or KDE, that brought
a lot of people together.

Greetings,
Alan

-- 

Alan Aspuru-Guzik                    Dios mueve al jugador, y éste, la pieza.
(510)642-5911 UC Berkeley           ¿Qué Dios detrás de Dios la trama empieza
(925)422-8739 LLNL                de polvo y tiempo y sueño y agonías? -Borges

Re: About coordinated efforts on scientific software.

[Manoj Warrier]

Hi.

I was trying not to mail (because if someone asks me ..., "OK wise guy
so enough of talk, do something about it", I would be at a loss of words).

However here goes ...
Dare I try compressing Cristoph Siopi's mail into 3 main points:
1) Need to identify Core projects (numerical library, plotting/graphics
   language, etc)
2) Coding Standards.
3) Open source developers making a concerted effort to tackle points (1)
   and (2) above rather than digging thier own burrows.

I guess (hope rather) that GSL will eventually cover the numerical library
part of point (1). For plotting and graphics we again have a similar
situation as in the "mathematic packages" ... Check out
(http://scilinux.sf.net/graphvis.html" for a list of free packages.

It reminds me of what I read about the formation of the MPI forum,
creation of standards, etc... However even that did not stop mpich /
LAM MPI / and other implementations of the Message Passing Interface.
That makes me wonder more about the "evolution" picture of software
development. However periodic stock taking (Conference ?? and creation
of standards or a umbrella project) like Alan suggests seems necessary.

And yes Alberto you are hearing what the public wants.... All the best.
I could create another mailing list for more detailed discussions
on what can be done (the most I can do at present :-( ).

Manoj

On Mon, 21 Oct 2002, Alan Aspuru-Guzik wrote:
> This e-mail is a reply to Cristoph Siopi's email.
snip
snip
>
> So there are some people that are indeed looking at the big picture. Maybe
> what we need in the volunteer arena is leadership and organization like
> the one that started umbrella projects, such as GNOME or KDE, that brought
> a lot of people together.
>
> Greetings,
> Alan

-- 
Manoj Warrier (manoj.warrier@ipp.mpg.de)

Stellaratortheorie, Max-Planck Institut Fur Plasmaphysik
TeilInstitut Greifswald Wendelsteinstrasse 1
D-17491 Greifswald Germany Tel: +49-3834-882434

--------- History of Computing 10-11-3003 ---------------
Then there used to be this great user friendly OS which
overwrote your MBR whenever you installed it.
---------------------------------------------------------

RE: About coordinated efforts on scientific software.

[JJ Gomez-Cadenas]

Hi,
The discussion seems particullarly relevant for some fields that can benefit
enormously from free software such as high energy physics.

Recently a new project has been launched in the context of the Large Hadron
Collider project. The LHC will collide protons a TeV energies to search new
particles and explore the fundamental laws of physics. Very large detectors
will record the result of collisions and analyze terabites of data. It has
been recognized that robust numerical libraries, graphics packages,
simulations programs, data bases etc. are needed for this effort.

The LHC Grid Project (LHCGP) is trying to understand the possibilities and
make recommandations. In the past years, there was the trend at CERN and
other HEP labs to resort to commercial products (the use of objectivity or
oracle as data bases, for example, NAG as numerical library). This trend
seems, fortunately to be reversing now. LHCGP has recommended recently to
study the possibility to use GSL rather than NAG as the foundation
scientific library for HEP. THis seems today very likely.

However, one problem that free software faces is the diversity and lack of
coordination of the different efforts, as amply discussed here. In order to
be attractive to external clients, such as LHCGP or others, it would be
good, indeed, to see some coordination emerging. To me, one way to achieve
so --the trick is used very often in HEP-- is via one or more workshops.
Since some of us work at universities it doesn't seem impossible to find
some modest amount of funding if the thing is correctly and well-ahead
planned.

Also. If Brian and the other guys of GSL do not complain I believe this list
is a good place to continue the discussion for the time being, since it has
a very wide audience.

Let me bring an important subject to the discussion, that of wrappers. If
GSL is going to be truly the core of free-software numerical calculations,
wrappers are needed. For HEP it seems one would like C++ and python
wrappers. I have played for more than a year now with a C++ wrapper for GSL
and a number of non-trivial issues emerge, such as the speed, correct use of
error handling, minimizing of copying of structures and the like, which
often end up by suggesting re-implementation rather than wrapping (at least
for some parts of the library). It would be nice to understand better and to
discuss all those issues.

Best,

JJ Gomez-Cadenas


-----Mensaje original-----
De: gsl-discuss-owner@sources.redhat.com
[mailto:gsl-discuss-owner@sources.redhat.com]En nombre de Manoj Warrier
Enviado el: lunes, 21 de octubre de 2002 21:50
Para: Alan Aspuru-Guzik
CC: gsl-discuss@sources.redhat.com
Asunto: Re: About coordinated efforts on scientific software.


Hi.

I was trying not to mail (because if someone asks me ..., "OK wise guy
so enough of talk, do something about it", I would be at a loss of words).

However here goes ...
Dare I try compressing Cristoph Siopi's mail into 3 main points:
1) Need to identify Core projects (numerical library, plotting/graphics
   language, etc)
2) Coding Standards.
3) Open source developers making a concerted effort to tackle points (1)
   and (2) above rather than digging thier own burrows.

I guess (hope rather) that GSL will eventually cover the numerical library
part of point (1). For plotting and graphics we again have a similar
situation as in the "mathematic packages" ... Check out
(http://scilinux.sf.net/graphvis.html" for a list of free packages.

It reminds me of what I read about the formation of the MPI forum,
creation of standards, etc... However even that did not stop mpich /
LAM MPI / and other implementations of the Message Passing Interface.
That makes me wonder more about the "evolution" picture of software
development. However periodic stock taking (Conference ?? and creation
of standards or a umbrella project) like Alan suggests seems necessary.

And yes Alberto you are hearing what the public wants.... All the best.
I could create another mailing list for more detailed discussions
on what can be done (the most I can do at present :-( ).

Manoj

On Mon, 21 Oct 2002, Alan Aspuru-Guzik wrote:
> This e-mail is a reply to Cristoph Siopi's email.
snip
snip
>
> So there are some people that are indeed looking at the big picture. Maybe
> what we need in the volunteer arena is leadership and organization like
> the one that started umbrella projects, such as GNOME or KDE, that brought
> a lot of people together.
>
> Greetings,
> Alan

--
Manoj Warrier (manoj.warrier@ipp.mpg.de)

Stellaratortheorie, Max-Planck Institut Fur Plasmaphysik
TeilInstitut Greifswald Wendelsteinstrasse 1
D-17491 Greifswald Germany Tel: +49-3834-882434

--------- History of Computing 10-11-3003 ---------------
Then there used to be this great user friendly OS which
overwrote your MBR whenever you installed it.
---------------------------------------------------------

Re: About coordinated efforts on scientific software.

[Wartan Hachaturow]

On Tue, Oct 22, 2002 at 10:32:00AM +0200, JJ Gomez-Cadenas wrote:

> seems, fortunately to be reversing now. LHCGP has recommended recently to
> study the possibility to use GSL rather than NAG as the foundation
> scientific library for HEP. THis seems today very likely.

I must say I'd be _happy_ if it'd happen. I'm working on the part of the ALICE
project in Russian group, and I do use GSL to perform needed
computations ;) 
I can only dream on ALIROOT being rewritten using python and C/GSL..

-- 
Regards, Wartan.
"Computers are not intelligent. They only think they are."

RE: About coordinated efforts on scientific software.

[Brian Gough]

JJ Gomez-Cadenas writes:
 > If Brian and the other guys of GSL do not complain I believe this list
 > is a good place to continue the discussion for the time being, since it has
 > a very wide audience.

That's fine by me.  

If anyone wants to reduce the number of messages there is a digest
version of this list available (1 message per day).

To subscribe to it send an empty  message to 

  gsl-discuss-digest-subscribe@sources.redhat.com

and  to 

  gsl-discuss-unsubscribe@sources.redhat.com

to unsubscribe from the non-digest version.

Re: About coordinated efforts on scientific software.

[Christos Siopis]

On Mon, 21 Oct 2002, Manoj Warrier wrote:

> I guess (hope rather) that GSL will eventually cover the numerical library
> part of point (1). For plotting and graphics we again have a similar
> situation as in the "mathematic packages" ... Check out
> (http://scilinux.sf.net/graphvis.html" for a list of free packages.

I think the problem is not so much with lack of libraries as it is with
lack of an "integrated" environment where one can start with raw data,
pass them through various mathematical transformations, and finally plot
some result, all from inside the same "package"/environment that 
encourages trial-and-error, what-if experiments, and rapid prototyping.

The first thought that one might have for achieving this would be to
somehow wrap a number of relevant libraries and use them from inside a
scripting language like python. I can see at least three kinds of problems
with this:

- First, most of the existing libraries are too low-level for direct use
from an interactive scripting environment. Things like memory allocation
(needed e.g. by GSL) or opening a window for plotting (needed by e.g. by
PGPLOT)  are *show-stoppers* in an interactive environment. Some heroic
people are going through the pain of actually creating usable interfaces,
such as the PyGSL folks. This is fine, except two things:  how do the
wrapper functions interoperate with functions from other wrapped libraries
(see next item below), and how do we ensure we do not enter into a
versioning hell, where the wrapper uses some version A of the library, but
the library has now moved on to version B; add some RPM versioning issues
if you use RedHat's stuff and multiply all this by the number of libraries
which you want to wrap and enjoy the mess...

- Second, there is the issue of consistency of the user interface. For
instance, a NumPy (numeric python) user is used to the ufuncs, "universal"  
functions with return type that depends on the input type. So if a NumPy
user wanted to compute the mean of an array, he or she would expect that a
function call like mean(arrayx) would return an long int or a float,
depending on whether arrayx is an array of longs or of floats. But doing
this through GSL/PyGSL, the user would have to use pygsl.statistics.mean
for a float or pygsl.statistics.long.mean for a long int, i.e. the user is
asked to think in terms of C, a strongly typed language. This is both
annoying and prone to hard-to-find errors. A related issue is the overlap
between wrappers of different libraries (e.g., NumPy already has a couple
of mean/average functions from other libraries!). And there is also the
issue of performance, as NumPy objects are converted back and forth to
different formats (some wrappers do a better job at this than others).

- Third, it's the question of "putting this all together". Wrappers are
good for wrapping a small number of small libraries. As you add more and
more, there's all sorts of issues related to the distribution of the
"final" package, the quality and homogeneity of the documentation, and so
on. If there was no other solution at hand, maybe this would all be
acceptable. But with commercial packages offering a "one-stop" solution
(despite a number of other disadvantages) i think the open-source science
community has to do better than that.

SciPy ( www.scipy.org ) is a package that tries to solve some of these
problems but i this it is a little too early to tell how good the outcome
will be, and i cannot help wondering how many more times the open source
numerical community will have to code and debug e.g. an FFT transform or a
statistical package and whether this is the best use of our resources...

Christos

Re: About coordinated efforts on scientific software.

[Alan Aspuru-Guzik]

Christos,

Greetings again.

In reply to your long message's concerns. I believe that all the community
should gather (and support) the Common Components Architecture and BABEL
(which interoperate together) for the issues that you talk about.

If the community agreed on a representation for
vectors/matrices/operators/statistics functions, etc. current libraries
can be wrapped with things like Babel.

Take a look at this nice tutorial:
http://www.llnl.gov/CASC/components/docs/2001-pict-intro.pdf

And the documentation here:
http://www.llnl.gov/CASC/components/overview.html

The languages they support are:

Fortran 77, Fortran 90, C, C++, Java, MATLAB, and Python. 

That means: if we had a group of people creating BABEL interfaces to GSL,
it could be automatically used from all these different languages at the
cost of a C++ call (at least that is what the Babel people say that their
wrappers cost in CPU time).

The common components architecture, which can use Babel components, has
even a GUI builder where you can assemble prebuilt components (like an
optimizer, a Monte Carlo integrator, etc.) for creating applications in
literally seconds.

Babel is LGPL, CCA and people are componentizing a lot of stuff. 

Jaideep Ray showed a very complex example of a solver for the diffusion
equation. For example, they componentized stuff like a mesh refiner
http://www.cca-forum.org/cca-sc01/sandbox/jaray/samr/GrACEComponent/docs/README.html
or a mesh statistics tool:
http://www.cca-forum.org/cca-sc01/sandbox/jaray/samr/StatsComponent/docs/README.html
an optimizer:
http://www.cca-forum.org/cca-sc01/sandbox/norris/nonlinear/taosolver/README.html

I think that these guys are very advanced in this project, and if
all the community joined in, this could become our own umbrella project.

I am a graduate student trying to finish my work, so I decided not to
componentize anything right now, but later I might work on it.

Anyway, a simple effort of componentizing what we already have, might get
us farther than we think.

Alan



On Tue, 22 Oct 2002, Christos Siopis wrote:

> On Mon, 21 Oct 2002, Manoj Warrier wrote:
> 
> > I guess (hope rather) that GSL will eventually cover the numerical library
> > part of point (1). For plotting and graphics we again have a similar
> > situation as in the "mathematic packages" ... Check out
> > (http://scilinux.sf.net/graphvis.html" for a list of free packages.
> 
> I think the problem is not so much with lack of libraries as it is with
> lack of an "integrated" environment where one can start with raw data,
> pass them through various mathematical transformations, and finally plot
> some result, all from inside the same "package"/environment that 
> encourages trial-and-error, what-if experiments, and rapid prototyping.
> 
> The first thought that one might have for achieving this would be to
> somehow wrap a number of relevant libraries and use them from inside a
> scripting language like python. I can see at least three kinds of problems
> with this:
> 
> - First, most of the existing libraries are too low-level for direct use
> from an interactive scripting environment. Things like memory allocation
> (needed e.g. by GSL) or opening a window for plotting (needed by e.g. by
> PGPLOT)  are *show-stoppers* in an interactive environment. Some heroic
> people are going through the pain of actually creating usable interfaces,
> such as the PyGSL folks. This is fine, except two things:  how do the
> wrapper functions interoperate with functions from other wrapped libraries
> (see next item below), and how do we ensure we do not enter into a
> versioning hell, where the wrapper uses some version A of the library, but
> the library has now moved on to version B; add some RPM versioning issues
> if you use RedHat's stuff and multiply all this by the number of libraries
> which you want to wrap and enjoy the mess...
> 
> - Second, there is the issue of consistency of the user interface. For
> instance, a NumPy (numeric python) user is used to the ufuncs, "universal"  
> functions with return type that depends on the input type. So if a NumPy
> user wanted to compute the mean of an array, he or she would expect that a
> function call like mean(arrayx) would return an long int or a float,
> depending on whether arrayx is an array of longs or of floats. But doing
> this through GSL/PyGSL, the user would have to use pygsl.statistics.mean
> for a float or pygsl.statistics.long.mean for a long int, i.e. the user is
> asked to think in terms of C, a strongly typed language. This is both
> annoying and prone to hard-to-find errors. A related issue is the overlap
> between wrappers of different libraries (e.g., NumPy already has a couple
> of mean/average functions from other libraries!). And there is also the
> issue of performance, as NumPy objects are converted back and forth to
> different formats (some wrappers do a better job at this than others).
> 
> - Third, it's the question of "putting this all together". Wrappers are
> good for wrapping a small number of small libraries. As you add more and
> more, there's all sorts of issues related to the distribution of the
> "final" package, the quality and homogeneity of the documentation, and so
> on. If there was no other solution at hand, maybe this would all be
> acceptable. But with commercial packages offering a "one-stop" solution
> (despite a number of other disadvantages) i think the open-source science
> community has to do better than that.
> 
> SciPy ( www.scipy.org ) is a package that tries to solve some of these
> problems but i this it is a little too early to tell how good the outcome
> will be, and i cannot help wondering how many more times the open source
> numerical community will have to code and debug e.g. an FFT transform or a
> statistical package and whether this is the best use of our resources...
> 
> Christos
> 

-- 

Alan Aspuru-Guzik                    Dios mueve al jugador, y éste, la pieza.
(510)642-5911 UC Berkeley           ¿Qué Dios detrás de Dios la trama empieza
(925)422-8739 LLNL                de polvo y tiempo y sueño y agonías? -Borges

RE: About coordinated efforts on scientific software.

[JJ Gomez-Cadenas]

Hi again,
I have been looking into Babel
http://www.llnl.gov/CASC/components/babel.html

And it looks as a promising way of generating common interfaces for all
major scientific dialects.

However, I have a number of doubts. Let's see:

1) How much is the overhead to be payed? Very often in numerical
calculations performance is an issue.

2) SIDL, the scientific interface design language in which BABEL is based
has its own arrays and complex numbers. So it does GSL. Does that mean a lot
of copying to and fro, if one wants to stick to Babel types? Or one could
define GSL types, this seems to be easy enough in SIDL.

3) Python internals are based on Numphy which also has its own arrays. Same
question.

4) Is it worth? It depends on how many languages one really needs to glue
together. For C++ only it would clearly be an overkill. Add python and it
starts to be interesting. This two, I am sure are heavily used. Babel also
supports Java and Fortran. That makes it quite interesting, there is a lot
of people out there using FORTRAN still.

I would like that some body else (Alan?, Brian?) takes a look to this Babel
business and bring the discussion on line. I am becoming convinced that it
is a fundamental issue to be answered in order to help GSL to become the
core library for scientific calculations.

Best, JJ

-----Mensaje original-----
De: gsl-discuss-owner@sources.redhat.com
[mailto:gsl-discuss-owner@sources.redhat.com]En nombre de Alan
Aspuru-Guzik
Enviado el: lunes, 21 de octubre de 2002 19:05
Para: gsl-discuss@sources.redhat.com
Asunto: About coordinated efforts on scientific software.



This e-mail is a reply to Cristoph Siopi's email.

Some points:

There is a relevant coordinated effort by the Department of Energy for
creating a common base for scientific software. I actually went to a
workshop for the ACTS Toolkit:
http://acts.nersc.gov/

Which is a collection of software that is released using the GPL licencse
for most cases, and that is well suited for some of the tasks that people
need.  For example, if one needs parallel Matrix and Vector operations, I
recommend to take a look at PetSc. It looks really good.

I have been using GSL, and doing my vector MPI distribution by hand, just
because I did not know that such tools were in such advanced state, and
that they were actually written in C, etc.

Another very, VERY interesting effort, similar to what the GNOME people
wanted to do using CORBA as a glue for their component architecture,
Bonobo, is the Common Components Architecture:
http://www.cca-forum.org/

During the same workshop, I attended a one day tutorial on it, and it
seems very promising. When it becomes more mature, it would be very
interesting for one of us (maybe me) to CCA-ize the GSL, in such a way
that it can be used in conjuction with the tools from ACTS, that are
already starting to be CCAized (like TAO, the Toolkit for Advanced
Optimization).

So there are some people that are indeed looking at the big picture. Maybe
what we need in the volunteer arena is leadership and organization like
the one that started umbrella projects, such as GNOME or KDE, that brought
a lot of people together.

Greetings,
Alan

--

Alan Aspuru-Guzik                    Dios mueve al jugador, y éste, la
pieza.
(510)642-5911 UC Berkeley           ¿Qué Dios detrás de Dios la trama
empieza
(925)422-8739 LLNL                de polvo y tiempo y sueño y
agonías? -Borges

RE: About coordinated efforts on scientific software.

[Brian Gough]

A workshop about free scientific software would be interesting.

Something I have suggested in the past was a "steering committee" for
GSL, similar to SLATEC.  I sent an email to Fred James, chair of the
committee at CERN looking at numerical libraries, saying that this
could be an area where CERN could be involved.

Re: About coordinated efforts on scientific software.

[Christos Siopis]

On Tue, 22 Oct 2002, JJ Gomez-Cadenas wrote:

> Let me bring an important subject to the discussion, that of
> wrappers. If GSL is going to be truly the core of free-software
> numerical calculations, wrappers are needed. For HEP it seems
> one would like C++ and python wrappers. I have played for more
> than a year now with a C++ wrapper for GSL and a number of
> non-trivial issues emerge, such as the speed, correct use of
> error handling, minimizing of copying of structures and the
> like, which often end up by suggesting re-implementation rather
> than wrapping (at least for some parts of the library). It would
> be nice to understand better and to discuss all those issues.

Interesting issues, indeed! Some of our very own GSL developers have
thought about some of them, see e.g.:

http://t8web.lanl.gov/people/jungman/except.pdf

My little personal experience with this is that there is a dichotomy
between "big project" folks and "Joe Numeric" folks.

If you talk to people involved with the development and maintenance of big
codes, they will probably propose some elegant top-bottom design, which
usually requires an object-oriented programming methodology to implement
and, increasingly, component programming. These folks will also often
insist that any code bigger than a hundred lines needs to follow a similar
path :) and that "a project rarely starts with the intention to become
big" but in the end, it does, so better plan early on.

If you talk to Joe Numeric, he'll wonder why you need anything else but
Fortran, he'll tell you how annoyed he is by NAG's requirement to return
error numbers, and will reprogram a new single-variable equation solver
each time he needs one because he hates black boxes :)

Seriously though, i wonder if your experience with C++ wrapping ultimately
means that we are doomed to reprogram our libraries every time we move
from one language family to another (at least if we want to do a good
job)...

Christos