Dumb idea for accelerating FOSS development

Dumb idea for accelerating FOSS development

[Bill Cox]

First, the problem:  I've got a C library I want to share.  There are
many users who want to use it.  This should be as easy as breathing,
but it's not!

My users and I face what I'm calling "GNU/Linux Innovation Red Tape".
This library is two files: sonic.c and sonic.h.  To share it in
Debian, first, I had to learn how to make a clean Debian package.
Something like 30 files are involved, and in my case a few days of
learning and laboring.  Next, I have to grovel on the debian-mentor's
list, and pray that Debian Sponsor Gods will take pity on me.  No
response from the Sponsor Gods so far...  maybe I'll have more luck
after they've finished the current release.  Shouldn't we be ashamed
that getting a package into Debian is harder than publishing an app
for iPhones?  Assuming I find a Debian sponsor, my package will be
uploaded to Unstable, then Testing, and after a few years, Stable.

Guess where my package has to be before my users are willing to link
to it?  Debian Stable.  We're hosed.  There's just no freaking way for
my users to use my library.  You know what they do?  Every one of them
bypasses the Innovation Red Tape, and simply copies my source code
directly into their application.  How's that as a reuse paradigm?

My dumb idea for today is a way to break through this red tape.
Unfortunately, while I could implement this idea in a few days, the
red tape would keep it in limbo so long that I'll likely die of old
age before it gets into Debian Stable.  Oh, well... here's the dumb
idea anyway...  In short, support a syntax in gcc like:

    $ gcc myprog.c -lgit://github/~waywardgeek/sonic=0.1

In theory, this would cause gcc to do the following:

    - Check and see if the requested library has already been
downloaded and built, and still in the cache
    - If not, download the latest compatible signed version, verify
the signature, and compile it
    - link to the library in the cache

A feature like this would make sharing C libraries as easy as
breathing.  My users wouldn't even have to 'apt-get install
libsonic0-dev'.  They would stop copying my source code, and just link
to my library.  Sharing of C libraries in FOSS land might increase
dramatically.  The year of GNU/Linux on the desktop would finally
come, and we'd all march off to Paradise as heroes.  Or not.

Anyway, just my dumb idea for the day... If through some miracle this
particular dumb idea appeals to the GCC Gods, I volunteer to write it.

Bill

Re: Dumb idea for accelerating FOSS development

[Basile Starynkevitch]

On Wed, Jan 12, 2011 at 06:07:48AM -0500, Bill Cox wrote:
> Unfortunately, while I could implement this idea in a few days, the
> red tape would keep it in limbo so long that I'll likely die of old
> age before it gets into Debian Stable.  Oh, well... here's the dumb
> idea anyway...  In short, support a syntax in gcc like:
> 
>     $ gcc myprog.c -lgit://github/~waywardgeek/sonic=0.1
> 
> In theory, this would cause gcc to do the following:
> 
>     - Check and see if the requested library has already been
> downloaded and built, and still in the cache
>     - If not, download the latest compatible signed version, verify
> the signature, and compile it
>     - link to the library in the cache

At a first glance, I find your idea crazy but nice (but what about header
files *.h of the requested library?). I would be delighted to have it. Maybe
others don't think that way (in particular, those not working on Linux, and
those not working on a compiler running with a network connection).

Very probably ccache is the better place to implement that, since it already do
some (preprocessed) source code cache. http://ccache.samba.org/

However, my personal feeling is that GCC is quite conservative on the user
features it is accepting. So I won't bet that your idea will be positively
accepted.

Nevertheless, you could propose a patch to GCC to implement what you are
dreaming of. See http://gcc.gnu.org/contribute.html for more (and don't
forget the legal part, i.e. the copyright assignment to FSF).

I do find your idea nice, but I am quite pessimistic on the chances it has
to get accepted into the GCC trunk. Don't expect others to work on it. If
you really want it, work hard to make a patch for it and propose that
working patch.

Besides, what you propose probably need mostly also a support in the build
tools, in particular inside GNU make. Probably, ccache & make are a better
place for it.

So while I am intellectually appealed by your idea, I am quite pessimistic
in practice to get it accepted (even within several years), and it probably
belongs more to other tools (make, ccache) than to GCC.

However, to have the GCC Community to consider your idea, you'll have to
first show a patch implementing it.

Maybe, what you are suggesting would be better dealt with a plugin machinery
inside gcc itself (we don't have any plugin hooks inside gcc/gcc.h yet).

Regards.

PS. I would believe most GCC contributors won't be interested.

-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***

Re: Dumb idea for accelerating FOSS development

[Robert Millan]

2011/1/12 Bill Cox <waywardgeek@gmail.com>:
>    $ gcc myprog.c -lgit://github/~waywardgeek/sonic=0.1

You already have this, it's called FUSE. E.g.

$ sshfs $publicrepo $tmp
$ gcc myprog.c -I$tmp $tmp/sonic.c

If you want it to speak GIT protocol, just write a GIT
extension, etc.

-- 
Robert Millan

Re: Dumb idea for accelerating FOSS development

[Axel Freyn]

Hi,
On Wed, Jan 12, 2011 at 01:43:51PM +0100, Basile Starynkevitch wrote:
> On Wed, Jan 12, 2011 at 06:07:48AM -0500, Bill Cox wrote:
> > Unfortunately, while I could implement this idea in a few days, the
> > red tape would keep it in limbo so long that I'll likely die of old
> > age before it gets into Debian Stable.  Oh, well... here's the dumb
> > idea anyway...  In short, support a syntax in gcc like:
> > 
> >     $ gcc myprog.c -lgit://github/~waywardgeek/sonic=0.1
> > 
> > In theory, this would cause gcc to do the following:
> > 
> >     - Check and see if the requested library has already been
> > downloaded and built, and still in the cache
> >     - If not, download the latest compatible signed version, verify
> > the signature, and compile it
> >     - link to the library in the cache
> 
> At a first glance, I find your idea crazy but nice (but what about header
> files *.h of the requested library?). I would be delighted to have it. Maybe
> others don't think that way (in particular, those not working on Linux, and
> those not working on a compiler running with a network connection).
> 
> Very probably ccache is the better place to implement that, since it already do
> some (preprocessed) source code cache. http://ccache.samba.org/
> 
> However, my personal feeling is that GCC is quite conservative on the user
> features it is accepting. So I won't bet that your idea will be positively
> accepted.
Well, it is not exactly what you are asking for, but the depot_tools
used in the chromium developement might be a solution:
http://www.chromium.org/developers/how-tos/depottools
The can keep track of different libraries checkout out from svn or git
repositories, and calling "gclient sync" connects to all repositories
belonging to the project and verifies that all files are up-to-date.
May be that would be a better starting point than using gcc?

Axel

Re: Dumb idea for accelerating FOSS development

[Bill Cox]

On Wed, Jan 12, 2011 at 8:43 AM, Axel Freyn <axel-freyn@gmx.de> wrote:
> Hi,
> On Wed, Jan 12, 2011 at 01:43:51PM +0100, Basile Starynkevitch wrote:
>> On Wed, Jan 12, 2011 at 06:07:48AM -0500, Bill Cox wrote:
>> > Unfortunately, while I could implement this idea in a few days, the
>> > red tape would keep it in limbo so long that I'll likely die of old
>> > age before it gets into Debian Stable.  Oh, well... here's the dumb
>> > idea anyway...  In short, support a syntax in gcc like:
>> >
>> >     $ gcc myprog.c -lgit://github/~waywardgeek/sonic=0.1
>> >
>> > In theory, this would cause gcc to do the following:
>> >
>> >     - Check and see if the requested library has already been
>> > downloaded and built, and still in the cache
>> >     - If not, download the latest compatible signed version, verify
>> > the signature, and compile it
>> >     - link to the library in the cache
>>
>> At a first glance, I find your idea crazy but nice (but what about header
>> files *.h of the requested library?). I would be delighted to have it. Maybe
>> others don't think that way (in particular, those not working on Linux, and
>> those not working on a compiler running with a network connection).
>>
>> Very probably ccache is the better place to implement that, since it already do
>> some (preprocessed) source code cache. http://ccache.samba.org/
>>
>> However, my personal feeling is that GCC is quite conservative on the user
>> features it is accepting. So I won't bet that your idea will be positively
>> accepted.
> Well, it is not exactly what you are asking for, but the depot_tools
> used in the chromium developement might be a solution:
> http://www.chromium.org/developers/how-tos/depottools
> The can keep track of different libraries checkout out from svn or git
> repositories, and calling "gclient sync" connects to all repositories
> belonging to the project and verifies that all files are up-to-date.
> May be that would be a better starting point than using gcc?
>
> Axel
>

Well, after a short nap, the thought of fixing this in gcc seems even
dumber to me, though the problem is quite real.  Another tool called
before gcc could get the header and library files into a place where
they could be used.  It could even be compiler independent.

A bit of Googling reveals Zero Install.  Looks like these guys are
trying to tackle the problem.

Bill

Re: Dumb idea for accelerating FOSS development

[David Brown]

On 12/01/2011 12:07, Bill Cox wrote:
> First, the problem:  I've got a C library I want to share.  There are
> many users who want to use it.  This should be as easy as breathing,
> but it's not!
>
> My users and I face what I'm calling "GNU/Linux Innovation Red Tape".
> This library is two files: sonic.c and sonic.h.  To share it in
> Debian, first, I had to learn how to make a clean Debian package.
> Something like 30 files are involved, and in my case a few days of
> learning and laboring.  Next, I have to grovel on the debian-mentor's
> list, and pray that Debian Sponsor Gods will take pity on me.  No
> response from the Sponsor Gods so far...  maybe I'll have more luck
> after they've finished the current release.  Shouldn't we be ashamed
> that getting a package into Debian is harder than publishing an app
> for iPhones?  Assuming I find a Debian sponsor, my package will be
> uploaded to Unstable, then Testing, and after a few years, Stable.
>
> Guess where my package has to be before my users are willing to link
> to it?  Debian Stable.  We're hosed.  There's just no freaking way for
> my users to use my library.  You know what they do?  Every one of them
> bypasses the Innovation Red Tape, and simply copies my source code
> directly into their application.  How's that as a reuse paradigm?
>
> My dumb idea for today is a way to break through this red tape.
> Unfortunately, while I could implement this idea in a few days, the
> red tape would keep it in limbo so long that I'll likely die of old
> age before it gets into Debian Stable.  Oh, well... here's the dumb
> idea anyway...  In short, support a syntax in gcc like:
>
>      $ gcc myprog.c -lgit://github/~waywardgeek/sonic=0.1
>
> In theory, this would cause gcc to do the following:
>
>      - Check and see if the requested library has already been
> downloaded and built, and still in the cache
>      - If not, download the latest compatible signed version, verify
> the signature, and compile it
>      - link to the library in the cache
>
> A feature like this would make sharing C libraries as easy as
> breathing.  My users wouldn't even have to 'apt-get install
> libsonic0-dev'.  They would stop copying my source code, and just link
> to my library.  Sharing of C libraries in FOSS land might increase
> dramatically.  The year of GNU/Linux on the desktop would finally
> come, and we'd all march off to Paradise as heroes.  Or not.
>
> Anyway, just my dumb idea for the day... If through some miracle this
> particular dumb idea appeals to the GCC Gods, I volunteer to write it.
>
> Bill
>

I can see the point of your idea, but I think it's too limited, and gcc 
is the wrong place to put it.

Are you familiar with Python?  Python has a system called 
"easy_install", which lets you easy download libraries or packages for 
Python (as Python is interpreted, there is no distinction between 
development libraries and runtime libraries).  If your package were a 
python library, you would publish it in at one of the commonly used 
Python package library sites, and users could then type "easy_install 
sonic" and use your library.

I believe Perl has a similar idea, as does (La)TeX.

The reason I think gcc is the wrong place for such a feature, is that 
there are many parts to a library.  Most importantly, there is the 
headers, the source files, the configuration files, the documentation, 
the static libraries, and the dynamic libraries.  Many of these parts 
need to be installed before getting as far as the gcc command line.

Secondly, git is just one of many ways to get hold of the files.  A 
useful tool would support a range of protocols, including http, svn, mg, 
etc.  Practically speaking, this means a separate tool and plugins.

Thirdly, to be truly useful the tool should support searching of a 
number of standard sites (github, sourceforge, google code, etc.) 
automatically, to make the job as easy as possible for the user.

Fourthly, it should support additional features such as searching, 
updating, etc.

In other words, what I am really suggesting here is a sort of 
specialised package manager that can be used in addition to the 
distribution's main package manager.

Of course, it is important to consider conflicts with the distribution's 
libraries - who has priority if there are different versions?

Re: Dumb idea for accelerating FOSS development

[Jonathan Wakely]

On 12 January 2011 14:07, Bill Cox wrote:
>
> Well, after a short nap, the thought of fixing this in gcc seems even
> dumber to me, though the problem is quite real.  Another tool called
> before gcc could get the header and library files into a place where
> they could be used.  It could even be compiler independent.

  gcc myprog.c $(magic-lib-installer $publicrepo)

where magic-lib-installer takes care of any fetching, building and
installing of the lib in $tmp, then printing the necessary
-I$tmp/include, -L$tmp/lib and -lblah options for use by gcc.

However it  works, it shouldn't need changes to gcc.

Re: Dumb idea for accelerating FOSS development

[Ian Lance Taylor]

Bill Cox <waywardgeek@gmail.com> writes:

>     $ gcc myprog.c -lgit://github/~waywardgeek/sonic=0.1

In Go we have a program goinstall which looks at import statements and
pulls in required libraries, where the libraries are named based on
where the sources live.  A similar process could work in the C/C++
world, based on #include statements.

http://golang.org/cmd/goinstall/

Ian

Re: Dumb idea for accelerating FOSS development

[Florian Weimer]

* Ian Lance Taylor:

> Bill Cox <waywardgeek@gmail.com> writes:
>
>>     $ gcc myprog.c -lgit://github/~waywardgeek/sonic=0.1
>
> In Go we have a program goinstall which looks at import statements and
> pulls in required libraries, where the libraries are named based on
> where the sources live.  A similar process could work in the C/C++
> world, based on #include statements.

Oh.  Has anybody tried to implement this for passing the appropriate
-L/-l flags to the linker?  And why isn't everybody doing this? 8-)