Which library implementation to use/work on?

Which library implementation to use/work on?

[Mike Hearn]

Hi there,

I'm investigating GCJ for a project of mine to open up a Java library
to the world of [Objective] C++ via the quite lovely CNI interface.

My project depends on the Google protocol buffers library. Attempting
to compile this jar reveals that there is a slight gap in
java.util.regex in GNU Classpath.

I'm trying to figure out what I should do with my time:

1) Write a patch to classpath - problem, project appears to be dead?
2) Try and somehow compile against Apache Harmony - problem, project
also appears to be dead?
3) Change the protobufs library so it avoids the stubs in Classpath

It seems that trying to support GCJ will keep my project at the level
of Java 1.5 (or 1.4?) forever, as it's no longer being developed,
which is a shame.

I recall that some time ago Sun open sourced Java, but I don't know if
it's compatible with what I want to do or how it plays together with
GCJ (if at all).

Any tips for what I should do?

thanks
-mike

Re: Which library implementation to use/work on?

[Andrew Haley]

On 03/13/2012 04:56 PM, Mike Hearn wrote:
> Hi there,
>
> I'm investigating GCJ for a project of mine to open up a Java library
> to the world of [Objective] C++ via the quite lovely CNI interface.
>
> My project depends on the Google protocol buffers library. Attempting
> to compile this jar reveals that there is a slight gap in
> java.util.regex in GNU Classpath.
>
> I'm trying to figure out what I should do with my time:
>
> 1) Write a patch to classpath - problem, project appears to be dead?
> 2) Try and somehow compile against Apache Harmony - problem, project
> also appears to be dead?
> 3) Change the protobufs library so it avoids the stubs in Classpath
>
> It seems that trying to support GCJ will keep my project at the level
> of Java 1.5 (or 1.4?) forever, as it's no longer being developed,
> which is a shame.
>
> I recall that some time ago Sun open sourced Java, but I don't know if
> it's compatible with what I want to do or how it plays together with
> GCJ (if at all).
>
> Any tips for what I should do?

I'd just fix up java.util.regex in Classpath as long as it wasn't
too hard.

Updating gcj to use the class library from Sun's now-free OpenJDK is
on my list of long-term projects.  It's a big job though, even for an
expert.

Andrew.

Re: Which library implementation to use/work on?

[Mike Hearn]

Just to double check, if OpenJDK is now under a compatible license,
does that mean copy/paste of code from it is OK?

Re: Which library implementation to use/work on?

[Andrew Haley]

On 03/13/2012 06:26 PM, Mike Hearn wrote:
> Just to double check, if OpenJDK is now under a compatible license,
> does that mean copy/paste of code from it is OK?

In theory yes, but all code contributed to GNU Classpath has to be
assigned to the Free Software Foundation.  You can't assign copyright
with code (like OpenJDK) that you don't own.  So OpenJDK code can't be
checked in to GNU Classpath.

Andrew.

Re: Which library implementation to use/work on?

[Mike Hearn]

Thanks (sorry for the broken threading, I'm subscribed now).

I understand the reasons for this policy. However, as you are planning
on (eventually) replacing Classpath with OpenJDK completely, an
exception in this case would seem to make logical sense. The code will
end up not owned by the FSF no matter what.

Re: Which library implementation to use/work on?

[Chris Burdess]

Mike Hearn wrote:
> I understand the reasons for this policy. However, as you are planning
> on (eventually) replacing Classpath with OpenJDK completely, an
> exception in this case would seem to make logical sense. The code will
> end up not owned by the FSF no matter what.

Well, no. The FSF will still own Classpath.

Re: Which library implementation to use/work on?

[Mike Hearn]

>> I understand the reasons for this policy. However, as you are planning
>> on (eventually) replacing Classpath with OpenJDK completely, an
>> exception in this case would seem to make logical sense. The code will
>> end up not owned by the FSF no matter what.
>
> Well, no. The FSF will still own Classpath.

Yes. But drawing a line in the sand and saying "after this point, we
no longer have the ability to relicense as we wish, but that's OK
because we don't care about this codebase anymore" seems prudent as a
migration strategy.

My basic issue is, given license compatibility, reimplementing things
purely so the FSF has the option of relicenseing a codebase that it
probably never will is not a good use of time. In practice it just
means people won't submit patches because unless they're working on
AWT or some other library where copy/pasting code is hard, it never
makes sense to upstream patches. That in turn makes it harder for
people to benefit from those patches.

Re: Which library implementation to use/work on?

[Andrew Haley]

On 03/14/2012 11:11 AM, Mike Hearn wrote:
>>> I understand the reasons for this policy. However, as you are planning
>>> on (eventually) replacing Classpath with OpenJDK completely, an
>>> exception in this case would seem to make logical sense. The code will
>>> end up not owned by the FSF no matter what.
>>
>> Well, no. The FSF will still own Classpath.
> 
> Yes. But drawing a line in the sand and saying "after this point, we
> no longer have the ability to relicense as we wish, but that's OK
> because we don't care about this codebase anymore" seems prudent as a
> migration strategy.

We're not there yet.

> My basic issue is, given license compatibility, reimplementing things
> purely so the FSF has the option of relicenseing a codebase that it
> probably never will is not a good use of time. In practice it just
> means people won't submit patches because unless they're working on
> AWT or some other library where copy/pasting code is hard, it never
> makes sense to upstream patches. That in turn makes it harder for
> people to benefit from those patches.

Fair enough.

Andrew.

Re: Which library implementation to use/work on?

[Andrew Hughes]

----- Original Message -----
> Hi there,
> 
> I'm investigating GCJ for a project of mine to open up a Java library
> to the world of [Objective] C++ via the quite lovely CNI interface.
> 
> My project depends on the Google protocol buffers library. Attempting
> to compile this jar reveals that there is a slight gap in
> java.util.regex in GNU Classpath.
> 

Can you provide details please?

> I'm trying to figure out what I should do with my time:
> 
> 1) Write a patch to classpath - problem, project appears to be dead?

Hardly:

http://icedtea.classpath.org/mirror/git/classpath/classpath/ (commits 6 days ago)
http://developer.classpath.org/pipermail/classpath/2012-March/thread.html (active discussion)

How is this dead?  There's less than there was, yes, but that's because Sun (now Oracle)
have largely succeeded in persuading people to switch to their less-open OpenJDK project.
We always had problems generating interest before OpenJDK (quite a lot of people seemed happy
to put up with the proprietary but gratis JDK downloads from Sun).  Having a more complete
open source project is obviously going to sway those who want a full JDK now (notably distros), but
there will be those still interested in the intellectual challenge of producing a JDK themselves.
I think there are clear issues with having a single implementation of a platform; it leads to
confusion between what is part of the specification and what is merely an artefact of implementation,
an issue we've hit many times in developing GNU Classpath.

> 2) Try and somehow compile against Apache Harmony - problem, project
> also appears to be dead?

I can't comment much on Harmony, but it always seemed to have less of a user-driven community than
Classpath to begin with and has officially been put in the 'attic' as far as I'm aware.

> 3) Change the protobufs library so it avoids the stubs in Classpath
> 
> It seems that trying to support GCJ will keep my project at the level
> of Java 1.5 (or 1.4?) forever, as it's no longer being developed,
> which is a shame.
> 

It would help if you did a little research before making such claims.

> I recall that some time ago Sun open sourced Java, but I don't know
> if
> it's compatible with what I want to do or how it plays together with
> GCJ (if at all).
> 

Depends on what you want to do.  It doesn't have CNI.

> Any tips for what I should do?
> 
> thanks
> -mike
> 

-- 
Andrew :)

Free Java Software Engineer
Red Hat, Inc. (http://www.redhat.com)

PGP Key: 248BDC07 (https://keys.indymedia.org/)
Fingerprint = EC5A 1F5E C0AD 1D15 8F1F  8F91 3B96 A578 248B DC07

Re: Which library implementation to use/work on?

[Mike Hearn]

> Can you provide details please?

It's missing the usePattern method.

> How is this dead?

My mistake. I was looking at the website which does not appear to have
been updated (or talk about new releases) since 2009:

http://www.gnu.org/software/classpath/

It's the one that's linked from the GCJ page.

> It would help if you did a little research before making such claims.

Well, that's what I'm doing right now :-)

What I'm really after is CNI. I'll check out the latest classpath and
take a look at how hard it'd be to implement the missing methods.

Re: Which library implementation to use/work on?

[Andrew Hughes]

----- Original Message -----
> > Can you provide details please?
> 
> It's missing the usePattern method.

Ok.  This still seems to be missing in GNU Classpath (quoteReplacement was
added recently) but it looks trivial to implement.  I'll have a look myself
later today.

> 
> > How is this dead?
> 
> My mistake. I was looking at the website which does not appear to
> have
> been updated (or talk about new releases) since 2009:
> 
> http://www.gnu.org/software/classpath/
> 
> It's the one that's linked from the GCJ page.
> 

Yes, that's because the page only lists releases and there hasn't been one in
a while.  We are on the verge of 0.99; the work is done, but we're currently
stalled on GNU project admin issues in uploading the tarball :-(  I'm working
on merging 0.99 into gcj right now.

It's always a good idea to check the mailing lists of a project before posting.

> > It would help if you did a little research before making such
> > claims.
> 
> Well, that's what I'm doing right now :-)
> 

Ah good!  It just comes across a little offensive to call a project dead when
people are actively working on it :-)

> What I'm really after is CNI. I'll check out the latest classpath and
> take a look at how hard it'd be to implement the missing methods.

For CNI, you will need gcj.  Keep an eye on java-patches@gcc.gnu.org for the latest
Classpath code being merged in.

> 

-- 
Andrew :)

Free Java Software Engineer
Red Hat, Inc. (http://www.redhat.com)

PGP Key: 248BDC07 (https://keys.indymedia.org/)
Fingerprint = EC5A 1F5E C0AD 1D15 8F1F  8F91 3B96 A578 248B DC07

Re: Which library implementation to use/work on?

[Mike Hearn]

> Ok.  This still seems to be missing in GNU Classpath (quoteReplacement was
> added recently) but it looks trivial to implement.  I'll have a look myself
> later today.

Thanks! That'd be fantastic. Great to hear that a new release is imminent.

> It's always a good idea to check the mailing lists of a project before posting.

Yeah, sorry, I checked this list (which is pretty quiet) and the
classpath@ list:

http://lists.gnu.org/archive/html/classpath/

which has not seen any posts for ~2 years. But I see that indeed the
patches list has activity.

Re: Which library implementation to use/work on?

[Andrew Hughes]

----- Original Message -----
> > Ok.  This still seems to be missing in GNU Classpath
> > (quoteReplacement was
> > added recently) but it looks trivial to implement.  I'll have a
> > look myself
> > later today.
>
> Thanks! That'd be fantastic. Great to hear that a new release is
> imminent.
>
> > It's always a good idea to check the mailing lists of a project
> > before posting.
>
> Yeah, sorry, I checked this list (which is pretty quiet) and the
> classpath@ list:
>
> http://lists.gnu.org/archive/html/classpath/
>
> which has not seen any posts for ~2 years. But I see that indeed the
> patches list has activity.
>

Ah, the list moved here: http://developer.classpath.org/pipermail/classpath/

Was there an old link you followed to find that one?  If so, we'll try and update
it.  I don't remember now why it was moved, but as you can see, that has posts
in the last month, including ones about the release.

Hope that helps,
--
Andrew :)

Free Java Software Engineer
Red Hat, Inc. (http://www.redhat.com)

PGP Key: 248BDC07 (https://keys.indymedia.org/)
Fingerprint = EC5A 1F5E C0AD 1D15 8F1F  8F91 3B96 A578 248B DC07

Re: Which library implementation to use/work on?

[Mike Hearn]

> Was there an old link you followed to find that one?  If so, we'll try and
> update it.

Well, I don't think I did anything unexpected.

1) Search google for "gnu classpath"
2) Go to the website: http://www.gnu.org/software/classpath/
3) Click mailing lists in the sidebar
4) Click classpath

I see now there's a message at the top of the mailman page saying it's
temporarily moved. I guess I didn't spot that before because I knew I
wanted the archives.

Re: Which library implementation to use/work on?

[Andrew Hughes]

----- Original Message -----
> > Was there an old link you followed to find that one?  If so, we'll
> > try and
> > update it.
>
> Well, I don't think I did anything unexpected.
>
> 1) Search google for "gnu classpath"
> 2) Go to the website: http://www.gnu.org/software/classpath/
> 3) Click mailing lists in the sidebar
> 4) Click classpath
>
> I see now there's a message at the top of the mailman page saying
> it's
> temporarily moved. I guess I didn't spot that before because I knew I
> wanted the archives.
>

Thanks for the feedback.  I'll look into either reinstating the original
lists with full archives or switching the link to go directly to the current ones.
--
Andrew :)

Free Java Software Engineer
Red Hat, Inc. (http://www.redhat.com)

PGP Key: 248BDC07 (https://keys.indymedia.org/)
Fingerprint = EC5A 1F5E C0AD 1D15 8F1F  8F91 3B96 A578 248B DC07

Re: Which library implementation to use/work on?

[Mike Hearn]

Thanks Andrew, I see that you checked in usePattern a few days ago.
Any idea when it might get synced to GCC and released?

Unfortunately I had trouble compiling it. I took gcc 4.7 and brought
across the latest Matcher.java. However it appears that you aren't
supposed to edit the copy of classpath included with GCC. Introducing
syntax errors into the file and then running make shows that the .java
file itself does not appear to actually be read. Instead the .class
files are included the tree as well and I didn't see any obvious
script or build step that is supposed to regenerate them.

Are there instructions anywhere on what I'm supposed to do here?

Re: Which library implementation to use/work on?

[Bryce McKinlay]

On Sat, Apr 7, 2012 at 4:29 PM, Mike Hearn <mike@plan99.net> wrote:

> Unfortunately I had trouble compiling it. I took gcc 4.7 and brought
> across the latest Matcher.java. However it appears that you aren't
> supposed to edit the copy of classpath included with GCC. Introducing
> syntax errors into the file and then running make shows that the .java
> file itself does not appear to actually be read. Instead the .class
> files are included the tree as well and I didn't see any obvious
> script or build step that is supposed to regenerate them.
>
> Are there instructions anywhere on what I'm supposed to do here?

Try configuring with --enable-java-maintainer-mode - this should
ensure that the .java files get rebuilt.

See the "Java specific options" section at
http://gcc.gnu.org/install/configure.html

Bryce

Re: Which library implementation to use/work on?

[Mike Hearn]

Thanks, I got it to work eventually. That switch disables building of
ecj so I had to first compile without it, then install, then adjust my
path, then reconfigure/make/make install - bit confusing.

I then encountered a couple of other issues. Firstly it seems there's
something odd about the Google protocol buffers library that gcj does
not like:

com/google/protobuf/SingleFieldBuilder.java: In method
'com.google.protobuf.SingleFieldBuilder.mergeFrom(com.google.protobuf.GeneratedMessage)':
com/google/protobuf/SingleFieldBuilder.java:192:0: error: class
'com.google.protobuf.GeneratedMessage' has no method named
'getDefaultInstanceForType' matching signature
'()Lcom/google/protobuf/Message;'

The code is some morass of generics so it's hard for me to figure out
what the issue is here. The relevant code is:

public class SingleFieldBuilder
    <MType extends GeneratedMessage,
     BType extends GeneratedMessage.Builder,
     IType extends MessageOrBuilder>
    implements GeneratedMessage.BuilderParent {

  ....
  private MType message;

  public SingleFieldBuilder<MType, BType, IType> mergeFrom(
      MType value) {
    if (builder == null && message == message.getDefaultInstanceForType()) {
      message = value;
    } else {
      getBuilder().mergeFrom(value);
    }
    onChanged();
    return this;
  }
}

Sure enough the GeneratedMessage class doesn't have
getDefaultInstanceForType - it's implemented by subclasses.

GeneratedMessage says this at the top:


/**
 * All generated protocol message classes extend this class.  This class
 * implements most of the Message and Builder interfaces using Java reflection.
 * Users can ignore this class and pretend that generated messages implement
 * the Message interface directly.
 *
 * @author kenton@google.com Kenton Varda
 */

I suspect the reference to "implementing most of the interfaces using
reflection" is key. I don't know what that means exactly, the class is
very large and complicated:

http://code.google.com/p/protobuf/source/browse/trunk/java/src/main/java/com/google/protobuf/GeneratedMessage.java

I switched to the "lite" runtime which is simpler and the issue went away.

The next problem is more stubs in classpath: specifically the elliptic
curve related classes in java.security.spec. I'm already using
BouncyCastle to implement the EC crypto - these classes are literally
just data structures. They're probably quite trivial to do.

I'll try copying the equivalent code from openjdk and see if that works.

Re: Which library implementation to use/work on?

[Mike Hearn]

I finally got there, w00t! It turned out to be easier to switch to the
"lite" versions of BouncyCastle and protobufs than to upgrade
classpath, so that's what I did, and now I have a spiffy CNI exposed
API that can be used from C++, which is what I was after. Thanks!

On Sun, Apr 8, 2012 at 3:18 PM, Mike Hearn <mike@plan99.net> wrote:
> Thanks, I got it to work eventually. That switch disables building of
> ecj so I had to first compile without it, then install, then adjust my
> path, then reconfigure/make/make install - bit confusing.
>
> I then encountered a couple of other issues. Firstly it seems there's
> something odd about the Google protocol buffers library that gcj does
> not like:
>
> com/google/protobuf/SingleFieldBuilder.java: In method
> 'com.google.protobuf.SingleFieldBuilder.mergeFrom(com.google.protobuf.GeneratedMessage)':
> com/google/protobuf/SingleFieldBuilder.java:192:0: error: class
> 'com.google.protobuf.GeneratedMessage' has no method named
> 'getDefaultInstanceForType' matching signature
> '()Lcom/google/protobuf/Message;'
>
> The code is some morass of generics so it's hard for me to figure out
> what the issue is here. The relevant code is:
>
> public class SingleFieldBuilder
>    <MType extends GeneratedMessage,
>     BType extends GeneratedMessage.Builder,
>     IType extends MessageOrBuilder>
>    implements GeneratedMessage.BuilderParent {
>
>  ....
>  private MType message;
>
>  public SingleFieldBuilder<MType, BType, IType> mergeFrom(
>      MType value) {
>    if (builder == null && message == message.getDefaultInstanceForType()) {
>      message = value;
>    } else {
>      getBuilder().mergeFrom(value);
>    }
>    onChanged();
>    return this;
>  }
> }
>
> Sure enough the GeneratedMessage class doesn't have
> getDefaultInstanceForType - it's implemented by subclasses.
>
> GeneratedMessage says this at the top:
>
>
> /**
>  * All generated protocol message classes extend this class.  This class
>  * implements most of the Message and Builder interfaces using Java reflection.
>  * Users can ignore this class and pretend that generated messages implement
>  * the Message interface directly.
>  *
>  * @author kenton@google.com Kenton Varda
>  */
>
> I suspect the reference to "implementing most of the interfaces using
> reflection" is key. I don't know what that means exactly, the class is
> very large and complicated:
>
> http://code.google.com/p/protobuf/source/browse/trunk/java/src/main/java/com/google/protobuf/GeneratedMessage.java
>
> I switched to the "lite" runtime which is simpler and the issue went away.
>
> The next problem is more stubs in classpath: specifically the elliptic
> curve related classes in java.security.spec. I'm already using
> BouncyCastle to implement the EC crypto - these classes are literally
> just data structures. They're probably quite trivial to do.
>
> I'll try copying the equivalent code from openjdk and see if that works.

Re: Which library implementation to use/work on?

[Mike Hearn]

Sorry Bryce, +mailing list ...

On Wed, May 30, 2012 at 11:13 PM, Mike Hearn <mike@plan99.net> wrote:
> Ugh, mailing lists that reject HTML ...
>
> OK, looks like GMP does indeed implement "next probable prime". All
> that's required is plumbing it through.
>
>> One step forward, one step back.
>>
>> I recently upgraded to the latest Bouncy Castle. It now contains an
>> implementation of "NTRU", an obscure crypto system that is theoretically
>> resistant to quantum computer attacks.
>>
>> Unfortunately the NTRU library uses BigInteger.nextProbablePrime(), which
>> is not implemented. I don't know if the GNU biginteger library makes it easy
>> to support this, so will take a quick look at how easy it is to expunge NTRU
>> from Bouncy Castle.
>>
>> On Tue, Apr 10, 2012 at 5:53 PM, Mike Hearn <mike@plan99.net> wrote:
>>>
>>> I finally got there, w00t! It turned out to be easier to switch to the
>>> "lite" versions of BouncyCastle and protobufs than to upgrade
>>> classpath, so that's what I did, and now I have a spiffy CNI exposed
>>> API that can be used from C++, which is what I was after. Thanks!
>>>
>>> On Sun, Apr 8, 2012 at 3:18 PM, Mike Hearn <mike@plan99.net> wrote:
>>> > Thanks, I got it to work eventually. That switch disables building of
>>> > ecj so I had to first compile without it, then install, then adjust my
>>> > path, then reconfigure/make/make install - bit confusing.
>>> >
>>> > I then encountered a couple of other issues. Firstly it seems there's
>>> > something odd about the Google protocol buffers library that gcj does
>>> > not like:
>>> >
>>> > com/google/protobuf/SingleFieldBuilder.java: In method
>>> >
>>> > 'com.google.protobuf.SingleFieldBuilder.mergeFrom(com.google.protobuf.GeneratedMessage)':
>>> > com/google/protobuf/SingleFieldBuilder.java:192:0: error: class
>>> > 'com.google.protobuf.GeneratedMessage' has no method named
>>> > 'getDefaultInstanceForType' matching signature
>>> > '()Lcom/google/protobuf/Message;'
>>> >
>>> > The code is some morass of generics so it's hard for me to figure out
>>> > what the issue is here. The relevant code is:
>>> >
>>> > public class SingleFieldBuilder
>>> >    <MType extends GeneratedMessage,
>>> >     BType extends GeneratedMessage.Builder,
>>> >     IType extends MessageOrBuilder>
>>> >    implements GeneratedMessage.BuilderParent {
>>> >
>>> >  ....
>>> >  private MType message;
>>> >
>>> >  public SingleFieldBuilder<MType, BType, IType> mergeFrom(
>>> >      MType value) {
>>> >    if (builder == null && message ==
>>> > message.getDefaultInstanceForType()) {
>>> >      message = value;
>>> >    } else {
>>> >      getBuilder().mergeFrom(value);
>>> >    }
>>> >    onChanged();
>>> >    return this;
>>> >  }
>>> > }
>>> >
>>> > Sure enough the GeneratedMessage class doesn't have
>>> > getDefaultInstanceForType - it's implemented by subclasses.
>>> >
>>> > GeneratedMessage says this at the top:
>>> >
>>> >
>>> > /**
>>> >  * All generated protocol message classes extend this class.  This
>>> > class
>>> >  * implements most of the Message and Builder interfaces using Java
>>> > reflection.
>>> >  * Users can ignore this class and pretend that generated messages
>>> > implement
>>> >  * the Message interface directly.
>>> >  *
>>> >  * @author kenton@google.com Kenton Varda
>>> >  */
>>> >
>>> > I suspect the reference to "implementing most of the interfaces using
>>> > reflection" is key. I don't know what that means exactly, the class is
>>> > very large and complicated:
>>> >
>>> >
>>> > http://code.google.com/p/protobuf/source/browse/trunk/java/src/main/java/com/google/protobuf/GeneratedMessage.java
>>> >
>>> > I switched to the "lite" runtime which is simpler and the issue went
>>> > away.
>>> >
>>> > The next problem is more stubs in classpath: specifically the elliptic
>>> > curve related classes in java.security.spec. I'm already using
>>> > BouncyCastle to implement the EC crypto - these classes are literally
>>> > just data structures. They're probably quite trivial to do.
>>> >
>>> > I'll try copying the equivalent code from openjdk and see if that
>>> > works.
>>
>>