Incompatibility between the Android runtime and Kawa-generated field names

Incompatibility between the Android runtime and Kawa-generated field names

[Kay Zheng]

Hi folks,

I built a simple Android app against Kawa's master branch. I've been
able to generate the APK, but the ART (Android runtime) kept
complaining about this when trying to launch the app:

    08-30 17:57:28.998 15990 15990 W art     : Failure to verify dex
file '/data/app/com.theerrorlog.superbapp-1/base.apk': Invalid field
name: '1+'

It turned out that some class files in Kawa's standard library
contained fields with exotic names such as '1+', '1-',
'%provide%srfi-41', which seemed to be legal JVM names, but treated as
illegal by the ART.

And I noticed that the 2.4 branch used a different naming scheme that
"translates" these exotic names into names with "escape sequences".
For example '1+' will be renamed to '$N1$Pl', making the translated
names all legal ART names too.

So what's the reason for abandoning the "escaping" naming scheme? Can
I remedy the incompatibility issue in some way?

Regards,
Kay Z.

Re: Incompatibility between the Android runtime and Kawa-generated field names

[Per Bothner]

On 08/30/2017 09:10 AM, Kay Zheng wrote:
>      08-30 17:57:28.998 15990 15990 W art     : Failure to verify dex
> file '/data/app/com.theerrorlog.superbapp-1/base.apk': Invalid field
> name: '1+'
> 
> It turned out that some class files in Kawa's standard library
> contained fields with exotic names such as '1+', '1-',
> '%provide%srfi-41', which seemed to be legal JVM names, but treated as
> illegal by the ART.

Could you check if this limitation has been fixed in the latest Android?
If so, we could select the old "mangling" style based on a #ifdef JAVA8
pre-processor directive.

> And I noticed that the 2.4 branch used a different naming scheme that
> "translates" these exotic names into names with "escape sequences".
> For example '1+' will be renamed to '$N1$Pl', making the translated
> names all legal ART names too.
> 
> So what's the reason for abandoning the "escaping" naming scheme? Can
> I remedy the incompatibility issue in some way?

One reason for avoiding needless escaping is to have more readable stack traces.

Another is the new escape scheme (which escapes many fewer characters)
appears to be somewhat standard, and used by other languages:
ttps://blogs.oracle.com/jrose/entry/symbolic_freedom_in_the_vm
That means it is more like to be supported by tools such as IDEs.
-- 
	--Per Bothner
per@bothner.com   http://per.bothner.com/

Re: Incompatibility between the Android runtime and Kawa-generated field names

[Kay Zheng]

Unfortunately, the verification of the names seems to occur only when
installing an app, and I don't have access to an Android 8 device, nor
can I run an emulator at the moment.

But I did come across the official naming specification for dex files though:

    https://source.android.com/devices/tech/dalvik/dex-format#membername

According to the documentation, the limitation still applies.

2017-08-31 0:24 GMT+08:00 Per Bothner <per@bothner.com>:
> On 08/30/2017 09:10 AM, Kay Zheng wrote:
>>
>>      08-30 17:57:28.998 15990 15990 W art     : Failure to verify dex
>> file '/data/app/com.theerrorlog.superbapp-1/base.apk': Invalid field
>> name: '1+'
>>
>> It turned out that some class files in Kawa's standard library
>> contained fields with exotic names such as '1+', '1-',
>> '%provide%srfi-41', which seemed to be legal JVM names, but treated as
>> illegal by the ART.
>
>
> Could you check if this limitation has been fixed in the latest Android?
> If so, we could select the old "mangling" style based on a #ifdef JAVA8
> pre-processor directive.
>
>> And I noticed that the 2.4 branch used a different naming scheme that
>> "translates" these exotic names into names with "escape sequences".
>> For example '1+' will be renamed to '$N1$Pl', making the translated
>> names all legal ART names too.
>>
>> So what's the reason for abandoning the "escaping" naming scheme? Can
>> I remedy the incompatibility issue in some way?
>
>
> One reason for avoiding needless escaping is to have more readable stack
> traces.
>
> Another is the new escape scheme (which escapes many fewer characters)
> appears to be somewhat standard, and used by other languages:
> ttps://blogs.oracle.com/jrose/entry/symbolic_freedom_in_the_vm
> That means it is more like to be supported by tools such as IDEs.
> --
>         --Per Bothner
> per@bothner.com   http://per.bothner.com/

Re: Incompatibility between the Android runtime and Kawa-generated field names

[Kay Zheng]

Does the old "mangling" code still exist somewhere? Could you point it
out for me, to see if I can do something with it?

2017-08-31 0:41 GMT+08:00 Kay Zheng <l04m33@gmail.com>:
> Unfortunately, the verification of the names seems to occur only when
> installing an app, and I don't have access to an Android 8 device, nor
> can I run an emulator at the moment.
>
> But I did come across the official naming specification for dex files though:
>
>     https://source.android.com/devices/tech/dalvik/dex-format#membername
>
> According to the documentation, the limitation still applies.
>
> 2017-08-31 0:24 GMT+08:00 Per Bothner <per@bothner.com>:
>> On 08/30/2017 09:10 AM, Kay Zheng wrote:
>>>
>>>      08-30 17:57:28.998 15990 15990 W art     : Failure to verify dex
>>> file '/data/app/com.theerrorlog.superbapp-1/base.apk': Invalid field
>>> name: '1+'
>>>
>>> It turned out that some class files in Kawa's standard library
>>> contained fields with exotic names such as '1+', '1-',
>>> '%provide%srfi-41', which seemed to be legal JVM names, but treated as
>>> illegal by the ART.
>>
>>
>> Could you check if this limitation has been fixed in the latest Android?
>> If so, we could select the old "mangling" style based on a #ifdef JAVA8
>> pre-processor directive.
>>
>>> And I noticed that the 2.4 branch used a different naming scheme that
>>> "translates" these exotic names into names with "escape sequences".
>>> For example '1+' will be renamed to '$N1$Pl', making the translated
>>> names all legal ART names too.
>>>
>>> So what's the reason for abandoning the "escaping" naming scheme? Can
>>> I remedy the incompatibility issue in some way?
>>
>>
>> One reason for avoiding needless escaping is to have more readable stack
>> traces.
>>
>> Another is the new escape scheme (which escapes many fewer characters)
>> appears to be somewhat standard, and used by other languages:
>> ttps://blogs.oracle.com/jrose/entry/symbolic_freedom_in_the_vm
>> That means it is more like to be supported by tools such as IDEs.
>> --
>>         --Per Bothner
>> per@bothner.com   http://per.bothner.com/

Re: Incompatibility between the Android runtime and Kawa-generated field names

[Per Bothner]

On 08/30/2017 10:00 AM, Kay Zheng wrote:
> Does the old "mangling" code still exist somewhere? Could you point it
> out for me, to see if I can do something with it?

Looks like its still there, in Language.mangleName.

However, I think it's best for me to do it.  I've already started looking
into it, and the actual amount of work looks fairly easy.  Easier than
explaining it ...

-- 
	--Per Bothner
per@bothner.com   http://per.bothner.com/

Re: Incompatibility between the Android runtime and Kawa-generated field names

[Kay Zheng]

Cool! Thank you for doing this. I'll just lay back a bit then :)

2017-08-31 1:03 GMT+08:00 Per Bothner <per@bothner.com>:
> On 08/30/2017 10:00 AM, Kay Zheng wrote:
>>
>> Does the old "mangling" code still exist somewhere? Could you point it
>> out for me, to see if I can do something with it?
>
>
> Looks like its still there, in Language.mangleName.
>
> However, I think it's best for me to do it.  I've already started looking
> into it, and the actual amount of work looks fairly easy.  Easier than
> explaining it ...
>
>
> --
>         --Per Bothner
> per@bothner.com   http://per.bothner.com/

Re: Incompatibility between the Android runtime and Kawa-generated field names

[Per Bothner]

On 08/30/2017 10:03 AM, Per Bothner wrote:
> On 08/30/2017 10:00 AM, Kay Zheng wrote:
>> Does the old "mangling" code still exist somewhere? Could you point it
>> out for me, to see if I can do something with it?
> 
> Looks like its still there, in Language.mangleName.
> 
> However, I think it's best for me to do it.  I've already started looking
> into it, and the actual amount of work looks fairly easy.  Easier than
> explaining it ...

Fix checked in.  Please test it.

The "controlling" change is the USE_SYMBOLIC field in
gnu/expr/Mangling.java.  This field's value depends on
whether Android is configured.

Some complications did turn up - though it was probably still
easier than explaining what to do ...

-- 
	--Per Bothner
per@bothner.com   http://per.bothner.com/

Re: Incompatibility between the Android runtime and Kawa-generated field names

[Kay Zheng]

Yes, I tested it and I think it worked. My app can launch cleanly. Thank you.

One little glitch though. I have built the Kawa source tree before,
and I had to remove patch-source-list manually after I applied the
fix, or Mangling.java wouldn't get pre-processed during the next
build. This had me confused for a while.

2017-08-31 23:39 GMT+08:00 Per Bothner <per@bothner.com>:
> On 08/30/2017 10:03 AM, Per Bothner wrote:
>>
>> On 08/30/2017 10:00 AM, Kay Zheng wrote:
>>>
>>> Does the old "mangling" code still exist somewhere? Could you point it
>>> out for me, to see if I can do something with it?
>>
>>
>> Looks like its still there, in Language.mangleName.
>>
>> However, I think it's best for me to do it.  I've already started looking
>> into it, and the actual amount of work looks fairly easy.  Easier than
>> explaining it ...
>
>
> Fix checked in.  Please test it.
>
> The "controlling" change is the USE_SYMBOLIC field in
> gnu/expr/Mangling.java.  This field's value depends on
> whether Android is configured.
>
> Some complications did turn up - though it was probably still
> easier than explaining what to do ...
>
>
> --
>         --Per Bothner
> per@bothner.com   http://per.bothner.com/