Fully dynamically compiling/dexing/loading Kawa on Android

Fully dynamically compiling/dexing/loading Kawa on Android

[F. Rafael Leon]

I think I may have discovered something rather significant ...

I started with the work of Helmut Eller from this post:

https://sourceware.org/ml/kawa/2012-q1/msg00040.html

It didn't work on the first try, so I added this line to AndroidManifest.xml:

    <uses-permission android:name="android.permission.INTERNET"/>

and I rebuilt the hello app and installed and ran it and then ran this
command on my computer:

rlwrap telnet 192.168.1.157 4444

and I saw this prompt and tried the repl:

Trying 192.168.1.157...
Connected to 192.168.1.157.
Escape character is '^]'.
#|kawa:1|# (+ 1 1)
2
#|kawa:2|#

I was very pleased, but then I tried compiling a scm file and loading
the jar and did not have any success.

At first. But then ...

My test file was called "plusone.scm" and looked like this:

(define-simple-class myclass ()
  ((plusone (x::int))::int allocation: 'static
   (+ 1 x)
   ))

I read a few previous posts in the kawa mail archive about loading and
dexing and it all sounded rather discouraging.

But then I was browsing the android build scripts and tools and found
this jewel:

$ANDROID_HOME/build-tools/21.1.1/lib/dx.jar

so I copied it into my project /libs directory and added it to my
app/build.gradle so it looked like this:

dependencies {
    compile files('libs/kawa.jar')
    compile files('libs/dx.jar')
}

And I built and installed and ran and telnetted and I was back to this:

#|kawa:1|#

and then things became really interesting when I started browsing the
source code for dx.jar:

https://android.googlesource.com/platform/dalvik/+/master/dx/src/com/android/dx/command/dexer/Main.java

After fixing paths and errors and warnings and avoiding calls to
System.exit(), I came up with this:

(define compiledexload
  ;; classname and filename
  (lambda (cls::String fn::String)
    (let* (
           (act::android.app.Activity *activity*)
           (bn (fn:replaceAll ".*/([^/]+).scm$" "$1"))
           (fdir ((act:getFilesDir):toString))
           (cdir ((act:getCacheDir):toString))
           (jarfn (string-append fdir "/" bn ".jar"))
           (dexfn (string-append fdir "/" bn ".dex"))
           (cfn (string-append cdir "/" bn ".dex"))
           ;; the args to dexer as a string array
           (dexsa (String[] (string-append "--output=" dexfn) jarfn))
           ;; the args to dexer as a parsed object
           (dexargs (com.android.dx.command.dexer.Main$Arguments))
           (cloader (act:getClassLoader))
           )
      ;; compile it.
      (compile-file fn jarfn)
      ;; dex it.
      (dexargs:parse dexsa)
      (com.android.dx.command.dexer.Main:run dexargs)
      ;;load it and return the loaded class.
      (if ((<java.io.File> cfn):exists)
          ((<java.io.File> cfn):delete))
      ((<dalvik.system.DexClassLoader> dexfn cdir #!null cloader):loadClass cls)
      )))

 ^^^ This thing does what you think it does. ^^^

I put it in a file called "cdl.scm" and ran this:

adb push cdl.scm /sdcard/Download

and then I ran this:

adb push plusone.scm /sdcard/Download

and then back to the repl:

#|kawa:5|# (load "/sdcard/Download/cdl.scm")
#|kawa:6|# (define loadedclass (compiledexload "myclass"
"/sdcard/Download/plusone.scm"))
#|kawa:7|# (invoke-static loadedclass 'plusone 1)
2
#|kawa:8|#

I did not use proguard. I used Dalvik runtime, not ART. I am on KitKat
4.4. I have no idea if any of this matters.

That is my story.

I was hoping someone else could try this and confirm that it actually works.

I solved of all the errors and warnings that I saw, but I always have doubts.

Thanks.

Re: Fully dynamically compiling/dexing/loading Kawa on Android

[Per Bothner]

On 12/06/2014 12:40 AM, F. Rafael Leon wrote:
> I think I may have discovered something rather significant ...
>
> But then I was browsing the android build scripts and tools and found
> this jewel:
> ....
> $ANDROID_HOME/build-tools/21.1.1/lib/dx.jar

That does sound very promising!  Cool.

The next step would be modify ArrayClassLoader so it automatically
does the .class to .dex translation.  Or whatever is the cleanest
way to tie a DexClassLoader to uses the classes of the ArrayClassLoader.

Could you find in the sources a way to convert .class to .dex
*without* actually writing a file in the file-system?  I.e. take a byte
array representing a kawa-generated-class, convert it to another byte
array representing the converted dex class, and then pass that
to ClassLoader.defineClass?

If there is a simple call to do that, then it seems like it's a matter
of adding a few lines in ArrayClassLoader.loadClass, to do the
conversion before calling defineClass.

OTOH if the conversion needs to work on a *set* of classes (like in a
jar file), then the changes are a little more difficult.  Could we convert
all the byte arrays in an ArrayClassLoader in a single operation?
Preferably if we didn't have to actually read and write files, but if
we have to do that, that's ok.
-- 
	--Per Bothner
per@bothner.com   http://per.bothner.com/