the right way to compile and load modules

the right way to compile and load modules

[mikel evins]

I have a main file, foo.scm. The code in that file uses a function bar, defined in bar.scm. 

The kawa docs claim that by default, all definitions in bar.scm are exported, but that does not appear to be the case. When I compile the two files with --main, I get a foo.class that throws a runtime error because bar is not defined.

Okay, so I add a module-export form to bar.scm and recompile. Now foo.class runs as expected, but there's a new problem: if I load bar.scm, kawa complains that bar is exported but never defined. 

In fact, though, bar is defined; checking in the kawa repl shows that it's defined, and, of course, the built foo.class runs as expected, including the call to bar.

I can only hope I'm doing something wrong.

What's the right way to make symbols from bar visible in foo, and also avoid generating spurious warnings about undefined symbols when loading bar.scm interactively?

Re: the right way to compile and load modules

[Charles Turner]

Hi Mikel,

Can you post your test code so that the people on this list can see
what you're trying to do?

I did just run an experiment of my own, which resulted in unexpected behaviour:

;; foo.scm
(require bar)
(display (bar))

;; bar.scm
(define (bar) "message from bar")

At this point, the BAR procedure from bar.scm is publicly visible
(exported) as you correctly said. I'm guessing your first runtime
error is because you're not
requiring (using REQUIRE) or loading (using LOAD) the bar.scm file from foo.scm.

I now compile these files like so,
prompt> java kawa.repl -C bar.scm
prompt> java jawa.repl --main -C foo.scm

I'm guessing that you're second observation about Kawa "complains that
bar is exported but never defined" is because you're compiling bar.scm
with the --main switch.

Here's where I now get confused,

prompt> java foo
#<procedure bar>

What? I was expecting as output "message from bar", as you get in the REPL:

prompt> java kawa.repl
#|kawa:1|# (define (bar) "message from bar")
#|kawa:2|# (display (bar))
message from bar

In fact, if foo.scm is modified to this,

;; foo.scm
(define (bar) "message from bar")
(display (bar))

compiled like this,
prompt> java jawa.repl --main -C foo.scm

Then I see this,
prompt> java foo
message from bar

I this behaviour is a bug.

prompt> java kawa.repl --version
Kawa 1.14 (revision 7636M)
Copyright (C) 2013 Per Bothner

Re: the right way to compile and load modules

[mikel evins]

On May 13, 2014, at 2:29 AM, Charles Turner <chturne@gmail.com> wrote:

> Hi Mikel,
> 
> Can you post your test code so that the people on this list can see
> what you're trying to do?

Your code is not significantly different from what I would provide. In fact, it closely resembles my own test case.

I don't mind supplying a boiled-down test case, but I doubt it will shed any new light.

> At this point, the BAR procedure from bar.scm is publicly visible
> (exported) as you correctly said. I'm guessing your first runtime
> error is because you're not
> requiring (using REQUIRE) or loading (using LOAD) the bar.scm file from foo.scm.

No; I see the same error if I require the file unless the required file contains a module-export form.

> I now compile these files like so,
> prompt> java kawa.repl -C bar.scm
> prompt> java jawa.repl --main -C foo.scm
> 
> I'm guessing that you're second observation about Kawa "complains that
> bar is exported but never defined" is because you're compiling bar.scm
> with the --main switch.

No; I see the warnings when loading interactively into the repl, not when compiling. 

I'd like to reach a state where I can work interactively without seeing gobs of warnings, and where I can also compile the same sources to an executable jar.

I'm partway there; I can in fact build an executable jar. I can also load the code into an interactive session; I just can't do so without kawa spewing warnings at me.

Re: the right way to compile and load modules

[Per Bothner]

On 05/13/2014 12:29 AM, Charles Turner wrote:
> I now compile these files like so,
> prompt> java kawa.repl -C bar.scm
> prompt> java jawa.repl --main -C foo.scm
> ...
> Here's where I now get confused,
>
> prompt> java foo
> #<procedure bar>
>
> What? I was expecting as output "message from bar", as you get in the REPL:

Hm.  I got the expected result.

Could you be having a classpath issue?

-- 
	--Per Bothner
per@bothner.com   http://per.bothner.com/

Re: the right way to compile and load modules

[Per Bothner]

On 05/12/2014 09:09 PM, mikel evins wrote:
> Okay, so I add a module-export form to bar.scm and recompile.
>Now foo.class runs as expected, but there's a new problem: if I load bar.scm, kawa complains that bar is exported but never defined.

A possible problem is that load is line-by-line, which is fundamentally
incompatible with modules as a unit - and thus with export.  If you use
load, leave off the exports.

Better to always use require (or r7rs import).  You can use require to
import a source file (though you can't yet use import for that).
I don't have an example - it late, I'm tired, and my initial experiment
is doing something weird. :-(

More tomorrow.
-- 
	--Per Bothner
per@bothner.com   http://per.bothner.com/

Re: the right way to compile and load modules

[mikel evins]

On May 13, 2014, at 2:54 AM, Per Bothner <per@bothner.com> wrote:

> On 05/12/2014 09:09 PM, mikel evins wrote:
>> Okay, so I add a module-export form to bar.scm and recompile.
>> Now foo.class runs as expected, but there's a new problem: if I load bar.scm, kawa complains that bar is exported but never defined.
> 
> A possible problem is that load is line-by-line, which is fundamentally
> incompatible with modules as a unit - and thus with export.  If you use
> load, leave off the exports.

If I leave off the exports I cannot compile.

I can try requiring or importing from the repl and see how that works.

> Better to always use require (or r7rs import).  You can use require to
> import a source file (though you can't yet use import for that).
> I don't have an example - it late, I'm tired, and my initial experiment
> is doing something weird. :-(

Another point: I was using parameters for certain things, but getting null pointer exceptions when referencing them in compiled code. I'm guessing that the compiled modules did not execute the toplevel defines that should have defined the parameters. I suppose that might reasonbly be an expected result, though it's a little inconvenient.

Re: the right way to compile and load modules

[mikel evins]

On May 13, 2014, at 3:01 AM, mikel evins <mevins@me.com> wrote:

>> A possible problem is that load is line-by-line, which is fundamentally
>> incompatible with modules as a unit - and thus with export.  If you use
>> load, leave off the exports.
> 
> If I leave off the exports I cannot compile.
> 
> I can try requiring or importing from the repl and see how that works

Okay, that didn't work out so well.

I've been using load because a couple of people are working on this project and the exact paths to the sources are not going to be the same across machines. It's convenient to use load because I can simply find out the path to the load file at launch time and then compute the pathnames to all the sources, then for-each over those paths to load them.

That doesn't work the same way with import or require, since they're syntax and don't evaluate their arguments.

How can I make a system load file that will import or require all of the sources of a project into an interactive session without using load?

This does not work:

(define (load-client)
  (require "/Users/mikel/Workshop/fabric/scm/src/invoke.scm"))

...yielding an error like this:

#|kawa:1|# #|kawa:2|# cannot find module invoke
	at gnu.expr.ModuleContext.findInstance(ModuleContext.java:55)
	at gnu.expr.ModuleInfo.getInstance(ModuleInfo.java:272)
	at kawa.standard.require.find(require.java:120)
	at atInteractiveLevel$7.loadClient(load-fabric.scm:62)
	at atInteractiveLevel$7.apply0(load-fabric.scm:61)
	at gnu.expr.ModuleMethod.apply0(ModuleMethod.java:186)
	at gnu.expr.ModuleMethod.apply(ModuleMethod.java:160)
	at gnu.mapping.CallContext.runUntilDone(CallContext.java:234)
	at gnu.expr.ModuleExp.evalModule2(ModuleExp.java:373)
	at gnu.expr.ModuleExp.evalModule(ModuleExp.java:210)
	at kawa.Shell.run(Shell.java:279)
	at kawa.Shell.run(Shell.java:194)
	at kawa.Shell.run(Shell.java:175)
	at kawa.repl.main(repl.java:871)


I can load the file interactively at the repl, but loading a few dozen files interactively at the repl every time I start a session is going to be a non-starter.

Re: the right way to compile and load modules

[mikel evins]

On May 13, 2014, at 3:40 AM, mikel evins <mevins@me.com> wrote:

> How can I make a system load file that will import or require all of the sources of a project into an interactive session without using load?

Turns out it's easy. If my source files are foo.scm, bar.scm, and baz,scm, I just make a a loadfile like this:

(require "foo,scm")
(require "bar,scm")
(require "baz,scm")

...and place the loadfile in the same directory as "foo.scm", "bar.scm", and "baz.scm".

Re: the right way to compile and load modules

[mikel evins]

My question was in the subject line: what's the right way to compile and load modules. By "right" I mean a way that works both when batch-compiling kawa for packaging in a jar and when interactively loading sources for repl-session development.

The answer appears to be that for interactive loading, one should use a load file that consists of a series of require forms, one for each source file, and the load file should reside in the same directory as the sources. That approach loads all of our code and silences the warnings I was previously seeing. (I still see warnings about symbols that are not visibly referenced anywhere, but that is an accurate and informative warning, so I have no complaint).

For compiling, the answer is that each source file must have a module-export form that exports each symbol that is used in some other source file. That's contrary to what the kawa docs say, but I found that in practice I saw a number of problems if I didn't use module-export forms to expose a file's definitions. In some cases kawa refused to complete the compile; in others it built everything without complaint, but crashed into a stack trace at runtime on an undefined location.

Using module-export cures these symptoms.

Since this is contrary to the kawa docs, I'm open to the possibility that I'm still doing something wrong that causes the unexpected behavior.

parameters not working?

[mikel evins]

My next issue: parameters appear to work fine in interactive sessions from loaded files, but not in required or compiled files. 

More specifically, if I define a parameter like so:

(define foo (make-parameter #f))

at the top level of a source file and then require or compile that file, I will find at run time that foo is bound to a null pointer.

I assume that this is because requiring or compiling the source file does not execute the toplevel forms (unless it's the main file).

Am I guessing correctly about the execution model of required and compiled code? If so, how are parameters meant to be used in compiled code? 

(Where's eval-when when you need it? ;-)

Re: parameters not working?

[Per Bothner]

On 05/13/2014 06:19 PM, mikel evins wrote:
> My next issue: parameters appear to work fine in interactive sessions from loaded files, but not in required or compiled files.
>
> More specifically, if I define a parameter like so:
>
> (define foo (make-parameter #f))
>
> at the top level of a source file and then require or compile that file, I will find at run time that foo is bound to a null pointer.
>
> I assume that this is because requiring or compiling the source file does not execute the toplevel forms (unless it's the main file).
>
> Am I guessing correctly about the execution model of required and compiled code? If so, how are parameters meant to be used in compiled code?

No, that is supposed to work - and seems to do so:

[bothner@toshie work1]$ cat ~/tmp/pfoo.scm
(define foo (make-parameter #f))
(format #t "foo (in pfoo): ~w~%" (foo))
[bothner@toshie work1]$ cat ~/tmp/pbar.scm
(require pfoo)
(format #t "foo (in pbar): ~w~%" (foo))
[bothner@toshie work1]$ bin/kawa  -C ~/tmp/pfoo.scm
(compiling /home/bothner/tmp/pfoo.scm to pfoo)
[bothner@toshie work1]$ bin/kawa  -C ~/tmp/pbar.scm
(compiling /home/bothner/tmp/pbar.scm to pbar)
[bothner@toshie work1]$ bin/kawa pbar
foo (in pfoo): #f
foo (in pbar): #f
[bothner@toshie work1]$ bin/kawa --main -C ~/tmp/pbar.scm
(compiling /home/bothner/tmp/pbar.scm to pbar)
[bothner@toshie work1]$ java pbar
foo (in pfoo): #f
foo (in pbar): #f

Could you write a simple test-program that shows the problem?
Note it should be complete and self-contained, and we need to
see the command-line(s) you use to compile and run it.

(Note I'm testing with the latest Kawa from Subversion.)
-- 
	--Per Bothner
per@bothner.com   http://per.bothner.com/

Re: parameters not working?

[mikel evins]

On May 13, 2014, at 8:31 PM, Per Bothner <per@bothner.com> wrote:

> Could you write a simple test-program that shows the problem?
> Note it should be complete and self-contained, and we need to
> see the command-line(s) you use to compile and run it.

Sure; will do.

Give me a few minutes. I just pushed some code that *does* work as intended and need to report to the team. :-)

Re: parameters not working?

[mikel evins]

On May 13, 2014, at 8:31 PM, Per Bothner <per@bothner.com> wrote:

> Could you write a simple test-program that shows the problem?
> Note it should be complete and self-contained, and we need to
> see the command-line(s) you use to compile and run it.

The simple test case I contrived worked, which suggests I need to look more closely at the context there the parameters are defined in the bigger files where it doesn't work.

Re: parameters not working?

[mikel evins]

On May 13, 2014, at 9:12 PM, mikel evins <mevins@me.com> wrote:

> 
> On May 13, 2014, at 8:31 PM, Per Bothner <per@bothner.com> wrote:
> 
>> Could you write a simple test-program that shows the problem?
>> Note it should be complete and self-contained, and we need to
>> see the command-line(s) you use to compile and run it.
> 
> The simple test case I contrived worked, which suggests I need to look more closely at the context there the parameters are defined in the bigger files where it doesn't work.

So presumably some of the code I'm requiring for the app is somehow breaking parameters. I'll have to search more, but clearly the onus is on me; thanks for your help so far.

A simple test in which I define a parameter in one nearly-empty source file and reference it from another works fine.

The context where it's not working is that I'm defining a couple of parameters at the top of a source file after requiring 20 other modules. In that case, a reference to either of the parameters I defined yields #!null.

I'll have to do a binary search of the requires and see where parameters go south.

Thanks again for the help.

--me

Re: parameters not working?

[mikel evins]

On May 13, 2014, at 8:31 PM, Per Bothner <per@bothner.com> wrote:

> 
> 
> On 05/13/2014 06:19 PM, mikel evins wrote:
>> My next issue: parameters appear to work fine in interactive sessions from loaded files, but not in required or compiled files.

....

> No, that is supposed to work - and seems to do so:

Yep; it works. I discovered my mistake. The failing parameter definition was in a file named client.scm that had a define-simple-class for a class named <client>. Turns out that's a bad idea. Changing the name of the class fixed the problem.

Re: parameters not working?

[Per Bothner]

On 05/14/2014 02:32 AM, mikel evins wrote:

> Yep; it works. I discovered my mistake. The failing parameter definition was in a file named
> client.scm that had a define-simple-class for a class named <client>. Turns out that's a bad idea.
> Changing the name of the class fixed the problem.

That's best.  If you do want to have the "module class" and a class defined by define-simple-class
have the same name (i.e. be the same class) then I recommend using:
   (module-static 'init-run)
so the module initialization gets done in the class initializer.

http://www.gnu.org/software/kawa/Module-classes.html
-- 
	--Per Bothner
per@bothner.com   http://per.bothner.com/

Re: the right way to compile and load modules

[Per Bothner]

On 05/13/2014 06:12 PM, mikel evins wrote:
> My question was in the subject line: what's the right way to compile and load modules. By "right" I mean a way that works both when batch-compiling kawa for packaging in a jar and when interactively loading sources for repl-session development.
>
> The answer appears to be that for interactive loading, one should use a load file that consists of a series of require forms, one for each source file, and the load file should reside in the same directory as the sources. That approach loads all of our code and silences the warnings I was previously seeing. (I still see warnings about symbols that are not visibly referenced anywhere, but that is an accurate and informative warning, so I have no complaint).

Mixing load and require can be a bit fragile, but if it works for you that's ok.
In general I say it's best to avoid using either load and eval.
(People tend to overuse them.)

I just checked in a fix to this problem:

#|kawa:1|# (require "foo.scm")
/dev/stdin:1:1: not found: /dev/foo.scm (No such file or directory)

This mysterious problem was because the filename of the default input port
is "/dev/stdin" and require resolves the specified filename against the
current filename.  Ooops.  Using require with a filename should work now.

If you need to require multiple source files in the repl, you can use the -e option:

$ kawa -e '(require "foo.scm")' -e '(require "bar.scm")' --

You can also use ~/.kawarc.scm.

Even better is you put all the require forms in a file file, and require that.
However, in that case you need to add module-exports statements to re-export
the definitions you need.

> For compiling, the answer is that each source file must have a module-export form that exports each symbol that is used in some other source file. That's contrary to what the kawa docs say, but I found that in practice I saw a number of problems if I didn't use module-export forms to expose a file's definitions. In some cases kawa refused to complete the compile; in others it built everything without complaint, but crashed into a stack trace at runtime on an undefined location.
>
> Using module-export cures these symptoms.
>
> Since this is contrary to the kawa docs, I'm open to the possibility that I'm still doing something wrong that causes the unexpected behavior.

I don't know.  Sounds mysterious.  If you have *no* module-exports it's supposed to be equivalent to
exporting everything that isn't define-private etc.  There are some exceptions - for examples names that
are imported using require aren'r re-exported by default.Again, perhaps you can show us a simple test-case?

-- 
	--Per Bothner
per@bothner.com   http://per.bothner.com/

Re: the right way to compile and load modules

[mikel evins]

On May 13, 2014, at 8:43 PM, Per Bothner <per@bothner.com> wrote:

> On 05/13/2014 06:12 PM, mikel evins wrote:
>> My question was in the subject line: what's the right way to compile and load modules. By "right" I mean a way that works both when batch-compiling kawa for packaging in a jar and when interactively loading sources for repl-session development.
>> 
>> The answer appears to be that for interactive loading, one should use a load file that consists of a series of require forms, one for each source file, and the load file should reside in the same directory as the sources. That approach loads all of our code and silences the warnings I was previously seeing. (I still see warnings about symbols that are not visibly referenced anywhere, but that is an accurate and informative warning, so I have no complaint).
> 
> Mixing load and require can be a bit fragile, but if it works for you that's ok.

I'd rather not do anything that's not recommended, and I won't unless there's no other way to make things work.

> In general I say it's best to avoid using either load and eval.
> (People tend to overuse them.)
> 
> I just checked in a fix to this problem:
> 
> #|kawa:1|# (require "foo.scm")
> /dev/stdin:1:1: not found: /dev/foo.scm (No such file or directory)
> 
> This mysterious problem was because the filename of the default input port
> is "/dev/stdin" and require resolves the specified filename against the
> current filename.  Ooops.  Using require with a filename should work now.

Yeo, that squares with the behavior I saw.

> If you need to require multiple source files in the repl, you can use the -e option:
> 
> $ kawa -e '(require "foo.scm")' -e '(require "bar.scm")' --

That seems like the way to go for my project. I use a project-specific shell script to launch kawa anyway.

> You can also use ~/.kawarc.scm.

Not as good. I don't necessarily want to limit use of kawa to this one project, so I don't necessarily want to require its sources every time I start kawa.

> Even better is you put all the require forms in a file file, and require that.
> However, in that case you need to add module-exports statements to re-export
> the definitions you need.

That also seems fine to me. Thanks for the advice.

>> For compiling, the answer is that each source file must have a module-export form that exports each symbol that is used in some other source file. That's contrary to what the kawa docs say, but I found that in practice I saw a number of problems if I didn't use module-export forms to expose a file's definitions. In some cases kawa refused to complete the compile; in others it built everything without complaint, but crashed into a stack trace at runtime on an undefined location.
>> 
>> Using module-export cures these symptoms.
>> 
>> Since this is contrary to the kawa docs, I'm open to the possibility that I'm still doing something wrong that causes the unexpected behavior.
> 
> I don't know.  Sounds mysterious.  If you have *no* module-exports it's supposed to be equivalent to
> exporting everything that isn't define-private etc.  There are some exceptions - for examples names that
> are imported using require aren'r re-exported by default.Again, perhaps you can show us a simple test-case?

Sure. In a few minutes.

Re: the right way to compile and load modules

[mikel evins]

On May 13, 2014, at 8:43 PM, Per Bothner <per@bothner.com> wrote:

>> 
>> Using module-export cures these symptoms.
>> 
>> Since this is contrary to the kawa docs, I'm open to the possibility that I'm still doing something wrong that causes the unexpected behavior.
> 
> I don't know.  Sounds mysterious.  If you have *no* module-exports it's supposed to be equivalent to
> exporting everything that isn't define-private etc.  There are some exceptions - for examples names that
> are imported using require aren'r re-exported by default.Again, perhaps you can show us a simple test-case?

Two source files:

src/fact.scm:

(define (fact n)
  (if (<= n 1)
      1
      (* n (fact (- n 1)))))



src/factest.scm:

(require "fact.scm")


(newline)
(display (fact 6))
(newline)



Ant build file in the directory that contains the src directory:

...

  <path id="kawa.class.path">
    <pathelement location="./lib/kawa-1.14.jar"/>
    <pathelement path="."/>
  </path>

...


  <!--compile fact -->
  <target name="fact">
    <java taskname="kawa"
          classname="kawa.repl"
          failOnError="true"
          dir="./src"
          fork="true">
      <classpath refid="kawa.class.path"/>
      <arg line="-C 'fact.scm'"/>
    </java>
  </target>

  <!--compile factest -->
  <target name="factest" depends="fact">
    <java taskname="kawa"
          classname="kawa.repl"
          failOnError="true"
          dir="./src"
          fork="true">
      <classpath refid="kawa.class.path"/>
      <arg line="--main -C 'factest.scm'"/>
    </java>
  </target>


Build results:

$ ant factest
Buildfile: /Users/mikel/Workshop/fabric/build.xml

fact:
     [kawa] (compiling fact.scm to fact)

factest:
     [kawa] (compiling factest.scm to factest)
     [kawa] /Users/mikel/Workshop/fabric/src/fact.scm:3:9: warning - no use of fact
     [kawa] factest.scm:3:1: warning - no declaration seen for fact


Run test:

$ java -cp src:lib/kawa-1.14.jar factest
factest.scm:3:1: unbound location fact
	at gnu.mapping.SharedLocation.get(SharedLocation.java:22)
	at gnu.mapping.ThreadLocation.get(ThreadLocation.java:105)
	at factest.run(factest.scm:3)
	at gnu.expr.ModuleBody.runAsMain(ModuleBody.java:152)
	at factest.main(factest.scm)

BUILD SUCCESSFUL


Add a module-export form to fact.scm:

Build results:

$ ant factest
Buildfile: /Users/mikel/Workshop/fabric/build.xml

fact:
     [kawa] (compiling fact.scm to fact)

factest:
     [kawa] (compiling factest.scm to factest)

BUILD SUCCESSFUL


Run test:

$ java -cp src:lib/kawa-1.14.jar factest

720

Re: the right way to compile and load modules

[Per Bothner]

On 05/13/2014 07:05 PM, mikel evins wrote:
> factest:
>       [kawa] (compiling factest.scm to factest)
>       [kawa] /Users/mikel/Workshop/fabric/src/fact.scm:3:9: warning - no use of fact
>       [kawa] factest.scm:3:1: warning - no declaration seen for fact

Not quite sure what is going on, but:

Compiling fact.scm creates fact.class.
Compiling factest.scm *recompiles* fact.scm (reasonable because you
only specify the source file name), but it does so *differently* than
the first time (i.e. without the fact function being exported), and
then overwrites fact.class with the bad version.

Still need to look into what is going on, but I suggest replacing the
(require "fact.scm")
by:
(require fact)

However, this may not work when importing into the REPL.

What you're doing is somewhat unusual (and thus poorly tested):
Compiling ahead-of-time, and then in a different module requiring the
source file rather than the compiled module.

Fairly soon I want to combine the functions of import and
require, and both use some kind of search path.  However,
that is orthogonal to the problem that fact is sometimes
not exported - which is a bug.
-- 
	--Per Bothner
per@bothner.com   http://per.bothner.com/

Re: the right way to compile and load modules

[mikel evins]

On May 13, 2014, at 10:52 PM, Per Bothner <per@bothner.com> wrote:

> On 05/13/2014 07:05 PM, mikel evins wrote:
>> factest:
>>      [kawa] (compiling factest.scm to factest)
>>      [kawa] /Users/mikel/Workshop/fabric/src/fact.scm:3:9: warning - no use of fact
>>      [kawa] factest.scm:3:1: warning - no declaration seen for fact
> 
> Not quite sure what is going on, but:
> 
> Compiling fact.scm creates fact.class.
> Compiling factest.scm *recompiles* fact.scm (reasonable because you
> only specify the source file name), but it does so *differently* than
> the first time (i.e. without the fact function being exported), and
> then overwrites fact.class with the bad version.
> 
> Still need to look into what is going on, but I suggest replacing the
> (require "fact.scm")
> by:
> (require fact)

Here's why I'm not doing that:

$ ant factest
Buildfile: /Users/mikel/Workshop/fabric/build.xml

fact:
     [kawa] (compiling fact.scm to fact)

factest:
     [kawa] (compiling factest.scm to factest)
     [kawa] factest.scm:3:1: invalid specifier for 'require'

BUILD FAILED


> However, this may not work when importing into the REPL.

As you can see, it doesn't work for me when compiling, either.

> What you're doing is somewhat unusual (and thus poorly tested):
> Compiling ahead-of-time, and then in a different module requiring the
> source file rather than the compiled module.

I'm not married to what I'm currently doing. I'm happy to do anything else that 

1. actually works

2. enables me to build a jar containing the compiled code of the project

3. enables me to load (or require or import or inject or assimilate or devour or adjoin or intertwingle or whatever you want to call it) all of the project sources into an interactive session for further development and testing.

Just tell me what to do and I'm there.

> Fairly soon I want to combine the functions of import and
> require, and both use some kind of search path.  However,
> that is orthogonal to the problem that fact is sometimes
> not exported - which is a bug.
> -- 
> 	--Per Bothner
> per@bothner.com   http://per.bothner.com/

Re: the right way to compile and load modules

[Per Bothner]

On 05/13/2014 08:59 PM, mikel evins wrote:
>
> On May 13, 2014, at 10:52 PM, Per Bothner <per@bothner.com> wrote:
>> Still need to look into what is going on, but I suggest replacing the
>> (require "fact.scm")
>> by:
>> (require fact)
>
> Here's why I'm not doing that:
>
> $ ant factest
> Buildfile: /Users/mikel/Workshop/fabric/build.xml
>
> fact:
>       [kawa] (compiling fact.scm to fact)
>
> factest:
>       [kawa] (compiling factest.scm to factest)
>       [kawa] factest.scm:3:1: invalid specifier for 'require'
>
> BUILD FAILED

This means it couldn't find the class 'test'.  You means to tweak the
classpath (when compiling facttest) so it finds fact.class.

This error message is rather unhelpful; I checked in an improvement.
-- 
	--Per Bothner
per@bothner.com   http://per.bothner.com/

Re: the right way to compile and load modules

[mikel evins]

On May 14, 2014, at 1:29 AM, Per Bothner <per@bothner.com> wrote:

> On 05/13/2014 08:59 PM, mikel evins wrote:
>> 
>> On May 13, 2014, at 10:52 PM, Per Bothner <per@bothner.com> wrote:
>>> Still need to look into what is going on, but I suggest replacing the
>>> (require "fact.scm")
>>> by:
>>> (require fact)
>> 
>> Here's why I'm not doing that:
>> 
>> $ ant factest
>> Buildfile: /Users/mikel/Workshop/fabric/build.xml
>> 
>> fact:
>>      [kawa] (compiling fact.scm to fact)
>> 
>> factest:
>>      [kawa] (compiling factest.scm to factest)
>>      [kawa] factest.scm:3:1: invalid specifier for 'require'
>> 
>> BUILD FAILED
> 
> This means it couldn't find the class 'test'.  You means to tweak the
> classpath (when compiling facttest) so it finds fact.class.
> 
> This error message is rather unhelpful; I checked in an improvement.

Thanks; that solved it for the test program. The real deal still doesn't compile using class names instead of source-file names, but I'll assume that's because of something I've overlooked and can find with further examination.

Have I mentioned lately how great Kawa is? I've used it before, though it's been some years. I picked it again for this project after a short period of trying it alongside ABCL and Clojure. I like Common Lisp just fine, and I like Clojure too, but Kawa was just easier to work with and faster. I'm particularly impressed by its quick launch times compared to other JVM tools.

Kawa experience [was: the right way to compile and load modules]

[Per Bothner]

On 05/13/2014 11:51 PM, mikel evins wrote:
> Have I mentioned lately how great Kawa is?

No - but we're always happy to hear it!

Please tell your friends and co-workers and on blogs.
Kawa's biggest problem is lack of word-of-mouth and mind-share.

> I've used it before  though it's been some years.
>I picked it again for this project  after a short period of trying
>it alongside ABCL and Clojure. I like  Common Lisp just fine, and
>I like Clojure too, but Kawa was just easier
> to work with and faster. I'm particularly impressed by its quick launch
>  times compared to other JVM tools.

Interesting.  I'm aware that Kawa generally runs quite a bit faster than
other "scripting" languages (especially with suitable type specifiers),
but I haven't really thought about it having faster startup. Can you share
a bit about which tools are (or seem) fast or slow in your experience?

Also, what makes Kawa "easier to work with"?

I'll be writing an "intro to Kawa" article for LWN soon,
so it is useful to know what to emphasize.
-- 
	--Per Bothner
per@bothner.com   http://per.bothner.com/

Re: Kawa experience [was: the right way to compile and load modules]

[Charles Turner]

On 14 May 2014 19:59, Per Bothner <per@bothner.com> wrote:
> I'll be writing an "intro to Kawa" article for LWN soon,
> so it is useful to know what to emphasize.

I'm very pleased to hear you'll be writing an article. I look forward
to reading it!

I gave a 30 minute talk recently about Kawa, its history, its design
and demo'd some code. The majority of people were amazed they had
never heard of this project before and seemed enthusiastic to at least
play with it. A minority (~ 5 out of ~ 20 people) were of the "why are
people still messing around with this Lisp bullshit?" persuasion; the
ones you lose as soon as they see the parens. I didn't have slides or
anything, it was informal get together. My limited experience of this
sort of thing however is that people want to see what "real stuff"
looks like in the language, they're not as interested in the
implementation details and history of Lisp. They're dying to know what
it buys them. I think the art of expository pieces is to provide
instant gratification by example. Once people are interested, they'll
look up the gory details for themselves. I'm very excited & interested
to hear what people are doing with Kawa beyond just messing about, I
think case studies from these groups are immensely valuable.

Just my opinion of course, not an experienced speaker/teacher by any
stretch of the imagination!

Kind regards,
Charles.

Re: Kawa experience [was: the right way to compile and load modules]

[mikel evins]

On May 14, 2014, at 1:59 PM, Per Bothner <per@bothner.com> wrote:

> On 05/13/2014 11:51 PM, mikel evins wrote:
>> Have I mentioned lately how great Kawa is?
> 
> No - but we're always happy to hear it!
> 
> Please tell your friends and co-workers and on blogs.
> Kawa's biggest problem is lack of word-of-mouth and mind-share.
> 
>> I've used it before  though it's been some years.
>> I picked it again for this project  after a short period of trying
>> it alongside ABCL and Clojure. I like  Common Lisp just fine, and
>> I like Clojure too, but Kawa was just easier
>> to work with and faster. I'm particularly impressed by its quick launch
>> times compared to other JVM tools.
> 
> Interesting.  I'm aware that Kawa generally runs quite a bit faster than
> other "scripting" languages (especially with suitable type specifiers),
> but I haven't really thought about it having faster startup. Can you share
> a bit about which tools are (or seem) fast or slow in your experience?

Cold start to kawa prompt is something like a quarter of a second on my machine. For a Clojure prompt it's eight to twelve times longer. For ABCL it's around twice the Clojure startup time.

Those startup times carry over to my built app. Cold start to first window on the screen is under a second for Kawa, and longer in rough proportion to the above numbers for Clojure and ABCL.

Kawa's launch is so fast that it's a kind of camouflage. Everyone knows that Java apps start up slowly, so our app, built with Kawa, appears superficially to be a non-Java app.

That doesn't mean Kawa is faster overall than Clojure and ABCL. I didn't do benchmarks and I don't plan to. All I can say is that it seems noticeably faster executing many tasks in our app (an immersive 3D multiplayer networked game). 

> Also, what makes Kawa "easier to work with"?
> 
> I'll be writing an "intro to Kawa" article for LWN soon,
> so it is useful to know what to emphasize.

It's simpler and easier to get started using Kawa. Kawa is one jar plus whatever JVM-related tools you want to use. You just run it and write Scheme code. You can add Jave-related features incrementally as you reach the point where you need them.

Using ABCL with Java libraries most likely means organizing your project as a regular Java project that happens to include ABCL as a JSR-223 scripting component. You don't *have* to do that, but if you're going to be using Java libraries, that's probably the easiest way to do it. If those Java libraries want you to write Java classes in order to integrate them, as libraries often do, then you're pretty much going to have to do things that way, because ABCL doesn't know how to implement novel Java classes, so you're going to have to drop into Java to do it.

That's not my preferred solution. I want to use some Java libraries in my Lisp program. I don't want to use Lisp scripting in a Java program.

Clojure isn't just a jar, it's a whole tools ecosystem. It's *possible* to use the main clojure jar alone, but that's masochism. You really want to start with Leiningen and its config files and all the various tools and libraries it will download and set up, and its idea of what a Clojure project looks like, because that's the de facto standard Clojure project. The tools are good and helpful, and they do a lot for you, especially if you're working on enterprise software, which is where Clojure's bread and butter is. But it's a lot of stuff to learn about, download, and configure before you ever reach the Clojure prompt.

The hole in ABCL's java interop means it's not as easy to work with Java libraries. Its interop is fine, if slightly more verbose than Kawa's, but besides the lack of class definition, it's also complicated a little in that there are actually two different Java interop layers. One is more verbose, but more mature and better integrated. The other is more terse and more convenient, but also less mature and less well-integrated, so that you end up deciding on a case-by-case basis whether to use one interop layer or the other. That's not a deal killer; it's just a little gratuitous complexity, especially since the interop between the two different interops is not perfectly transparent.

Clojure's Java interop is comprehensive and mature and it works very very well. It is intrinsically a little more complicated than Kawa's simply because Clojure is a primarily functional language that disdains side effects without completely forbidding them. Java, of course, is not like that, so you have to take some extra care whenever the two languages talk to each other that one isn't violating invariants assumed by the other. 

Also, Clojure's been evolving a novel type system lately, and there are pieces and scaffolding of that type system lying around in the language. Since those pieces and scaffolding more often than not involve some sort of interaction with the JVM and Java types, they constitute something of a distraction. You have to learn which features are actually meant to be used for Java interop and which are artifacts of the evolving type system. You have to learn where you really want to use the new types and where you really want to use Java interop.

If I sound like I dislike Clojure or ABCL, that's unintentional. In fact I like them both and would use either of them with pleasure in any case where they seem to be the best option. Clojure is broadly successful and a nice piece of work. ABCL has reached a remarkable level of compliance with the ANSI Common Lisp standard, and steadily keeps getting better.

But for my uses, Kawa is clearly the best choice. It's smaller, faster to launch, easier to integrate with our app, and it imposes fewer arbitrary requirements on us in using it.

What's not to like?

Re: the right way to compile and load modules

[mikel evins]

On May 14, 2014, at 1:29 AM, Per Bothner <per@bothner.com> wrote:

> On 05/13/2014 08:59 PM, mikel evins wrote:
>> 
>> On May 13, 2014, at 10:52 PM, Per Bothner <per@bothner.com> wrote:
>>> Still need to look into what is going on, but I suggest replacing the
>>> (require "fact.scm")
>>> by:
>>> (require fact)
>> 
>> Here's why I'm not doing that:
>> 
>> $ ant factest
>> Buildfile: /Users/mikel/Workshop/fabric/build.xml
>> 
>> fact:
>>      [kawa] (compiling fact.scm to fact)
>> 
>> factest:
>>      [kawa] (compiling factest.scm to factest)
>>      [kawa] factest.scm:3:1: invalid specifier for 'require'
>> 
>> BUILD FAILED
> 
> This means it couldn't find the class 'test'.  You means to tweak the
> classpath (when compiling facttest) so it finds fact.class.

Well, it works for batch-compiling, but you were right: it screws the repl load.

Back to the drawing board.

Re: the right way to compile and load modules

[Per Bothner]

On 05/13/2014 08:52 PM, Per Bothner wrote:
> On 05/13/2014 07:05 PM, mikel evins wrote:
>> factest:
>>       [kawa] (compiling factest.scm to factest)
>>       [kawa] /Users/mikel/Workshop/fabric/src/fact.scm:3:9: warning - no use of fact
>>       [kawa] factest.scm:3:1: warning - no declaration seen for fact
>
> Not quite sure what is going on, but:

The problem is that the --main option "bleeds through" to the
require'd class fact.  And the --main option changes the default to
"export nothing".  (This can be a performance win, since it allows
some optimizations.)

A work-around: leave off the --main.  Recent versions of Kawa
treat:
   $ java kawa.repl -C foo.scm
   $ java kawa.repl foo
as equivalent to:
   $ java kawa.repl --main -C foo.scm
   $ java foo

(If you do specify --main, you can still invoke java kawa.repl foo.)
-- 
	--Per Bothner
per@bothner.com   http://per.bothner.com/

Re: the right way to compile and load modules

[mikel evins]

On May 14, 2014, at 2:32 AM, Per Bothner <per@bothner.com> wrote:

> 
> 
> On 05/13/2014 08:52 PM, Per Bothner wrote:
>> On 05/13/2014 07:05 PM, mikel evins wrote:
>>> factest:
>>>      [kawa] (compiling factest.scm to factest)
>>>      [kawa] /Users/mikel/Workshop/fabric/src/fact.scm:3:9: warning - no use of fact
>>>      [kawa] factest.scm:3:1: warning - no declaration seen for fact
>> 
>> Not quite sure what is going on, but:
> 
> The problem is that the --main option "bleeds through" to the
> require'd class fact.  And the --main option changes the default to
> "export nothing".  (This can be a performance win, since it allows
> some optimizations.)
> 
> A work-around: leave off the --main.  Recent versions of Kawa
> treat:
>  $ java kawa.repl -C foo.scm
>  $ java kawa.repl foo
> as equivalent to:
>  $ java kawa.repl --main -C foo.scm
>  $ java foo
> 
> (If you do specify --main, you can still invoke java kawa.repl foo.)

Aha! That's a fine solution. Thanks again.

Re: the right way to compile and load modules

[Per Bothner]

On 05/14/2014 12:32 AM, Per Bothner wrote:
> The problem is that the --main option "bleeds through" to the
> require'd class fact.

I checked in a fix for this, so --main only effects modules
on the command line.

I also changed it so required files aren't implicitly written out.
-- 
	--Per Bothner
per@bothner.com   http://per.bothner.com/