Re: clisp crashes on startup

[Daniel Colascione <dancol@dancol.org>]

[-- Attachment #1.1: Type: text/plain, Size: 3811 bytes --]

On 7/13/12 9:30 AM, Reini Urban wrote:
> On Fri, Jul 13, 2012 at 9:26 AM, Corinna Vinschen wrote:
>> On Jul 13 07:52, Reini Urban wrote:
>>> On Fri, Jul 13, 2012 at 2:40 AM, Corinna Vinschen wrote:
>>>> On Jul 12 20:48, Daniel Colascione wrote:
>>>>> On 7/10/12 8:41 AM, Daniel Colascione wrote:
>>>>>> On 7/10/12 1:13 AM, Corinna Vinschen wrote:
>>>>>>> On Jul  9 21:59, Daniel Colascione wrote:
>>>>>>>> On 7/9/12 2:26 PM, Daniel Colascione wrote:
>>>>>>>>> [snip]
>>>>>>>>
>>>>>>>> It turns out that clisp crashes only when I've rebased DLLs into the
>>>>>>>> high portion of the 4GB WOW64 address space.
>>>>>>>
>>>>>>> Where did you rebase them to?  You know that on WOW64 and with the
>>>>>>> bigaddr flag on, the application heap is located at 0x80000000 by
>>>>>>> default, right?  Perhaps some of your DLLs just collide with that?
>>>>>>
>>>>>> I'm using a starting base address of 0xC8000000; I haven't had
>>>>>> problems with any other program.
>>>>>
>>>>> It turns out that clisp uses bit 31 of each pointer for its gc mark
>>>>> bit. No wonder the thing blows in bigaddr-aware mode. clisp _does_
>>>>
>>>> Ouch.
>>>>
>>>>> work, however, when compiled with -DWIDE. In this mode, clisp uses two
>>>>> words for each lisp value --- one for the pointer and one for the
>>>>> metadata.
>>>
>>> Hmm, I do not really want to maintain lisp32.exe and lisp64.exe
>>> variants, but maybe
>>> upstream can be persuaded to make that distinction in the clisp.exe driver.
>>> It's only for huge data and bad rebase addresses.
>>> And for huge data a self-compiled clisp makes sense to me.
>>> Serious lisp users use better lisp compilers anyway which compile
>>> to native code. (sbcl, lispworks, allegro).
>>> clisp's main strength is fast startup and fast IO.
>>>
>>>>> Also, clisp has a LINUX_NOEXEC_HEAPCODES mode that also
>>>>> works, and without bloating memory use, but that requires that no real
>>>>> virtual address be in the range [0xC0000000, 0xDFFFFFFF].
>>>>
>>>> That can't be guaranteed.  WOW64 provides the full 32 bit VM address
>>>> space.
>>>
>>> Maybe also a documentation issue for rebaseing.
>>
>> That has nothing to do with rebasing.  If you build clisp with a recent
>> gcc, it will have the "bigaddr" flag set in the file header since
>> Cygwin's gcc sets that by default.  While DLLs should be rebased from
>> 0x70000000 downwards as usual, the applications heap, as well as
>> thread stacks, as well as shared memory regions created with mmap(2)
>> or shmat(2) will be in the high memory area starting at 0x80000000.
>>
>> THe bottom line is, if the distro clisp isn't 32 bit clean, which it
>> apparently isn't using default settings, it should either be rebuilt
>> with -DWIDE, or it should have the bigaddr flag removed from the
>> file header by default (peflags -l0).
> 
> Thanks. This deserves an update now.

Indeed. One caveat is that WIDE seems to be incompatible with clisp
threading. It'd probably be a good idea to send an email upstream and
ask the clisp people whether there's anything they can do. You'll also
need the attached patch, which fixes some macro-name conflicts that
arise when compiling with WIDE.

I'd code up something like LINUX_NOEXEC_HEAPCODES myself, but I'm
nervous about making those changes: lots of places in the code (like
the bytecode interpreter?!) look at LINUX_NOEXEC_HEAPCODES and do
subtly different things if it'd set. In principle, I don't see why we
should have to reserve [0xC0000000, 0xDFFFFFFF]: the bits that force
us out of this range could come from the low bits of the pointer
instead, and the cost would just be a coarser alignment requirement.
I'm happier with requiring coarser alignment than I am with carving
out a portion of the address space.

[-- Attachment #1.2: clisp.diff --]
[-- Type: text/plain, Size: 2798 bytes --]

diff -r b689a1c1565e modules/syscalls/calls.c
--- a/modules/syscalls/calls.c	Wed Jun 27 11:08:08 2012 -0400
+++ b/modules/syscalls/calls.c	Fri Jul 13 09:42:45 2012 -0800
@@ -4550,7 +4550,7 @@
       }
     } else { /* Probably system just do not support UNICODE */
 #endif
-      gltext = GetClipboardData(CF_TEXT); /* ANSI TEXT */
+      HGLOBAL gltext = GetClipboardData(CF_TEXT); /* ANSI TEXT */
       if (gltext != NULL) {
         const char * str = (const char *)GlobalLock(gltext);
         if (str != NULL) {
diff -r b689a1c1565e src/lispbibl.d
--- a/src/lispbibl.d	Wed Jun 27 11:08:08 2012 -0400
+++ b/src/lispbibl.d	Fri Jul 13 09:42:45 2012 -0800
@@ -13704,7 +13704,7 @@
  < const chart* charptr: pointer to the characters
    (may be in string, may be on the stack) */
 #ifdef HAVE_SMALL_SSTRING
-  #define unpack_sstring_alloca_help_(string,len,offset,charptr_assignment,u) \
+  #define unpack_sstring_alloca_help_(string,len,offset,charptr_assignment,ign) \
     if (simple_nilarray_p(string)) {                                           \
       if ((len) > 0) error_nilarray_retrieve();                               \
       charptr_assignment NULL;                                                 \
@@ -13718,12 +13718,12 @@
         else if (sstring_eltype(TheSstring(string)) == Sstringtype_8Bit) \
           copy_8bit_32bit(&TheS8string(string)->data[offset],(cint32*)_unpacked_,len);\
         else                                                                   \
-          u;                                                            \
+          ign;                                                            \
       }                                                                        \
       charptr_assignment (const chart*) _unpacked_;                            \
     }
 #else
-  #define unpack_sstring_alloca_help_(string,len,offset,charptr_assignment,u) \
+  #define unpack_sstring_alloca_help_(string,len,offset,charptr_assignment,ign) \
     if (simple_nilarray_p(string)) {                                           \
       if ((len) > 0) error_nilarray_retrieve();                               \
       charptr_assignment NULL;                                                 \
@@ -13734,7 +13734,7 @@
 #define unpack_sstring_alloca(string,len,offset,charptr_assignment)     \
   unpack_sstring_alloca_help_(string,len,offset,charptr_assignment,NOTREACHED)
 /* is used by */
-%% export_def(unpack_sstring_alloca_help_(string,len,offset,charptr_assignment,u));
+%% export_def(unpack_sstring_alloca_help_(string,len,offset,charptr_assignment,ign));
 %% puts("#define unpack_sstring_alloca(s,l,o,c) unpack_sstring_alloca_help_(s,l,o,c,NOTREACHED)");
 
 /* UP: Fetches a character from a simple string.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 235 bytes --]