Re: i370 port - status

["Paul Edwards" <mutazilah@gmail.com>]

> So the fix below is not a fix but papering over an
> issue elswhere.

Ok, this problem nearly killed me.  It took months of
effort to try to isolate the fault, delving into unfamiliar code,
trying to get something that could be reliably reproduced.

I did finally succeed in getting an executable that reliably
crashed on the 3rd identical run.

However, when I attempted to debug at a level above, ie in
the emulator, the bug would disappear also.

My working hypothesis now is that it is a bug in the
Hercules emulator to do with paging.  Which means I am
at the mercy of the operating system as to when it pages.
That operating system is MUSIC/SP, and it doesn't come
with source code, and I'm not really familiar with it anyway,
and the author is dead (a couple of years ago).

I did succeed in finding a configuration option to stop the
huge amount of paging in MUSIC/SP.  That then allowed my
compiles to go through cleanly, so I am now able to
get GCC to reproduce itself under MUSIC/SP.

The underlying presumed paging bug in either Hercules or
MUSIC/SP I decided to just package up and leave for another
day, as neither thing was my main or even important focus.

Closer to my main focus - after getting GCC working on MUSIC/SP,
the final real ("real" = "commercially used operating system,
which people use to develop code, and which doesn't have a
free C compiler already") target that I knew of that remained
was DOS/VS (z/VSE).

In April 2010 I finally got this last environment to have a hosted
C compiler that was good enough to compile a "hello world".
The C runtime library is only bare bones, so it can't handle
#include yet, only reading from stdin, but that's enough to get
the right code for a "hello world" to be generated.  Getting GCC
on that environment found a bug in the DOS/VS operating
system, and other limitations, but these problems were
worked around.

As we speak, Michel et al over here:

http://tech.groups.yahoo.com/group/H390-DOSVS/

is battling to try to get the DOS/VS environment to process a 
PARM string (in a way compatible with z/VSE).  Previously
the parameter was kludged to read it from stdin.  The old
DOS/VS doesn't have parameters basically.  Then the rest
of the library can be dealt with (reading from private source
statement libraries is the main thing).

Meanwhile, I have released GCC 3.2.3 MVS 8.0, which
can be found here:

http://gccmvs.sourceforge.net

which includes a much nicer MVS environment (with default
DCB information) for GCC et al.  It also includes the afore
mentioned MUSIC/SP and DOS/VS ports, even if the latter
is bare-bones.

Perhaps a link to that site could be put here:

http://gcc.gnu.org/extensions.html

So in the short term, the sort of things I am likely to be involved in are:

1. MVS source code management (10 million lines of code).
2. Improve VSE/380 port.
3. GCC 3.4.6 upgrade (from 3.2.3).

Although I managed to get 3.4.6 working on MVS many months ago,
I haven't formally released that port because I was trying to get the
potentially last 3.2.3 out the door.  But that happened a couple of
days ago.  So now, as long as I can bring all 4 environments along,
I should be able to upgrade.

GCC 3.4.6 has the advantage that the Unix build tools are able to
generate all the required files for MVS etc.  Before I used to
handcraft them.  I could probably backport that to 3.2.3 but haven't
tried, as my plan was to go forward.

One problem with going foward though is that I will lose the ability
to generate the generated files from i370.md.  The reason for that
is that one of the generator programs opens lots of files by name,
and those names are not appropriate for MVS usage.  I can
probably work around that, but rather than spending the effort
doing that, I was planning on revisiting that in GCC 4, and in the
meantime, just do that part of the build on Unix instead.  If people
truly need a self-contained MVS (EBCDIC) C compiler, then 3.2.3
will always be available.

Hopefully I'll have more positive news on GCC on the EBCDIC
platfoms in the months ahead.  :-)  Thanks to everyone who
helped in getting it so far already.

BFN.  Paul.






----- Original Message ----- 
From: "Richard Guenther" <richard.guenther@gmail.com>
To: "Paul Edwards" <mutazilah@gmail.com>
Cc: <gcc@gcc.gnu.org>
Sent: Sunday, November 29, 2009 2:02 AM
Subject: Re: i370 port - music/sp - possible generic gcc problem


On Sat, Nov 28, 2009 at 4:13 PM, Paul Edwards <mutazilah@gmail.com> wrote:
> I think I have found a bug in gcc, that still exists in gcc 4.4
>
> I found the problem on 3.2.3 though.
>
> While MVS and VM have basically been working fine, when I did
> the port to MUSIC/SP I started getting strange compilation failures.
>
> Initializing the stack to NULs made the problem go away, but I
> persisted, and instead initialized the stack to x'01' to try to get
> consistent failures.
>
> Even with that, and the malloced memory initialized to consistent
> garbage, I still didn't get the desired consistency in failures.
> But it was consistent enough that I could just run it 6 times and
> see if there were any failures.
>
>
> Anyway, I tracked down the particular malloc() which gave changed
> behaviour depending on whether the malloc() did a memory initialization
> to NULs or not.

Well, GC hands out non-zeroed memory - the callers are responsible
for initializing it.  So the fix below is not a fix but papering over an
issue elswhere.

Richard.

> It's this one (showing 3.2.3):
>
> C:\devel\gcc\gcc>cvs diff -l -c15
> cvs diff: Diffing .
> Index: ggc-page.c
> ===================================================================
> RCS file: c:\cvsroot/gcc/gcc/ggc-page.c,v
> retrieving revision 1.1.1.1
> diff -c -1 -5 -r1.1.1.1 ggc-page.c
> *** ggc-page.c 15 Feb 2006 10:22:25 -0000 1.1.1.1
> --- ggc-page.c 28 Nov 2009 14:13:41 -0000
> ***************
> *** 640,670 ****
> #ifdef USING_MALLOC_PAGE_GROUPS
> else
> {
> /* Allocate a large block of memory and serve out the aligned
> pages therein. This results in much less memory wastage
> than the traditional implementation of valloc. */
>
> char *allocation, *a, *enda;
> size_t alloc_size, head_slop, tail_slop;
> int multiple_pages = (entry_size == G.pagesize);
>
> if (multiple_pages)
> alloc_size = GGC_QUIRE_SIZE * G.pagesize;
> else
> alloc_size = entry_size + G.pagesize - 1;
> ! allocation = xmalloc (alloc_size);
>
> page = (char *) (((size_t) allocation + G.pagesize - 1) &
> -G.pagesize);
> head_slop = page - allocation;
> if (multiple_pages)
> tail_slop = ((size_t) allocation + alloc_size) & (G.pagesize - 1);
> else
> tail_slop = alloc_size - entry_size - head_slop;
> enda = allocation + alloc_size - tail_slop;
>
> /* We allocated N pages, which are likely not aligned, leaving
> us with N-1 usable pages. We plan to place the page_group
> structure somewhere in the slop. */
> if (head_slop >= sizeof (page_group))
> group = (page_group *)page - 1;
> else
> --- 640,670 ----
> #ifdef USING_MALLOC_PAGE_GROUPS
> else
> {
> /* Allocate a large block of memory and serve out the aligned
> pages therein. This results in much less memory wastage
> than the traditional implementation of valloc. */
>
> char *allocation, *a, *enda;
> size_t alloc_size, head_slop, tail_slop;
> int multiple_pages = (entry_size == G.pagesize);
>
> if (multiple_pages)
> alloc_size = GGC_QUIRE_SIZE * G.pagesize;
> else
> alloc_size = entry_size + G.pagesize - 1;
> ! allocation = xcalloc (1, alloc_size);
>
> page = (char *) (((size_t) allocation + G.pagesize - 1) &
> -G.pagesize);
> head_slop = page - allocation;
> if (multiple_pages)
> tail_slop = ((size_t) allocation + alloc_size) & (G.pagesize - 1);
> else
> tail_slop = alloc_size - entry_size - head_slop;
> enda = allocation + alloc_size - tail_slop;
>
> /* We allocated N pages, which are likely not aligned, leaving
> us with N-1 usable pages. We plan to place the page_group
> structure somewhere in the slop. */
> if (head_slop >= sizeof (page_group))
> group = (page_group *)page - 1;
> else
>
>
> I suspect that it has stayed hidden for so long because most people
> probably have this mmap_anon:
>
> /* Define if mmap can get us zeroed pages using MAP_ANON(YMOUS). */
> #undef HAVE_MMAP_ANON
>
>
> #ifdef HAVE_MMAP_ANON
> # undef HAVE_MMAP_DEV_ZERO
>
> # include <sys/mman.h>
> # ifndef MAP_FAILED
> # define MAP_FAILED -1
> # endif
> # if !defined (MAP_ANONYMOUS) && defined (MAP_ANON)
> # define MAP_ANONYMOUS MAP_ANON
> # endif
> # define USING_MMAP
>
> #endif
>
> #ifndef USING_MMAP
> #define USING_MALLOC_PAGE_GROUPS
> #endif
>
>
> Seems pretty clear that zeroed pages are required.
>
> Looking at the code above and below this block, it uses xcalloc
> instead.
>
> It will take a couple of days to confirm that this is the last
> presumed bug affecting the MUSIC/SP port.
>
> In the meantime, can someone confirm that this code is wrong,
> and that xcalloc is definitely required?
>
> I had a look at the gcc4 code, and it is calling XNEWVEC which
> is also using xmalloc instead of xcalloc, so I presume it is still
> a problem today, under the right circumstances.
>
> It took about a week to track this one down. :-)
>
> One problem I have had for years, even on the MVS port, is that
> I need to use -Os to get the exact same register selection on the
> PC as the mainframe. -O0 and -O2 get slightly different register
> allocations, although all versions of the code are correct. If I'm
> lucky, this fix may make that problem go away, but as I said,
> it'll take a couple of days before the results are in, as each build
> takes about 2 hours (partly because i need to use -Os for
> consistency).
>
> Thanks. Paul.
>
>