[Bug libc/214] New: sbrk() doesn't detect brk() failures. Malloc doesn't handle sbrk() failures

[Bug libc/214] New: sbrk() doesn't detect brk() failures. Malloc doesn't handle sbrk() failures

[dlstevens at us dot ibm dot com]

On Linux systems, the "brk" system call returns 0 for success, but returns the
old end-of-data-segment when the RLIMIT_DATA soft limit is exceeded
(kernel 2.6.x).

"sbrk" in glibc has the following code:

       if (__brk (oldbrk + increment) < 0)
             return (void *) -1;
       return oldbrk;

On Linux systems, sbrk() returns the same value whether or not the allocation
exceeded the soft limit.

For Linux systems, this test should be:

        if (__brk (oldbrk + increment) != 0)
              return (void *) -1;
        return oldbrk;

I retested with a modified Linux kernel where brk() returns -1 for the failure
case and this led to another bug in malloc() [not yet filed or investigated].
Apparently, malloc() does not properly check for sbrk() failures in all cases,
so the retest resulted in a segmentation violation within the malloc code,
instead of malloc() returning NULL as expected.


Reproduce (sbrk issue) by:
     get current VmData size by reading /proc/self/status.
     use setrlimit() to set RLIMIT_DATA/rlim_cur to that value
     do an sbrk() of more than a page

Reproduce (malloc issue) by:
     same, but do enough malloc()'s to exhaust MMAPs; when it
     falls back to sbrk(), it gets a segmentation violation

-- 
           Summary: sbrk() doesn't detect brk() failures. Malloc doesn't
                    handle sbrk() failures
           Product: glibc
           Version: 2.3.2
            Status: NEW
          Severity: normal
          Priority: P2
         Component: libc
        AssignedTo: gotom at debian dot or dot jp
        ReportedBy: dlstevens at us dot ibm dot com
                CC: glibc-bugs at sources dot redhat dot com


http://sources.redhat.com/bugzilla/show_bug.cgi?id=214

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug libc/214] sbrk() doesn't detect brk() failures. Malloc doesn't handle sbrk() failures

[dlstevens at us dot ibm dot com]

------- Additional Comments From dlstevens at us dot ibm dot com  2004-06-18 17:35 -------
I looked a little more into this and I found code for Linux in glibc that is
supposed to remap brk to "-1" on error cases, so I'm no longer sure what's
going on here. What was failing in my test is that sbrk() didn't return "-1"
when the soft RLIMIT_DATA limit should have been exceeded, and strace() showed
brk() returning the break point, not a negative value, when this happened.

However, if the linux-specific __brk() code is on my system, that should've
given a "-1" to sbrk() as it expects and sbrk() itself should've returned "-1",
which isn't what I saw. I also had different results when using the static
library versus the dynamic library.

I'll put together a more comprehensive test program and run it on a fresh
install to see if the problem goes away, or if I can isolate what exactly is
going on. I was never able to trigger a malloc "NULL" return by manipulating
the soft limit-- only resulted in a SEGV within malloc when I got a failre at
all, so something appears not to be happy.

-- 


http://sources.redhat.com/bugzilla/show_bug.cgi?id=214

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug libc/214] sbrk() doesn't detect brk() failures. Malloc doesn't handle sbrk() failures

[drepper at redhat dot com]

------- Additional Comments From drepper at redhat dot com  2004-09-26 21:29 -------
if (__brk (oldbrk + increment) < 0)

is correct for the i386 code.  I'll need to examine what the other architectures
do but in general this change is not correct.

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
         AssignedTo|gotom at debian dot or dot  |drepper at redhat dot com
                   |jp                          |
             Status|NEW                         |ASSIGNED


http://sources.redhat.com/bugzilla/show_bug.cgi?id=214

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug libc/214] sbrk() doesn't detect brk() failures. Malloc doesn't handle sbrk() failures

[rsa at us dot ibm dot com]

------- Additional Comments From rsa at us dot ibm dot com  2006-03-03 22:01 -------
Created an attachment (id=901)
 --> (http://sourceware.org/bugzilla/attachment.cgi?id=901&action=view)
testcase to demonstrate setrlimit() and sbrk()

This source file provides a testcase.  It basically sets the rlimit and then
incrementally increases the data segment size with sbrk() until an error is
returned.  This happens to be when the data segment exceeds the hard limit of 1
page.

Compile with:

gcc -o glibc214 glibc214.c -DSBRK_EARLYSBRK

setrlimit() must be preceeded by an initial call to sbrk() or subsequent sbrk()
calls will fail.  That is what the flag demonstrates; take it out and the test
will fail very quickly.

I tested this on powerpc32, powerpc64, and an i486 and didn't experience the
problem that the bug addresses.

		 gcc		 glibc	   kernel
powerpc32    4.1.0 20060130	2.3.90	  2.6.5-7.244-pseries64
powerpc64    4.1.0 20060130	2.3.90	  2.6.5-7.244-pseries64
i486	     4.0.3 20051201	2.3.5	  2.6.12-1-k7

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=214

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug libc/214] sbrk() doesn't detect brk() failures. Malloc doesn't handle sbrk() failures

[rsa at us dot ibm dot com]

------- Additional Comments From rsa at us dot ibm dot com  2006-03-06 21:37 -------
I think I misunderstood the problem a it with my previous entry.

On Powerpc32 we have the following assembler code in brk.S

...

        DO_CALL(SYS_ify(brk)) /* new brk in r3 */
        lwz     r6,8(r1) /* saved old brk */

...

        cmplw   r6,r3 /* old brk vs. new brk */
        addi    r1,r1,16
        mtlr    r0
        li      r3,0 /* zero is the return value for less-than-or-equal-to */
        blelr+
        li      r3,ENOMEM /* else ENOMEM */

Where r6 holds 'oldbrk' and r3 holds the 'curbrk'.  The blerl checks to see if
r6 is less than or equal to r3.  If so it returns 0, meaning '0' is returned in
the normal success case as well as the case where the brk is not moved.

Changing the code to:

        if (__brk (oldbrk + increment) != 0)
              return (void *) -1;
        return oldbrk;

Would certainly be wrong on powerpc32 and powerpc64.

I'm beginning to think that the problem is related to the soft-limit not being
set properly.

GLIBC interpets sbrk() as __brk(curbrk + increment) and sends a brk syscall to
the kernel.  The Linux kernel, in mm/mmap.c:sys_brk subtracts the start of the
process memory area from the proposed brk() is supposed to return the oldbrk if
the rlimit would be exceeded.  It should not increment curbrk in this case.

We can see from my example that calling sbrk() after exceeding the soft-limit,
that we set with setrlimit(), doesn't prevent the kernel from issuing the brk().

Only when the brk() exceeds a page boundary does the kernel complain.

I will investigate setrlimit() in GLIBC and in the kernel.

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=214

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug libc/214] sbrk() doesn't detect brk() failures. Malloc doesn't handle sbrk() failures

[rsa at us dot ibm dot com]

------- Additional Comments From rsa at us dot ibm dot com  2006-03-08 19:21 -------
Found the 'issue'.

In the Linux kernel in the sys_brk syscall handler:

mm/mmap.c:sys_brk

...

        if (brk < mm->end_code)
                goto out;
        newbrk = PAGE_ALIGN(brk);
        oldbrk = PAGE_ALIGN(mm->brk);
        if (oldbrk == newbrk)
                goto set_brk;

...

set_brk:
        mm->brk = brk;
out:
        retval = mm->brk;
        up_write(&mm->mmap_sem);
        return retval;

The Linux kernel operates in pages.  It checks the page boundary of the current
brk and the new brk and if they're the same it simply increment's the brk the
requested amount and returns.  It never checks the soft-limit if the brk request
falls within the same page of the current brk.


-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=214

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug libc/214] sbrk() doesn't detect brk() failures. Malloc doesn't handle sbrk() failures

[rsa at us dot ibm dot com]

------- Additional Comments From rsa at us dot ibm dot com  2006-03-10 20:33 -------
Created an attachment (id=911)
 --> (http://sourceware.org/bugzilla/attachment.cgi?id=911&action=view)
test case to expose brk behavior

The attached test-case, when compiled with two different -DDATA_*00 flags
(representing the amount of global data inserted into the data segment of the
program) will result in brk() succeeding or failing for the same RLIMIT_DATA
softlimit.

setting rlimit == 1 PAGE and brk() succeeds.
gcc -o v2glibc214 v2glibc214.c -DDATA_200

setting rlimit = 1 PAGE and brk() fails.
gcc -o v2glibc214 v2glibc214.c -DDATA_200

I will following with some analysis data of the two runs, including what the
kernel thinks is happening.

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=214

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug libc/214] sbrk() doesn't detect brk() failures. Malloc doesn't handle sbrk() failures

[rsa at us dot ibm dot com]

------- Additional Comments From rsa at us dot ibm dot com  2006-03-10 22:18 -------
Created an attachment (id=914)
 --> (http://sourceware.org/bugzilla/attachment.cgi?id=914&action=view)
analysis of two runs, one with -DDATA_200 the other with -DDATA_300

The attached file highlights data gathered from two tests:
1.) 200 bytes of global data live in the program data segment.
2.) 300 bytes of global data live in the program data segment.

The loader chooses to push __data_start to a page boundary for the 300 byte
example and not for the 200 byte example.  RLIMIT_DATA rlim_cur is set to 1
PAGE.  The kernel addresses soft-limits only at page granularity so anything
less is effectively treated as 1 page, and anything more, as 2, 3, 4, etc.
pages.

1.) The 300 byte example fails setting brk() beyond the initial value.
2.) The 200 byte example allows brk() to push into the next page and fails when
trying to set brk() to a second page.

At least on ppc64 my analysis reveals the following (keeping in mind the
kernel's PAGE granularity for checking RLIMIT_DATA rlim_cur):

1.) Initial curbrk [kernel: mm->start_brk] is always set to start on a PAGE
boundary after _end [kernel: mm->end_data].

2.) newbrk is only checked against RLIMIT_DATA rlim_cur when the newbrk lies on
a page following curbrk.  Effectively this means that RLIMIT_DATA rlim_cur
always starts being checked at __data_start + 1 page.

3.) The loader positions __data_start (on a page boundary or not) based upon a
set of criteria dependent upon the amount of code and data in the program.  I
don't understand how the loader makes the distinction.

4.) If the loader has placed __data_start on a non-page boundary (the code and
data are sufficiently small) the mm->start_brk is set to the next page. 
RLIMIT_DATA starts accounting at the same place as mm->start_brk, a page off of
__data_start.  Subsequent brk() calls are allowed into another page. (case
-DDATA_200).

5.) If there is enough code and/or data the loader will place __data_start on a
page boundary.	The size of the data segment exceeds a page.  RLIMIT_DATA
starts accounting a page off of __data_start, hence before _end.  mm->start_brk
is then set to the page boundary off of _end (already exceeding __data_start +
rlim_cur [1 page]).  The first brk() call will fall on the next page, a
different one than the current brk and the resulting brk exceeds the
RLIMIT_DATA soft limit. (case -DDATA_300)

Here's some implications:
1.) It is misleading when the man page states that you can brk() up to the soft
limit, but this isn't true.  One can break beyond the soft-limit on a page
granularity keeping 2.) in mind.

2.) Depending on the size of the code and amount of program data the loader
inadvertently determines whether you're penalized a page or not when setting 
the RLIMIT_DATA rlim_cur soft-limit.  The brk() sys_call behavior is
unpredictable when RLIMIT_DATA rlim_cur is set.

3.) This could be a loader issue, a man page issue, a kernel issue, or all
three combined.

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=214

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug libc/214] sbrk() doesn't detect brk() failures. Malloc doesn't handle sbrk() failures

[rsa at us dot ibm dot com]

------- Additional Comments From rsa at us dot ibm dot com  2006-03-13 16:24 -------
Thanks to Alan Modra for the following explanation of how ld locates the data in
memory:

ld tries to optimize memory and disk usage for the executable.

Consider a small executable laid out like the following:

______________________
| text    |   data   |
~~~~~~~~~~~~~~~~~~~~~~
^               ^               ^
|- page boundaries

This will require two disk pages, and three memory pages,
one for text and two for data (demand paging requires that
vma and file offset are equal modulo page size, protection
requirements mean text and data must be on separate pages).


The same executable laid out like this:
____________________________
| text    | gap |   data   |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
^               ^               ^
|- page boundaries

still requires two disk pages, but now only two memory pages.
Sometimes adjusting the data start as above doesn't gain us
anything in memory and increases disk image, so ld doesn't
adjust.

See ld info DATA_SEGMENT_ALIGN.

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=214

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug libc/214] sbrk() doesn't detect brk() failures. Malloc doesn't handle sbrk() failures

[rsa at us dot ibm dot com]

------- Additional Comments From rsa at us dot ibm dot com  2006-03-14 18:34 -------
Synopsis:

o The brk() and sbrk() functions are working as designed in GLIBC and in the
Linux kernel though the man-page should be updated to reflect the actual behavior.

o A sufficiently small program and data segment will be given a free page of
data-segment data (as accounted by the kernel) due to segment alignment by 'ld'.
 (refer to the information in previous bug entries).

o The kernel only honors page granularity with regard to RLIMIT_DATA rlim_cur
soft limit.  If you ask for a (PAGE + 1) rlim_cur soft-limit the kernel will
honor brk() requests up to (PAGE * 2);  This is because the kernel allocates
memory in pages.

o /proc/self/Status : VmData is in kB, so if you want to use it as a basis for
your rlim_cur soft-limit then multiply VmData by 1024.  Setting the soft-limit
to VmData will prevent brk() calls beyond the data-segment size already required
by the program.  VmData + 1 will set the soft-limit to VmData + PAGE.

I'm pretty much done with this bug and I think it can be closed as INVALID. 
Basically the user looked at the man-page and was misled as to the functionality.

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=214

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug libc/214] sbrk() doesn't detect brk() failures. Malloc doesn't handle sbrk() failures

[dlstevens at us dot ibm dot com]

------- Additional Comments From dlstevens at us dot ibm dot com  2006-03-14 19:59 -------
"The user" would be me, and I didn't get incorrect results from a man page.
The problem, as I said in the first two entries, was that I was never able
to get a NULL return from malloc(). I always either got "success" (in some
cases without actually getting memory) or a segmentation violation. If I
recall, a malloc() that exceeded the soft limit returned the same pointer as
an already-allocated and not freed prior malloc(), which would be wrong.

It certainly is possible that I had a misconfiguration on my system, and I lost
the context beyond what I wrote here more than a year ago. So, closing the bug
is not unreasonable, but I'd be happier if you had a test that manipulates
the soft limits (correctly, if what I did was wrong) and results in a
successful allocation or NULL return from malloc(), always. No segmentation
faults, no garbage returns, etc.

In other words, if you can demonstrate a case where setting a soft limit
results in malloc() returning NULL (ever), then I think your test will be
farther than I ever got, and I'd be happy with having the bug closed. :-)

I don't care what the man page says. :-)

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=214

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug libc/214] sbrk() doesn't detect brk() failures. Malloc doesn't handle sbrk() failures

[rsa at us dot ibm dot com]

------- Additional Comments From rsa at us dot ibm dot com  2006-03-15 20:18 -------
David,

Sure I'll see what I can whip up.

Thanks for responding.  btw I was implying that the man-page's information is
wrong/outdated with-regard-to RLIMIT_DATA soft-limit, not that you actually
interpreted it wrong.

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=214

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug libc/214] sbrk() doesn't detect brk() failures. Malloc doesn't handle sbrk() failures

[dlstevens at us dot ibm dot com]

------- Additional Comments From dlstevens at us dot ibm dot com  2006-03-15 21:21 -------
Ryan, sounds good. Here's the full background on this, as far as I can
remember, in case the context helps:

I started out writing some LTP tests for IPv6 functions and I wanted to test
a memory allocation failure in a function that allocates the memory for its
returned structures. The idea was to set the soft limit in the test to whatever
was already allocated, and call the function under test, check its return for
the proper allocation failure error, raise the soft limit and move on.

When I couldn't provoke the error, I figured I was setting the soft limit
incorrectly, so I tried setting the soft limit and then allocating page-size
chunks (with malloc) to try to figure out why my soft-limit calculation was
off, and by how much. In those runs, I didn't get a NULL return at all, which
was unexpected, and the program would die with a segmentation violation at
least some of the time. (that test is long gone, or I'd append it-- I thought
this was easy to reproduce :-), so didn't in the original).

I looked at the malloc() code and discovered it appeared to expect a "-1"
return on _brk() failure, but strace() of the running program showed the Linux
kernel was returning the old brk value in that case (and looking at the Linux
kernel code verified that). Only later did I discover that glibc is remapping
the system call value, though I could tell if it was doing that correctly in
all cases-- so that may be a red herring, maybe not. I appeared to be getting
the same sbrk() value when it failed, but the memory after it was not usable
(referencing it caused a segmentation violation). So, I suspect that sbrk/brk
isn't returning the right thing for malloc() to detect the allocation failure,
which resulted in malloc() trying to use the invalid chunk of memory at the
sbrk() return value and SEGV-ing within glibc at that point.

The whole thing is complicated in malloc() by it not doing sbrk()/brk() quite
as often as I expected, which is what the business about the mmap comment
refers to. I looked at the code at the time, but don't remember enough about it
to be helpful, though you may already know more about it than I ever did. :-)

Bottom line for the bug report, as I said before, was that I was never able to
get a (graceful) failed memory allocation by setting the soft limit, and at
least some of those attempts resulted in SEGV's within glibc, which I wouldn't
expect to happen if everything is working correctly. :-)

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=214

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug libc/214] sbrk() doesn't detect brk() failures. Malloc doesn't handle sbrk() failures

[rsa at us dot ibm dot com]

------- Additional Comments From rsa at us dot ibm dot com  2006-03-16 00:29 -------
Created an attachment (id=921)
 --> (http://sourceware.org/bugzilla/attachment.cgi?id=921&action=view)
demonstration of malloc() with-regard-to RLIMIT_AS

The attached source file demonstrates three behaviors invoked with the
following:

1) gcc -o v6 v6glibc214.c
2) gcc -o v6 v6glibc214.c -DEXCEED_BRK
3) gcc -o v6 v6glibc214.c -DSET_RLIMIT_AS

1) RLIMIT_DATA rlim_cur soft limit of 2 pages is set.  ld maps this program on
a page boundary and gives us one page (out of the two) to work with so we brk
at the start_brk + 1 page.  We are allowed to read/write memory up to the brk
set by brk() without mallocing.  Subsequent mallocs grab memory further up the
address space, but not directly following curbrk.

2) We attempt to read/write to start_brk + 1 page + 4 bytes which exceeds cur
brk and creates a segmentation fault.

3) Setting RLIMIT_AS rlim_cur soft limit to RLIMIT_DATA rlim_cur soft limit
limits the address space memory that can be consumed by the program. 
Effectively this stops the mallocs from succeeding after we brk up to the
RLIMIT_DATA soft limit.  Malloc returns NULL.

synopsis:

1.) mmap isn't governed by RLIMIT_DATA and malloc may resort to mmap when
sbrk() fails.  Setting the RLIMIT_AS soft-limit to RLIMIT_DATA soft-limit will
prevent malloc from exceeding the RLIMIT_DATA soft-limit.  I haven't tried
setting RLIMIT_AS soft-limit below RLIMIT_DATA soft-limit but I 

1a.) Setting RLIMIT_AS below RLIMIT_DATA doesn't seem to prevent malloc from
using all the memory in the data segment up the RLIMIT_DATA because malloc
probably uses sbrk() until it returns 0.

2.) The kernel allocates memory on page boundaries so all soft-limits are only
enforced at the upward page boundary.

3.) I never experienced a seg fault with malloc() when mallocing data beyond
RLIMIT_DATA rlim_cur soft-limit.

After further analysis I still think this is working as designed.

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=214

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug libc/214] sbrk() doesn't detect brk() failures. Malloc doesn't handle sbrk() failures

[rsa at us dot ibm dot com]

------- Additional Comments From rsa at us dot ibm dot com  2006-03-20 19:59 -------
I've verified the same behavior on i386 as well.

David,

Can you take a look at my previous test/posting and let me know if you're
satisfied with the resolution?  RLIMIT_AS seems to do what you wanted to do in
the first place.

It seems that the SEGV you were experiencing is either fixed or you were getting
a SEGV for another reason.

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=214

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug libc/214] sbrk() doesn't detect brk() failures. Malloc doesn't handle sbrk() failures

[dlstevens at us dot ibm dot com]

------- Additional Comments From dlstevens at us dot ibm dot com  2006-03-20 20:27 -------
Yes, I wanted to (re)write some test cases to see if I can come up with an
explicit test that I think is wrong (or not) on a current glibc & linux-kernel
combination. I'll try to get to that in the next few days, but if you're in
a hurry :-), I'm ok with closing this bug. The original tests were simple, so
I don't think those were broken, but it's quite possible the self-built glibc
had a misconfiguration, too.

If I find something, we can re-open this bug or open a new one with the
particular test case.

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=214

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug libc/214] sbrk() doesn't detect brk() failures. Malloc doesn't handle sbrk() failures

[dlstevens at us dot ibm dot com]

------- Additional Comments From dlstevens at us dot ibm dot com  2006-03-21 23:56 -------
OK, I did some new testing and I agree there is no bug. I was able to
reproduce some segmentation violations, which I think were coming from
the fact that I was mixing brk()'s with the sbrk()'s, to set the end of
data segment back after growing. I expect doing debugging printf's after
an sbrk() resulted in internal glibc allocations which a later brk() was
then making invalid pointers. :-) I haven't seen any SEGV's when the sbrk()
and brk() don't have anything in between them.
I don't have the original malloc() test, but I was likely doing something
similarly silly there.

Sorry for the confusion, and thanks for all your efforts! I agree this one can
be closed.

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=214

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug libc/214] sbrk() doesn't detect brk() failures. Malloc doesn't handle sbrk() failures

[drepper at redhat dot com]

------- Additional Comments From drepper at redhat dot com  2006-04-01 20:20 -------
There is no problem in libc.

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |RESOLVED
         Resolution|                            |INVALID


http://sourceware.org/bugzilla/show_bug.cgi?id=214

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.