[Bug other/43955] New: linux hard lock-up when gcc output file is on filesystem with insufficient disk space

[Bug other/43955] New: linux hard lock-up when gcc output file is on filesystem with insufficient disk space

[jason dot vas dot dias at gmail dot com]

With gcc 4.1.x - gcc 4.4.x , on linux-2.6.x (many "x" versions tried over the
last few years) , whenever gcc tries to output to a file on a filesystem that
does not contain sufficient disk space, the linux kernel silently locks up - 
I'm not sure if this is a kernel bug or a gcc bug, but a bug it definitely is.
I thought it may be because I tend to run a "bleading edge" unstable kernel,
but
it also happens on the last three stable versions ( 2.6.33, 2.6.32.12, 2.6.32.1
) with gcc 4.4.2, 4.4.1, 4.3.4, 4.3.3, 4.2.4, 4.2.3, 4.2.1, 4.1.2, 4.1.1, 3.4.6
.
There is no kernel crash report or log message, just a total system freeze, 
necessitating a hard reboot (with potential filesytem damage) when this occurs
. 
This has happened to me too many times when a gcc compilation is the only
activity on the system for there to be any other possible cause . 

I've now enabled verbose kernel lock debugging, and will post any relevant
results here if it happens again.

I think gcc should be detecting the case when there is insufficient disk space
to complete a write(2) better, and should exit gracefully in that case. 

It appears to be retrying failed write(2)s infinitely ;
but even that should not cause the kernel to hang .

Any ideas?


-- 
           Summary: linux hard lock-up when gcc output file is on filesystem
                    with insufficient disk space
           Product: gcc
           Version: unknown
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: other
        AssignedTo: unassigned at gcc dot gnu dot org
        ReportedBy: jason dot vas dot dias at gmail dot com
 GCC build triplet: x86_64-linux-gnu
  GCC host triplet: x86_64-linux-gnu
GCC target triplet: x86_64-linux-gnu


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=43955

[Bug other/43955] linux hard lock-up when gcc output file is on filesystem with insufficient disk space

[rguenth at gcc dot gnu dot org]

------- Comment #1 from rguenth at gcc dot gnu dot org  2010-05-01 22:08 -------
GCC does not write infinitely.  This is a kernel bug.


-- 

rguenth at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |RESOLVED
         Resolution|                            |INVALID


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=43955

[Bug other/43955] linux hard lock-up when gcc output file is on filesystem with insufficient disk space

[jason dot vas dot dias at gmail dot com]

------- Comment #2 from jason dot vas dot dias at gmail dot com  2010-05-01 22:41 -------
OK, but you cannot state "this is a kernel bug" without proving this to be the
case .
Every time I make a mistake about available disk space and start a gcc compile,
and disk space is exhausted, the machine totally locks up: no response from
keyboard, (including magic "Sys-Rq" keystroke) , mouse, network or any attached
serial or USB device .
Why does this happen with no other command ?


-- 

jason dot vas dot dias at gmail dot com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jason dot vas dot dias at
                   |                            |gmail dot com
             Status|RESOLVED                    |UNCONFIRMED
         Resolution|INVALID                     |


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=43955

[Bug other/43955] linux hard lock-up when gcc output file is on filesystem with insufficient disk space

[pinskia at gcc dot gnu dot org]

------- Comment #3 from pinskia at gcc dot gnu dot org  2010-05-01 22:44 -------
GCC is an user level program.  No user program should cause the kernel to lock
up  therefore it is a kernel issue.


-- 

pinskia at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |RESOLVED
         Resolution|                            |INVALID


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=43955

[Bug other/43955] linux hard lock-up when gcc output file is on filesystem with insufficient disk space

[jason dot vas dot dias at gmail dot com]

------- Comment #4 from jason dot vas dot dias at gmail dot com  2010-05-01 22:47 -------
Has there been any history of reported linux kernel lock-ups for
gcc compilations outputting to filesystems with insufficient disk space ?

It could be something to do with using "make -j2" on my dual processor machine
and outputting stderr + stdout to a "tee logfile" process. 

Of course, because the machine totally ceases to respond to any device, and
does not create any crash report, the only evidence left is a few corrupt *.o
files and terminated log files such as :

  gcc -Wp,-MD,fs/nfs/.proc.o.d  -nostdinc -isystem
/usr/lib/gcc/x86_64-pc-linux-gnu/4.4.2/include
-I/usr/src/linux-2.6/arch/x86/include -Iinclude  -include
include/generated/autoconf.h -D__KERNEL__ -Wall -Wundef -Wstrict-prototypes
-Wno-trigraphs -fno-strict-aliasing -fno-common
-Werror-implicit-function-declaration -Wno-format-security
-fno-delete-null-pointer-checks -O2 -m64 -march=k8 -mno-red-zone
-mcmodel=kernel -funit-at-a-time -maccumulate-outgoing-args -fstack-protector
-DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -pipe -Wno-sign-compare
-fno-asynchronous-unwind-tables -mno-sse -mno-mmx -mno-sse2 -mno-3dnow
-Wframe-larger-than=2048 -fno-omit-frame-pointer -fno-optimize-sibling-calls -g
-pg -Wdeclaration-after-statement -Wno-pointer-sign -fno-strict-overflow
-fno-dwarf2-cfi-asm -fconserve-stack  -DMODULE -D"KBUILD_STR(s)=#s"
-D"KBUILD_BASENAME=KBUILD_STR(proc)"  -D"KBUILD_MODNAME=KBUILD_STR(nfs)"
-D"DEBUG_HASH=59" -D"DEBUG_HASH2=19" -c -o fs/nfs/proc.o fs/nfs/proc.c
  set -e ; perl /usr/src/linux-2.6/scripts/recordmcount.pl "x86_64" "little"
"64" "objdump" "objcopy" "gcc" "ld" "nm" "" "" "1"
"drivers/block/paride/on20.o";
  gcc -Wp,-MD,drivers/block/paride/.on26.o.d  -nostdinc -isystem
/usr/lib/gcc/x86_64-pc-linux-gnu/4.4.2/include
-I/usr/src/linux-2.6/arch/x86/include -Iinclude  -include
include/generated/autoconf.h -D__KERNEL__ -Wall -Wundef -Wstrict-prototypes
-Wno-trigraphs -fno-strict-aliasing -fno-common
-Werror-implicit-function-declaration -Wno-format-security
-fno-delete-null-pointer-checks -O2 -m64 -march=k8 -mno-red-zone
-mcmodel=kernel -funit-at-a-time -maccumulate-outgoing-args -fstack-protector
-DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -pipe -Wno-sign-compare
-fno-asynchronous-unwind-tables -mno-sse -mno-mmx -mno-sse2 -mno-3dnow
-Wframe-larger-than=2048 -fno-omit-fram


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=43955

[Bug other/43955] linux hard lock-up when gcc output file is on filesystem with insufficient disk space

[jason dot vas dot dias at gmail dot com]

------- Comment #5 from jason dot vas dot dias at gmail dot com  2010-05-01 22:55 -------
Sometimes disk space runs out before the machine locks up, 
but I get loads of invalid objects ;  

I think it is a bug that gcc can continue after producing invalid object files:

  gcc -Wp,-MD,arch/x86/power/.cpu.o.d  -nostdinc -isystem
/usr/lib/gcc/x86_64-pc-linux-gnu/4.4.2/include
-I/usr/src/linux-2.6/arch/x86/include -Iinclude  -include
include/generated/autoconf.h -D__KERNEL__ -Wall -Wundef -Wstrict-prototypes
-Wno-trigraphs -fno-strict-aliasing -fno-common
-Werror-implicit-function-declaration -Wno-format-security
-fno-delete-null-pointer-checks -O2 -m64 -march=k8 -mno-red-zone
-mcmodel=kernel -funit-at-a-time -maccumulate-outgoing-args -fstack-protector
-DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -pipe -Wno-sign-compare
-fno-asynchronous-unwind-tables -mno-sse -mno-mmx -mno-sse2 -mno-3dnow
-Wframe-larger-than=2048 -fno-omit-frame-pointer -fno-optimize-sibling-calls -g
-pg -Wdeclaration-after-statement -Wno-pointer-sign -fno-strict-overflow
-fno-dwarf2-cfi-asm -fconserve-stack -fno-stack-protector  
-D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(cpu)" 
-D"KBUILD_MODNAME=KBUILD_STR(cpu)" -D"DEBUG_HASH=28" -D"DEBUG_HASH2=39" -c -o
arch/x86/power/cpu.o arch/x86/power/cpu.c                                       
{standard input}: Assembler messages:                                           
{standard input}:66218: Fatal error: can't write
drivers/infiniband/ulp/iser/iser_initiator.o: No space left on device           
{standard input}:66218: Fatal error: can't close
drivers/infiniband/ulp/iser/iser_initiator.o: No space left on device           
make[3]: *** [drivers/infiniband/ulp/iser/iser_initiator.o] Error 1             
make[2]: *** [drivers/infiniband/ulp/iser] Error 2                              
make[1]: *** [drivers/infiniband] Error 2                                       
make: *** [drivers] Error 2                                                     
make: *** Waiting for unfinished jobs....                                       
  gcc -Wp,-MD,arch/x86/power/.hibernate_64.o.d  -nostdinc -isystem
/usr/lib/gcc/x86_64-pc-linux-gnu/4.4.2/include
-I/usr/src/linux-2.6/arch/x86/include -Iinclude  -include
include/generated/autoconf.h -D__KERNEL__ -Wall -Wundef -Wstrict-prototypes
-Wno-trigraphs -fno-strict-aliasing -fno-common
-Werror-implicit-function-declaration -Wno-format-security
-fno-delete-null-pointer-checks -O2 -m64 -march=k8 -mno-red-zone
-mcmodel=kernel -funit-at-a-time -maccumulate-outgoing-args -fstack-protector
-DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -pipe -Wno-sign-compare
-fno-asynchronous-unwind-tables -mno-sse -mno-mmx -mno-sse2 -mno-3dnow
-Wframe-larger-than=2048 -fno-omit-frame-pointer -fno-optimize-sibling-calls -g
-pg -Wdeclaration-after-statement -Wno-pointer-sign -fno-strict-overflow
-fno-dwarf2-cfi-asm -fconserve-stack   -D"KBUILD_STR(s)=#s"
-D"KBUILD_BASENAME=KBUILD_STR(hibernate_64)" 
-D"KBUILD_MODNAME=KBUILD_STR(hibernate_64)" -D"DEBUG_HASH=15"
-D"DEBUG_HASH2=16" -c -o arch/x86/power/hibernate_64.o
arch/x86/power/hibernate_64.c                                                   
{standard input}: Assembler messages:                                           
{standard input}:29504: Fatal error: can't write arch/x86/power/cpu.o: No space
left on device                                                                  
{standard input}:29504: Fatal error: can't close arch/x86/power/cpu.o: No space
left on device                                                                  
make[1]: *** [arch/x86/power/cpu.o] Error 1                                     
make[1]: *** Waiting for unfinished jobs....                                    
{standard input}: Assembler messages:                                           
{standard input}:25073: Fatal error: can't write arch/x86/power/hibernate_64.o:
No space left on device                                                         
{standard input}:25073: Fatal error: can't close arch/x86/power/hibernate_64.o:
No space left on device                                                         
make[1]: *** [arch/x86/power/hibernate_64.o] Error 1                            
make: *** [arch/x86/power] Error 2                                              
[ root@jvdspc:/usr/src/linux 23:47:05 545:44 ]                                  
$ find . -type f -a '(' '!' -type l ')' -a '(' -name '*.[oa]' -o -name '.so*'
')' | while read f; do if [[ ! $(file $f) =~ ELF ]]; then echo bad obj: $f; rm
-f $obj; else d=$(dirname $f); mkdir -p /mnt/sda3/linux/${d} && cp -fp $f
/mnt/sda3/linux/$d && rm -f $f && ln -s /mnt/sda3/linux/$f $f; fi; done         
bad obj: ./drivers/block/aoe/built-in.o                                         
bad obj: ./drivers/block/drbd/built-in.o                                        
bad obj: ./drivers/block/paride/built-in.o                                      
bad obj: ./drivers/bluetooth/built-in.o                                         
bad obj: ./drivers/char/ip2/built-in.o                                          
bad obj: ./drivers/char/ipmi/built-in.o                                         
bad obj: ./drivers/char/mwave/built-in.o                                        
bad obj: ./drivers/char/pcmcia/ipwireless/built-in.o                            
bad obj: ./drivers/crypto/built-in.o                                            
bad obj: ./drivers/dca/built-in.o                                               
bad obj: ./drivers/dma/ioat/built-in.o                                          
bad obj: ./drivers/firewire/built-in.o                                          
bad obj: ./drivers/gpu/drm/i2c/built-in.o                                       
bad obj: ./drivers/gpu/drm/i810/built-in.o                                      
bad obj: ./drivers/gpu/drm/i830/built-in.o                                      
bad obj: ./drivers/gpu/drm/i915/built-in.o                                      
bad obj: ./drivers/gpu/drm/mga/built-in.o                                       
bad obj: ./drivers/gpu/drm/r128/built-in.o                                      
bad obj: ./drivers/gpu/drm/savage/built-in.o                                    
bad obj: ./drivers/gpu/drm/sis/built-in.o                                       
bad obj: ./drivers/gpu/drm/tdfx/built-in.o                                      
bad obj: ./drivers/gpu/drm/via/built-in.o                                       
bad obj: ./drivers/hid/usbhid/built-in.o                                        
bad obj: ./drivers/hid/built-in.o                                               
bad obj: ./drivers/i2c/busses/built-in.o                                        
bad obj: ./drivers/idle/built-in.o                                              
bad obj: ./drivers/ieee1394/built-in.o                                          
bad obj: ./drivers/ieee802154/built-in.o                                        
bad obj: ./drivers/infiniband/core/built-in.o                                   
bad obj: ./drivers/infiniband/hw/amso1100/built-in.o                            
bad obj: ./drivers/infiniband/hw/cxgb3/built-in.o                               
bad obj: ./drivers/infiniband/hw/ipath/built-in.o                               
bad obj: ./drivers/infiniband/hw/mlx4/built-in.o                                
bad obj: ./drivers/infiniband/hw/mthca/built-in.o                               
bad obj: ./drivers/infiniband/hw/nes/built-in.o                                 
bad obj: ./drivers/infiniband/ulp/ipoib/built-in.o                              
bad obj: ./drivers/infiniband/ulp/iser/built-in.o                               
bad obj: ./drivers/infiniband/built-in.o                                        
bad obj: ./drivers/input/joystick/built-in.o                                    
bad obj: ./fs/ncpfs/.sock.o.cmd                                                 
bad obj: ./fs/nfs_common/built-in.o                                             
bad obj: ./fs/nfsd/built-in.o                                                   
bad obj: ./fs/nilfs2/built-in.o                                                 
bad obj: ./fs/ntfs/built-in.o                                                   
bad obj: ./fs/ocfs2/cluster/built-in.o                                          
bad obj: ./fs/ocfs2/dlm/built-in.o                                              
bad obj: ./fs/ocfs2/dlmfs/built-in.o                                            
bad obj: ./fs/ocfs2/built-in.o                                                  
bad obj: ./fs/omfs/built-in.o                                                   
bad obj: ./fs/proc/.softirqs.o.cmd                                              
bad obj: ./fs/qnx4/built-in.o                                                   
bad obj: ./fs/reiserfs/built-in.o                                               
bad obj: ./fs/smbfs/built-in.o                                                  
bad obj: ./fs/smbfs/.sock.o.cmd                                                 
bad obj: ./fs/squashfs/built-in.o                                               
bad obj: ./fs/sysv/built-in.o                                                   
bad obj: ./fs/ubifs/built-in.o                                                  
bad obj: ./fs/ufs/built-in.o                                                    
bad obj: ./fs/xfs/built-in.o                                                    
bad obj: ./sound/arm/built-in.o                                                 
bad obj: ./sound/atmel/built-in.o                                               
bad obj: ./sound/core/oss/built-in.o                                            
bad obj: ./sound/core/seq/oss/built-in.o                                        
bad obj: ./sound/core/seq/built-in.o                                            
bad obj: ./sound/drivers/mpu401/built-in.o                                      
bad obj: ./sound/drivers/opl3/built-in.o                                        
bad obj: ./sound/drivers/opl4/built-in.o                                        
bad obj: ./sound/drivers/pcsp/built-in.o                                        
bad obj: ./sound/drivers/vx/built-in.o                                          
bad obj: ./sound/drivers/built-in.o                                             
bad obj: ./sound/i2c/other/built-in.o                                           
bad obj: ./sound/i2c/built-in.o                                                 
bad obj: ./sound/isa/ad1816a/built-in.o                                         
bad obj: ./sound/isa/ad1848/built-in.o                                          
bad obj: ./sound/isa/cs423x/built-in.o                                          
bad obj: ./sound/isa/es1688/built-in.o                                          
bad obj: ./sound/isa/gus/built-in.o                                             
bad obj: ./sound/isa/msnd/built-in.o                                            
bad obj: ./sound/isa/opti9xx/built-in.o                                         
bad obj: ./sound/isa/sb/built-in.o                                              
bad obj: ./sound/isa/wavefront/built-in.o                                       
bad obj: ./sound/isa/wss/built-in.o                                             
bad obj: ./sound/isa/built-in.o                                                 
bad obj: ./sound/mips/built-in.o                                                
bad obj: ./sound/oss/built-in.o                                                 
bad obj: ./sound/oss/.soundcard.o.cmd                                           
bad obj: ./sound/oss/.sound_timer.o.cmd                                         
bad obj: ./sound/oss/.sound.o.cmd                                               
bad obj: ./sound/parisc/built-in.o                                              
bad obj: ./sound/pci/ac97/built-in.o                                            
bad obj: ./sound/pci/ali5451/built-in.o                                         
bad obj: ./sound/pci/au88x0/built-in.o                                          
bad obj: ./sound/pci/aw2/built-in.o                                             
bad obj: ./sound/pci/ca0106/built-in.o                                          
bad obj: ./sound/pci/cs46xx/built-in.o                                          
bad obj: ./sound/pci/cs5535audio/built-in.o                                     
bad obj: ./sound/pci/ctxfi/built-in.o                                           
bad obj: ./sound/pci/echoaudio/built-in.o                                       
bad obj: ./sound/pci/emu10k1/built-in.o                                         
bad obj: ./sound/pci/hda/built-in.o                                             
bad obj: ./sound/pci/ice1712/built-in.o                                         
bad obj: ./sound/pci/korg1212/built-in.o
bad obj: ./sound/pci/lx6464es/built-in.o
bad obj: ./sound/pci/mixart/built-in.o
bad obj: ./sound/pci/nm256/built-in.o
bad obj: ./sound/pci/oxygen/built-in.o
bad obj: ./sound/pci/pcxhr/built-in.o
bad obj: ./sound/pci/riptide/built-in.o
bad obj: ./sound/pci/rme9652/built-in.o
bad obj: ./sound/pci/trident/built-in.o
bad obj: ./sound/pci/vx222/built-in.o
bad obj: ./sound/pci/ymfpci/built-in.o
bad obj: ./sound/pci/built-in.o
bad obj: ./sound/pcmcia/pdaudiocf/built-in.o
bad obj: ./sound/pcmcia/vx/built-in.o
bad obj: ./sound/ppc/built-in.o
bad obj: ./sound/sh/built-in.o
bad obj: ./sound/soc/atmel/built-in.o
bad obj: ./sound/soc/au1x/built-in.o
bad obj: ./sound/soc/blackfin/built-in.o
bad obj: ./sound/soc/codecs/built-in.o
bad obj: ./sound/soc/davinci/built-in.o
bad obj: ./sound/soc/fsl/built-in.o
bad obj: ./sound/soc/imx/built-in.o
bad obj: ./sound/soc/omap/built-in.o
bad obj: ./sound/soc/pxa/built-in.o
bad obj: ./sound/soc/s3c24xx/built-in.o
bad obj: ./sound/soc/s6000/built-in.o
bad obj: ./sound/soc/sh/built-in.o
bad obj: ./sound/soc/txx9/built-in.o
bad obj: ./sound/soc/built-in.o
bad obj: ./sound/sparc/built-in.o
bad obj: ./sound/spi/built-in.o
bad obj: ./sound/synth/emux/built-in.o
bad obj: ./sound/synth/built-in.o
bad obj: ./sound/usb/caiaq/built-in.o
bad obj: ./sound/usb/usx2y/built-in.o
bad obj: ./sound/usb/built-in.o


-- 

jason dot vas dot dias at gmail dot com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |UNCONFIRMED
         Resolution|INVALID                     |


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=43955

[Bug other/43955] linux hard lock-up when gcc output file is on filesystem with insufficient disk space

[jason dot vas dot dias at gmail dot com]

------- Comment #6 from jason dot vas dot dias at gmail dot com  2010-05-01 23:13 -------
Aha ! I think I see the problem, or at least A problem :

$ strace -f gcc ~jason/t.c -o /tmp/t 2>&1 | grep sync
$

Nothing is doing a fsync(int fd) for any object file written by gcc .

Hence writes by gcc can be entirely to the filesystem cache, and when disk
space
runs out, many files may not have been sync()-ed to disk within a large make(1) 
build.

I propose that gcc (and binutils!) should do an fsync() of each output file
written before closing and returning success.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=43955

[Bug other/43955] linux hard lock-up when gcc output file is on filesystem with insufficient disk space

[jsm28 at gcc dot gnu dot org]

------- Comment #7 from jsm28 at gcc dot gnu dot org  2010-05-02 11:45 -------
It is the responsibility of your filesystem to ensure that write or close
return error status if there isn't sufficient disk space; if they do not,
this is a kernel bug.  Most applications should never call fsync; that's only
if the application is meant to be making data integrity guarantees across
unclean shutdown, and it never makes sense to make such guarantees for
compiler and assembler output since those files can always be recreated from
the associated input files; it doesn't matter if the last few seconds'
compilation output before a crash is lost or corrupted.


-- 

jsm28 at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |RESOLVED
         Resolution|                            |INVALID


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=43955