From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout.gmx.net (mout.gmx.net [212.227.17.21]) by sourceware.org (Postfix) with ESMTPS id B1E5E3858C2D for ; Wed, 23 Feb 2022 20:57:15 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org B1E5E3858C2D X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Received: from [192.168.178.29] ([79.251.15.46]) by mail.gmx.net (mrgmx104 [212.227.17.168]) with ESMTPSA (Nemesis) id 1MKKYx-1nfSFi0DGx-00Lnsm; Wed, 23 Feb 2022 21:57:13 +0100 Message-ID: <9cd697e4-eb65-bf4f-be28-8c67b7acdcd6@gmx.de> Date: Wed, 23 Feb 2022 21:57:11 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.5.0 Subject: Re: Problem setting buffer size for gfortran ( v 11.2) Content-Language: en-US To: "Bertini, Denis Dr." , "fortran@gcc.gnu.org" Newsgroups: gmane.comp.gcc.fortran References: <2916ad84-bfbe-757b-290e-61d9c5805367@gmx.de> From: Harald Anlauf In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable X-Provags-ID: V03:K1:Y+3ZjIEX0UiC9GAEzm/DWK1fGDO6krN31eiDlObPHxF2NBuIZKI Pz7zfsw4lF9OqAf71lqZY6V+YGLIqEbKx+rbpcg97TY48o8rTRqaMT6M+xMlpI3m5OtlmiW EN18PxQgD5TiO4lWfymxahzMtPBybdauAR2mI92RoWHmObr5GJmAN4iEkcAb6Nys6R0gnuw HHTIlEu1O5bxALsFV1d4Q== X-UI-Out-Filterresults: notjunk:1;V03:K0:DBE4yrXcp/g=:aWTZkvAev3dA3NCzzNQdph 9+eF0CyJ/XDz3UbV3/gjrFYYlyH6G3t+V7lqaIoADTQzQvqaclWLQzwN5xWg6jeeFcyetJu6M qGQwWPPhWdgjC68WZiyLlQkH1Yc/u0XvLU6plNAzAvXHkhOWb/J1nZa06iAw9BSrDDQ4BoAQc dNh4HQKdRW1bBjzGZxXlOlcQN4cEswPqxZL+JJVndBXny9SS+SJCWIMdxorYtwpZ1SJpG316e GWnTziWGYSRTvdMAc4uY91ha513AO+CzjSNTZ+Bk9AUSoDyBSKvb3wF2Ev7T3L+C6ohczKroN F+FDks7DSBPb6cJEle8cXAi3Vhi/LBX7Zt7IU9HI1GHMhatEeKqcP+/wWeM3OrT5cn3DidR9+ wieWONndaac4Qa18gFgNVqeU9rS8HubR8vgHCmlrtm8xf3dzk/iqAoyuoZZrkh2X2jYAFY+18 eX/TdYzdD3K0/fj0yTEnqU/Gx1VJn/dDgIwlCiCOe11P02JiidGOK99ggBWkjNLSNEo389yCA C7zpBgM2sYQrhrPQvx4g8EqyBUGrgsJgUU/xFkJrMH+MdndpN0YT9ym1roT8o9GQ4T0EKcOnl BK6oR5IBOSVSFwfKKGxtS2NUHcxNgSwXtnO9C/5wKB6P8oBxITYFsJorzZRoaw/FiLGLHgyBg QvqyhO0XusW7xVZsuK8hSSApKdZiiMXgVKedsT/TX8wLw/CBJ3Aq1Txq7EI+RBa7AAVd6dEte a3MkBXMnDE5e1wvg0QlhyHzdrcqN4o6KM03eFsGlMK5gWYAVvfPFypufzJN8fOgWNv4BT56xh 317Nw5PjNyMg917nptJUFDcWTJlqRo9vga7/oPnLJYinsKTad3M4pcc93Lof92/+XM00JJq6F bGdrw0mCSywvpc3fenzIwvP9SiXBlAIdllwl16ZB0RtcInhzqIY8GxTGKmeGBB5ZP4rXLPDMI PWheb0A5K9dBqY78TsEgb2/fJde1scnKoiYzQu4BQr30lYpdYs7QnhtEdC72vir5adMToGMB3 SQIG9kFYsKkxeNM0jGbuzeQWoxIw0LngoV25zb7m/qBv1P0sytpOoJmRqm7X8m/oKA== X-Spam-Status: No, score=-4.7 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, FREEMAIL_FROM, NICE_REPLY_A, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H4, RCVD_IN_MSPIKE_WL, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: fortran@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Fortran mailing list List-Unsubscribe: , List-Archive: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Feb 2022 20:57:17 -0000 Hi Denis, Am 23.02.22 um 21:28 schrieb Bertini, Denis Dr. via Fortran: > Hi Harald, > > In attachment you will find a typical strace output when running with > > gfortran8 on lustre filsystem. unfortunately I do not see any attachment to your mail. Note: GFORTRAN_FORMATTED_BUFFER_SIZE & friends was added in 10.1, and possibly backported to some 9.y, but *not* to 8.x. > As i said for gfortran version 8.x, 10.x, 11.2 setting the runtime > > env. variable to change buffer size did not work for me as you could > > see in the strace output. > > You could also see that "mmap" is used by the lustre client. > > The optimal blocksize given back by the STAT() routine indeed > > favor a 4 MiB buffer size. It might make sense to show your issue to the vendor of your system. Parallel file systems - not only Lustre - require special handling not only to get optimum performance, but also correct behavior. (E.g. the HDF5 library is known to hit bugs in our Lustre.) Cheers, Harald > > Best > > Denis > > > ________________________________ > From: Harald Anlauf > Sent: Wednesday, February 23, 2022 7:26 PM > To: Bertini, Denis Dr.; fortran@gcc.gnu.org > Subject: Re: Problem setting buffer size for gfortran ( v 11.2) > > Hi Denis, > > Am 23.02.22 um 17:10 schrieb Bertini, Denis Dr. via Fortran: >> Hi >> Investigating some sporadical performance drop when using Fortran IO on= our Lustre >> shared file system, we realized that the buffer size used as default by= the gnu fortran >> compiler is 8 kiB only. >> We wanted to change this default and according to the documentation, on= e has just to set >> the environment variables: >> >> GFORTRAN_UNFORMATTED_BUFFER_SIZE >> GFORTRAN_FORMATTED_BUFFER_SIZE=3D >> GFORTRAN_UNBUFFERED_ALL=3Dn >> >> to increase the buffer size during runtime. > > this works for me on gcc-12 development for the following > test program on Linux/x86: > > open(10,file=3D"bigfile") > write(10,*) "data" > close(10) > end > > Compile, link, and run under strace with > > GFORTRAN_FORMATTED_BUFFER_SIZE=3D4194304 strace ./a.out > > [...] > openat(AT_FDCWD, "bigfile", O_RDWR|O_CREAT|O_CLOEXEC, 0666) =3D 3 > fstat(3, {st_mode=3DS_IFREG|0644, st_size=3D6, ...}) =3D 0 > mmap(NULL, 4198400, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, > 0) =3D 0x146b69b1c000 > write(3, " data\n", 6) =3D 6 > ftruncate(3, 6) =3D 0 > munmap(0x146b69b1c000, 4198400) =3D 0 > close(3) =3D 0 > >> Unfortunately isetting these variables does not change the default bu= ffer size. >> I tried with gfortran 8, 10, and the new 11.2. >> Looking a the code, i found in >> io/unix.c >> that the default is set to the following value, >> >> static const int FORMATTED_BUFFER_SIZE_DEFAULT =3D 8192; >> static const int UNFORMATTED_BUFFER_SIZE_DEFAULT =3D 128*1024; >> >> Of course, changing the default value and recompiling works and change = accordingly the buffer size >> It should nevertheless be possible to change these values without recom= piling. >> Is there something one should be aware of when setting these variables? > > You said you're using Lustre. I guess you use the default > blocksize (4MB). Does your Lustre client support mmap? > What does strace report? > > I remember I once saw a system where we had to disable the use > of mmap in a library. > > Cheers, > Harald > >> Best regards, >> Denis >> >> --------- >> Denis Bertini >> Abteilung: CIT >> Ort: SB3 2.265a >> >> Tel: +49 6159 71 2240 >> Fax: +49 6159 71 2986 >> E-Mail: d.bertini@gsi.de >> >> GSI Helmholtzzentrum f=C3=BCr Schwerionenforschung GmbH >> Planckstra=C3=9Fe 1, 64291 Darmstadt, Germany, www.gsi.de >> >> Commercial Register / Handelsregister: Amtsgericht Darmstadt, HRB 1528 >> Managing Directors / Gesch=C3=A4ftsf=C3=BChrung: >> Professor Dr. Paolo Giubellino, Dr. Ulrich Breuer, J=C3=B6rg Blaurock >> Chairman of the GSI Supervisory Board / Vorsitzender des GSI-Aufsichtsr= ats: >> Ministerialdirigent Dr. Volkmar Dietz >> > > From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from ciao.gmane.io (ciao.gmane.io [116.202.254.214]) by sourceware.org (Postfix) with ESMTPS id ED2BD3858439 for ; Wed, 23 Feb 2022 20:57:20 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org ED2BD3858439 Received: from list by ciao.gmane.io with local (Exim 4.92) (envelope-from ) id 1nMyhK-0009bL-7n for fortran@gcc.gnu.org; Wed, 23 Feb 2022 21:57:18 +0100 X-Injected-Via-Gmane: http://gmane.org/ To: fortran@gcc.gnu.org From: Harald Anlauf Subject: Re: Problem setting buffer size for gfortran ( v 11.2) Date: Wed, 23 Feb 2022 21:57:11 +0100 Message-ID: <9cd697e4-eb65-bf4f-be28-8c67b7acdcd6@gmx.de> References: <2916ad84-bfbe-757b-290e-61d9c5805367@gmx.de> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.5.0 Content-Language: en-US In-Reply-To: X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00, FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS, KAM_DMARC_STATUS, NICE_REPLY_A, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: fortran@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Fortran mailing list List-Unsubscribe: , List-Archive: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Feb 2022 20:57:22 -0000 Message-ID: <20220223205711.FNpAhkrYSBS1DCMV8FsPBp294HsYErHHMp8OggxTqpY@z> Hi Denis, Am 23.02.22 um 21:28 schrieb Bertini, Denis Dr. via Fortran: > Hi Harald, > > In attachment you will find a typical strace output when running with > > gfortran8 on lustre filsystem. unfortunately I do not see any attachment to your mail. Note: GFORTRAN_FORMATTED_BUFFER_SIZE & friends was added in 10.1, and possibly backported to some 9.y, but *not* to 8.x. > As i said for gfortran version 8.x, 10.x, 11.2 setting the runtime > > env. variable to change buffer size did not work for me as you could > > see in the strace output. > > You could also see that "mmap" is used by the lustre client. > > The optimal blocksize given back by the STAT() routine indeed > > favor a 4 MiB buffer size. It might make sense to show your issue to the vendor of your system. Parallel file systems - not only Lustre - require special handling not only to get optimum performance, but also correct behavior. (E.g. the HDF5 library is known to hit bugs in our Lustre.) Cheers, Harald > > Best > > Denis > > > ________________________________ > From: Harald Anlauf > Sent: Wednesday, February 23, 2022 7:26 PM > To: Bertini, Denis Dr.; fortran@gcc.gnu.org > Subject: Re: Problem setting buffer size for gfortran ( v 11.2) > > Hi Denis, > > Am 23.02.22 um 17:10 schrieb Bertini, Denis Dr. via Fortran: >> Hi >> Investigating some sporadical performance drop when using Fortran IO on our Lustre >> shared file system, we realized that the buffer size used as default by the gnu fortran >> compiler is 8 kiB only. >> We wanted to change this default and according to the documentation, one has just to set >> the environment variables: >> >> GFORTRAN_UNFORMATTED_BUFFER_SIZE >> GFORTRAN_FORMATTED_BUFFER_SIZE= >> GFORTRAN_UNBUFFERED_ALL=n >> >> to increase the buffer size during runtime. > > this works for me on gcc-12 development for the following > test program on Linux/x86: > > open(10,file="bigfile") > write(10,*) "data" > close(10) > end > > Compile, link, and run under strace with > > GFORTRAN_FORMATTED_BUFFER_SIZE=4194304 strace ./a.out > > [...] > openat(AT_FDCWD, "bigfile", O_RDWR|O_CREAT|O_CLOEXEC, 0666) = 3 > fstat(3, {st_mode=S_IFREG|0644, st_size=6, ...}) = 0 > mmap(NULL, 4198400, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, > 0) = 0x146b69b1c000 > write(3, " data\n", 6) = 6 > ftruncate(3, 6) = 0 > munmap(0x146b69b1c000, 4198400) = 0 > close(3) = 0 > >> Unfortunately isetting these variables does not change the default buffer size. >> I tried with gfortran 8, 10, and the new 11.2. >> Looking a the code, i found in >> io/unix.c >> that the default is set to the following value, >> >> static const int FORMATTED_BUFFER_SIZE_DEFAULT = 8192; >> static const int UNFORMATTED_BUFFER_SIZE_DEFAULT = 128*1024; >> >> Of course, changing the default value and recompiling works and change accordingly the buffer size >> It should nevertheless be possible to change these values without recompiling. >> Is there something one should be aware of when setting these variables? > > You said you're using Lustre. I guess you use the default > blocksize (4MB). Does your Lustre client support mmap? > What does strace report? > > I remember I once saw a system where we had to disable the use > of mmap in a library. > > Cheers, > Harald > >> Best regards, >> Denis >> >> --------- >> Denis Bertini >> Abteilung: CIT >> Ort: SB3 2.265a >> >> Tel: +49 6159 71 2240 >> Fax: +49 6159 71 2986 >> E-Mail: d.bertini@gsi.de >> >> GSI Helmholtzzentrum für Schwerionenforschung GmbH >> Planckstraße 1, 64291 Darmstadt, Germany, www.gsi.de >> >> Commercial Register / Handelsregister: Amtsgericht Darmstadt, HRB 1528 >> Managing Directors / Geschäftsführung: >> Professor Dr. Paolo Giubellino, Dr. Ulrich Breuer, Jörg Blaurock >> Chairman of the GSI Supervisory Board / Vorsitzender des GSI-Aufsichtsrats: >> Ministerialdirigent Dr. Volkmar Dietz >> > >