From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp.bonedaddy.net (smtp.bonedaddy.net [45.33.94.42]) by sourceware.org (Postfix) with ESMTPS id 217D23857BB2 for ; Wed, 23 Nov 2022 02:04:27 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 217D23857BB2 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=bonedaddy.net Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=bonedaddy.net Received: from localhost (unknown [49.194.240.164]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: pabs3@bonedaddy.net) by smtp.bonedaddy.net (Postfix) with ESMTPSA id 6F72D305E41 for ; Tue, 22 Nov 2022 21:04:30 -0500 (EST) Authentication-Results: smtp.bonedaddy.net; dmarc=fail (p=none dis=none) header.from=bonedaddy.net DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bonedaddy.net; s=mail; t=1669169071; bh=OOOm7JE5R9e73mlxBgU+pndUGeeYTEcPRFSm8oE2qaA=; h=Subject:From:To:Date; b=flTQk7y+z1yIKe2avF8t9lzEr1E/gGB6DSvG2ay4wdPTUK0gJB/0ffOqwjpP+Wn+0 TpCQ4rHutUc1Kzr3WFVx8loPv9LoTBDWHWM4yI1KQA9xMw6AsksKGhDLPtSDPeA+MJ C8JbympPjCjev724q7fTthexfCSPlYI4Oge7a7kALLqHIM9q7y4/9Wqa+F/HMhEc6U /1U4efs9iLcm7hECgvEB863/7GiQFSjHyw030yfvo7Qi5g+XpxXLs0oGzUoCBcRl6/ 15FBzkunX8mBqouFXIw6Qrz/NUOxKlOA3BBiA8D7f4g+4u8rBpVZqmkR7P0G9OeJAH +zHfg2zHSrixw== Message-ID: <2cefc4fa95dd439c2581f4f06d520c004cd33708.camel@bonedaddy.net> Subject: is this a bug in glibc or readpst? From: Paul Wise To: libc-help Content-Type: multipart/signed; micalg="pgp-sha512"; protocol="application/pgp-signature"; boundary="=-64OxoEgE5N3bn/8w38h1" Date: Wed, 23 Nov 2022 10:02:57 +0800 MIME-Version: 1.0 User-Agent: Evolution 3.46.1-1 X-Spam-Status: No, score=-0.7 required=5.0 tests=BAYES_00,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_PASS,SPF_PASS,TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: --=-64OxoEgE5N3bn/8w38h1 Content-Type: multipart/mixed; boundary="=-IWXCpHr2v42yWBsXP/IE" --=-IWXCpHr2v42yWBsXP/IE Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi folks, I have a strange bug that might be an issue in glibc or it might be a bug in how readpst works and or uses freopen in multi-process mode. This is a summary of how the debugging process went up until now: readpst from Debian buster in multi-process mode works but readpst from Debian bullseye randomly loses some data. Current readpst works on Debian buster but not Debian bullseye. The problem isn't related to the GCC optimisation level. The problem isn't compiler related, clang exhibits the problem too. Upgrading libc6 from 2.28-10 to 2.29-1 caused the issue. Bisecting glibc pointed at commit 0b727ed4d, which is titled "libio: Flush stream at freopen (BZ#21037)" and looks legitimate as it aligns glibc freopen with POSIX specifications. readpst is using freopen() after fork() to get new *.pst FILE pointers for child processes. Both the parent and child FILE are opened read-only. The FILE position is 0 after freopen for both scenarios. readpst seems to be skipping some PST file blocks in the broken scenario. The debug logs seem to indicate that in the broken scenario it reads data from a wrong location, even though the file position is 0 after freopen. Switching the readpst code to use fclose()+fopen() after fork() instead of freopen() after fork() fixes the issue. Here are the glibc freopen commit and the initial libpst bug report. https://sourceware.org/git/?p=3Dglibc.git;a=3Dcommitdiff;h=3D0b727ed4d605d9= 318cb0d323c88abb0d5a441a9b https://github.com/pst-format/libpst/issues/7 The readpst forking and freopen functions are here: https://github.com/pst-format/libpst/blob/main/src/readpst.c#L203 https://github.com/pst-format/libpst/blob/main/src/libpst.c#L395 I have been debugging this using the attached scripts on a Debian system. The outside script sets up a chroot using Debian schroot and then runs the inside script to do the test inside the chroot. If you want to run the inside script you may want to customise the paths it uses so the script doesn't put files in your home directory. I am hoping someone can help give me an idea if this is likely to be a problem in glibc or readpst/libpst and or what other debugging strategy might be useful to figure that out and pinpoint where the problem is. --=20 bye, pabs https://bonedaddy.net/pabs3/ --=-IWXCpHr2v42yWBsXP/IE Content-Type: application/x-shellscript; name="outside" Content-Disposition: attachment; filename="outside" Content-Transfer-Encoding: base64 IyEvYmluL2Jhc2gKc2V0IC1leApzY2hyb290LWxpc3Qtc2Vzc2lvbnMgfCBncmVwIF9hbWQ2NC1k Y2hyb290LSB8IGN1dCAtZDogLWYxIHwgeGFyZ3MgLXJuMSBzY2hyb290IC1lIC1jCiNmb3IgZGlz dHJvIGluIGJ1c3RlciBidWxsc2V5ZSBib29rd29ybSBzaWQgOyBkbwojZm9yIGRpc3RybyBpbiBi dWxsc2V5ZSBib29rd29ybSA7IGRvCiNmb3IgZGlzdHJvIGluIGJ1bGxzZXllIDsgZG8KZm9yIGRp c3RybyBpbiBib29rd29ybSA7IGRvCglzZXNzaW9uaWQ9JChzY2hyb290IC1iIC1jICRkaXN0cm8p CglkZC1zY2hyb290LWNtZCAteSAtYyAkc2Vzc2lvbmlkIGFwdC1nZXQgdXBkYXRlCglkZC1zY2hy b290LWNtZCAteSAtYyAkc2Vzc2lvbmlkIGFwdC1nZXQgZGlzdC11cGdyYWRlCglkZC1zY2hyb290 LWNtZCAteSAtYyAkc2Vzc2lvbmlkIGFwdC1nZXQgYnVpbGQtZGVwIGxpYnBzdAoJZGQtc2Nocm9v dC1jbWQgLXkgLWMgJHNlc3Npb25pZCBhcHQtZ2V0IGluc3RhbGwgbmFubyBxdWlsdCBkZXZzY3Jp cHRzIGRlYmlhbi1nb29kaWVzIGRjdHJsLXRvb2xzIGJhc2gtY29tcGxldGlvbiBnaXQgd2dldCBh dXRvY29uZi1hcmNoaXZlIHhtbHRvIGRveHlnZW4gY2NhY2hlIHRpbWUgdmFsZ3JpbmQKCXNjaHJv b3QgLXIgLWMgJHNlc3Npb25pZCAtLSB+L2luc2lkZSAkZGlzdHJvCglzY2hyb290IC1lIC1jICRz ZXNzaW9uaWQKZG9uZQo= --=-IWXCpHr2v42yWBsXP/IE Content-Type: application/x-shellscript; name="inside" Content-Disposition: attachment; filename="inside" Content-Transfer-Encoding: base64 IyEvYmluL2Jhc2gKc2V0IC1leApkaXN0cm89IiQxIgoKb3B0PX4vb3B0LWxpYnBzdApkZWJ1Zz1+ L2RlYnVnLWxpYnBzdApsaWJwc3Q9fi9saWJwc3QKcHJvYz0kKG5wcm9jKQoKYmFzZT1hQnJlemhu ZXYKI2Jhc2U9NTU1NTU1CiNiYXNlPW9tcgpwc3Q9JGJhc2UucHN0CmRpcj0kYmFzZQp1cmw9aHR0 cHM6Ly9maWxlcy5lbmxhY2VoYWNrdGl2aXN0YS5vcmcvUHJvbmljbyUyMG1haWwvCgppZiBbICEg LWYgIiRwc3QiIF0gOyB0aGVuCgl3Z2V0ICIkdXJsJHBzdCIKZmkKCmlmIFsgISAtZCAiJGxpYnBz dCIgXSA7IHRoZW4KCWdpdCBjbG9uZSBodHRwczovL2dpdGh1Yi5jb20vcHN0LWZvcm1hdC9saWJw c3QuZ2l0ICIkbGlicHN0IgpmaQoKZXhwb3J0IFBBVEg9IiRvcHQvYmluOi91c3IvbGliL2NjYWNo ZTokUEFUSCIKCmNsZWFuICgpIHsKCXJtIC1yZiAkZGlyKi8gIiRvcHQvIgoJcHVzaGQgIiRsaWJw c3QiCglnaXQgY2xlYW4gLWR4ZgoJZ2l0IHJlc2V0IC0taGFyZCBACglwb3BkCn0KCmNsZWFuCnJt IC1yZiAiJGRlYnVnLyRkaXN0cm8iKgoKcHVzaGQgIiRsaWJwc3QiCmF1dG9yZWNvbmYgLXNpZgou L2NvbmZpZ3VyZSAtLXByZWZpeCAiJG9wdCIgLS1kaXNhYmxlLXB5dGhvbgpybSAtcmYgaHRtbC5p bnRlcm5hbApkb3h5Z2VuCnRhciBjZnogbGlicHN0Lmh0bWwudGFyLmd6IGh0bWwuaW50ZXJuYWwK bWFrZSAtQyB4bWwgLWoxMAptYWtlIC1qMTAKbWFrZSAtajEwIGluc3RhbGwKcG9wZAoKbWtkaXIg LXAgIiRkZWJ1Zy8iCgpyZWFkcHN0PSd0aW1lIHJlYWRwc3QnCgojJHJlYWRwc3QgLWQgIiRkZWJ1 Zy8kZGlzdHJvLWowLmxvZyIgLWogMCAtRCAtTSAtYiAiJHBzdCIKIyRyZWFkcHN0IC1kICIkZGVi dWcvJGRpc3Ryby1qMS5sb2ciIC1qIDEgLUQgLU0gLWIgIiRwc3QiCiMkcmVhZHBzdCAtZCAiJGRl YnVnLyRkaXN0cm8tajIubG9nIiAtaiAyIC1EIC1NIC1iICIkcHN0IgojJHJlYWRwc3QgLWQgIiRk ZWJ1Zy8kZGlzdHJvLWRlZmF1bHQubG9nIiAtRCAtTSAtYiAiJHBzdCIKCiNybSAtZiAiJGRlYnVn LyRkaXN0cm8tajAubG9nIioKCiRyZWFkcHN0IC1qIDAgLUQgLU0gLWIgIiRwc3QiCiMkcmVhZHBz dCAtaiAxIC1EIC1NIC1iICIkcHN0IgokcmVhZHBzdCAtaiAyIC1EIC1NIC1iICIkcHN0IgojJHJl YWRwc3QgLUQgLU0gLWIgIiRwc3QiCgpkaWZmIC1OYXVyIDwoY2QgIiRkaXIiIDsgZmluZCB8IHNv cnQpIDwoY2QgIiRkaXIiMSA7IGZpbmQgfCBzb3J0KSB8fCB0cnVlCgpjbGVhbgo= --=-IWXCpHr2v42yWBsXP/IE-- --=-64OxoEgE5N3bn/8w38h1 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- iQIzBAABCgAdFiEEYQsotVz8/kXqG1Y7MRa6Xp/6aaMFAmN9f00ACgkQMRa6Xp/6 aaPtsA//WacSrpGm/Te+QamGVnUEi4EE3QybzrXBzYgXcgJxSGPNOZcIdIz5A+sv 3V+tItZIBuDEfRhlD6siuWCv5K0aSYzT4z22O1ZBA4/MOL+h17rTgHYwS61CD4pk uqSWz6YCQYMh8iI1DeOaDjygWswTHY9YaUpjXEAGQLF0y0Z+VZg/paRoLKmid3i4 gSD15ifoMvruueZk1LrQsRRqQLTXJwSnwxrKukOeCAIhXB7BiEp5K3PionH8s/Zy c9mXfKqGB27ZFqRW2YyzUxn9dKV1duPRHsMHV2A621Ogv+djtKJ1KFWGxauqZPII pSvuQl6TwMca2yI1HIIhd4SSY3ybbMOF3Gfi7xbY8tusDKc7bxvFKyh7Ec8nMQ0O qqjlmlr2K6E/35ed0sZ4ms5Vtex8qOvlP0wZ9+Cariw2ayQlAw/VeP3svzkjqA4W TLBp7MfKlKgT2wUXJ0Sx4UoirnLRVbBW5pS1FmeTvTk6wEGXJ025DuUxeqjH3+ns Bdv/ns9jaSw1Yw8moOgNjSpGZOT2R3+y5kXkyP+C9CLas3Qv0/Nkryi/9HPUYesf 2UaStMbd2lMgsfpsdTIMrYTvijLTVxSJuY00ROcBS8RFu4Loxn0RrLMiRRdg7jXs FLQQfKj40igLj0yIwZkHfEnJHGixg/8g/wX5LSlkARF45AIfdD8= =mH4z -----END PGP SIGNATURE----- --=-64OxoEgE5N3bn/8w38h1--