From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mengyan1223.wang (mengyan1223.wang [89.208.246.23]) by sourceware.org (Postfix) with ESMTPS id A7B82385DC22 for ; Sun, 12 Apr 2020 02:08:09 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org A7B82385DC22 Received: from xry111-X57S1 (unknown [183.202.3.51]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: xry111@mengyan1223.wang) by mengyan1223.wang (Postfix) with ESMTPSA id D647C662CB; Sat, 11 Apr 2020 22:07:56 -0400 (EDT) Message-ID: <149f3853f0a495a5f993d38ad60567fa6d10a745.camel@mengyan1223.wang> Subject: Re: High memory usage compiling large =?gb2312?Q?=A1=AExxd?= =?gb2312?Q?_-i=A1=AF?= output From: Xi Ruoyao Reply-To: gcc-help To: "relliott@umn.edu" Cc: gcc-help Date: Sun, 12 Apr 2020 10:07:52 +0800 In-Reply-To: <3F1B5DA6-7224-4431-8070-123D18B69B3C@umn.edu> References: <3F1B5DA6-7224-4431-8070-123D18B69B3C@umn.edu> Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.36.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-2.3 required=5.0 tests=BAYES_05, CHARSET_FARAWAY_HEADER, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_2, JMQ_SPF_NEUTRAL, SPF_HELO_PASS, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: gcc-help@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-help mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Apr 2020 02:08:11 -0000 On 2020-04-11 16:31 -0500, relliott--- via Gcc-help wrote: > Hello, > > Thanks for your reply. I guess I don’t understand why this requires so much > memory all at once. Is it not possible to write to the object file as the > initializer is parsed? A compiler doesn't write to object file. It only writes to assembly file. Modern compilers are designed multi-layer so the parser (frontend) should not output assembly (it's the job of the backend). And, doing that so will miss optimizations. A simple code: const int b[] = {0, 1, 2}; int foo(void) { int i, x = 0; for (i = 0; i < 3; i++) x += b[i]; return x; } The body of "foo" will be optimized to "return 3;" at -O2. If we wrote the array content to assembly file and discarded the value in memory, this optimization will be impossible. "xxd" a binary file and then compile the result with a compiler is just like "converting a .jpg to .png then back to .jpg". A smarter way: ld -r -b binary some_big_binary_blob.bin -o some_big_binary_blob.o There will be symbols _binary_some_big_binary_blob_bin_start, _binary_some_big_binary_blob_end in some_big_binary_blob.o. Then it's possible to access the content of the binary blob with: extern const char _binary_some_big_binary_blob_bin_start; extern const char _binary_some_big_binary_blob_bin_end; int main(void) { const char *begin = &_binary_some_big_binary_blob_bin_start; const char *end = &_binary_some_big_binary_blob_bin_end; const char *p; for (p = begin; p != end; p++) play_with(*p); } -- Xi Ruoyao School of Aerospace Science and Technology, Xidian University