From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mta-p5.oit.umn.edu (mta-p5.oit.umn.edu [134.84.196.205]) by sourceware.org (Postfix) with ESMTPS id 8EF85385BF83 for ; Sun, 12 Apr 2020 22:01:25 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 8EF85385BF83 Received: from localhost (unknown [127.0.0.1]) by mta-p5.oit.umn.edu (Postfix) with ESMTP id 490m106dt2z9vYQf for ; Sun, 12 Apr 2020 22:01:24 +0000 (UTC) X-Virus-Scanned: amavisd-new at umn.edu Received: from mta-p5.oit.umn.edu ([127.0.0.1]) by localhost (mta-p5.oit.umn.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id wpIcAcZCB89j for ; Sun, 12 Apr 2020 17:01:24 -0500 (CDT) Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mta-p5.oit.umn.edu (Postfix) with ESMTPS id 490m105LY9z9vYQd for ; Sun, 12 Apr 2020 17:01:24 -0500 (CDT) Received: by mail-qk1-f199.google.com with SMTP id r64so6993190qkc.17 for ; Sun, 12 Apr 2020 15:01:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:content-transfer-encoding:from:mime-version :subject:date:message-id:references:in-reply-to:to; bh=xg3FyZXuhC9bgqmn1PH3ies2tkt/eQmsVM09ieDtDIw=; b=RvEFaJj6GwUWyAOX6VdlE/T8UuihvehycnhmBk/OF6un+VBhplADAKaOLLElcFcafv PHLMjAhjz3RREC6sGp4p79J2KV8DWAVqNJhSyo7Bru2KfU2NhJaIdVFiNTh1j5Rcn9TH vEXMzhtwCwmdYOtljz7A4ZN49IhIWpxzQRnc59Z8orBBgD3YOkwe+DPFxAu3JSBqKPU1 0uuNh/xT/a6JUVDe49HYUmrlW+poROmRkXI4vPZZEP6I+dpPSPVYz3/Af4L2Iy/WCRrJ K3SeDuZbbOiTW3fDgbpxeJQY8a73YYOBA19yFDuoUcE5AlQq+M0NrV8E98Cyp6RR4yIN 1sHw== X-Gm-Message-State: AGi0PubnGIssODuE8MMuFfwu8slMuVFd6D9brm4eSEXotXsm0lO2FQjo 901pgEIttNqdolHlQXaNuDhZVQF2Kv+/3nrEZSfe5nPVk/IxcKtr4xoUI4cq2lcmA1PNnKbl9QV xCU4HyfZxcMBxjr2eZdX9XmX41NQaLFkJOyuC7267rWH1iIKGoa+0KzGDJ97wmHS1RA== X-Received: by 2002:a05:620a:b83:: with SMTP id k3mr2024004qkh.412.1586728883487; Sun, 12 Apr 2020 15:01:23 -0700 (PDT) X-Google-Smtp-Source: APiQypLVadmYtZ8T+Z25N+SapJxKWUf9z12V8nUO8Ih3qGo2yhFcUrqqp5sFbtoqiLpEgXRO3nlOwQ== X-Received: by 2002:a05:620a:b83:: with SMTP id k3mr2023973qkh.412.1586728883112; Sun, 12 Apr 2020 15:01:23 -0700 (PDT) Received: from ?IPv6:2601:449:8301:520e:1455:15e9:520b:1a98? ([2601:449:8301:520e:1455:15e9:520b:1a98]) by smtp.gmail.com with ESMTPSA id w189sm7057218qka.99.2020.04.12.15.01.22 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 12 Apr 2020 15:01:22 -0700 (PDT) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable From: "relliott@umn.edu" Mime-Version: 1.0 (1.0) Subject: =?utf-8?Q?Re:_High_memory_usage_compiling_large_=E2=80=98xxd_-i?= =?utf-8?Q?=E2=80=99_output?= Date: Sun, 12 Apr 2020 17:01:21 -0500 Message-Id: <448F1DF6-A77F-43E1-80EA-3C6B99B9F185@umn.edu> References: <149f3853f0a495a5f993d38ad60567fa6d10a745.camel@mengyan1223.wang> In-Reply-To: <149f3853f0a495a5f993d38ad60567fa6d10a745.camel@mengyan1223.wang> To: gcc-help X-Mailer: iPad Mail (17E262) X-Spam-Status: No, score=-6.5 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_2, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: gcc-help@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-help mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Apr 2020 22:01:27 -0000 Hello, Thanks for the explanation; I can see the issue now. Ryan > On Apr 11, 2020, at 9:16 PM, Xi Ruoyao wrote: >=20 > =EF=BB=BFOn 2020-04-11 16:31 -0500, relliott--- via Gcc-help wrote: >> Hello, >>=20 >> Thanks for your reply. I guess I don=E2=80=99t understand why this requi= res so much >> memory all at once. Is it not possible to write to the object file as th= e >> initializer is parsed? >=20 > A compiler doesn't write to object file. It only writes to assembly file.= =20 > Modern compilers are designed multi-layer so the parser (frontend) should n= ot > output assembly (it's the job of the backend). And, doing that so will mi= ss > optimizations. A simple code: >=20 > const int b[] =3D {0, 1, 2}; >=20 > int foo(void) > { > int i, x =3D 0; > for (i =3D 0; i < 3; i++) > x +=3D b[i]; > return x; > } >=20 > The body of "foo" will be optimized to "return 3;" at -O2. If we wrote th= e > array content to assembly file and discarded the value in memory, this > optimization will be impossible. >=20 > "xxd" a binary file and then compile the result with a compiler is just li= ke > "converting a .jpg to .png then back to .jpg". A smarter way: >=20 > ld -r -b binary some_big_binary_blob.bin -o some_big_binary_blob.o >=20 > There will be symbols _binary_some_big_binary_blob_bin_start, > _binary_some_big_binary_blob_end in some_big_binary_blob.o. Then it's pos= sible > to access the content of the binary blob with: >=20 > extern const char _binary_some_big_binary_blob_bin_start; > extern const char _binary_some_big_binary_blob_bin_end; >=20 > int main(void) > { > const char *begin =3D &_binary_some_big_binary_blob_bin_start; > const char *end =3D &_binary_some_big_binary_blob_bin_end; > const char *p; > for (p =3D begin; p !=3D end; p++) > play_with(*p); > } > --=20 > Xi Ruoyao > School of Aerospace Science and Technology, Xidian University >=20