From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pj1-x102a.google.com (mail-pj1-x102a.google.com [IPv6:2607:f8b0:4864:20::102a]) by sourceware.org (Postfix) with ESMTPS id 3CAAC386FC1F for ; Sat, 29 May 2021 10:08:18 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 3CAAC386FC1F Received: by mail-pj1-x102a.google.com with SMTP id kr9so3918854pjb.5 for ; Sat, 29 May 2021 03:08:18 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=XAl2Z15tERop8BhnYx7znjJuIFWeCQBXm9WRl8bW254=; b=nPcGNSi08rw626q3P/MzpCRP7TmhFp1FR4f41GFFjDysdZwgoU0t0nOUn0l+WJh6Ts yxC6hlhBDPZyX7oLaIwSQBI2D/t8hwxFkM1+rQ/S/+rkW6J1r2S0ZWl9nWu3x6qEUIKf UpQM/viL0uQu9J+u1ndsrYZBbzLsKMlOJbgpLDuh2TzQa1t0Ln2/FV2kXrRZmleCamVI 3nFJ8ZKiljiMBMV8Osqz9lULHVUDpiQelod4QRuMyPT5qLbnW7gulj5VHIT6pNwEfdWd KtaMcIUFkeTgiK04nOVrIKWpyl9zMkN8RoHAAvGeuslxnJKYwFnPVySkdU8zUZQJv68Z HW7A== X-Gm-Message-State: AOAM532t4o5+On2fY/Ik1DoxqXIO6v4pJdhY0lbM4uUkPuI4qyKsHv6d NAZyl4gL1INFMrjsJ/JFvkE= X-Google-Smtp-Source: ABdhPJzTl4wy0R/MXJZmKT5IuqT54ymzUldQugTUpKkUagmTcL7UqXantVOY7d8BsKfH0mpLk698Sg== X-Received: by 2002:a17:902:7787:b029:f0:a7c0:f9e5 with SMTP id o7-20020a1709027787b02900f0a7c0f9e5mr11831291pll.5.1622282897259; Sat, 29 May 2021 03:08:17 -0700 (PDT) Received: from localhost (g176.222-224-212.ppp.wakwak.ne.jp. [222.224.212.176]) by smtp.gmail.com with ESMTPSA id l192sm6726321pfd.173.2021.05.29.03.08.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 29 May 2021 03:08:16 -0700 (PDT) Date: Sat, 29 May 2021 19:08:14 +0900 From: Stafford Horne To: Adhemerval Zanella Cc: GLIBC patches Subject: Re: GLIBC Localedata out of memory Message-ID: References: <20210130235020.GE2002709@lianli.shorne-pla.net> <36a492ba-facf-0745-b6f5-772e1a45f51f@linaro.org> <20210202214034.GJ2002709@lianli.shorne-pla.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20210202214034.GJ2002709@lianli.shorne-pla.net> X-Spam-Status: No, score=-3.4 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 May 2021 10:08:20 -0000 On Wed, Feb 03, 2021 at 06:40:34AM +0900, Stafford Horne wrote: > On Tue, Feb 02, 2021 at 10:40:02AM -0300, Adhemerval Zanella wrote: > > > > > > On 30/01/2021 20:50, Stafford Horne via Libc-alpha wrote: > > > Hello, > > > > > > I am working on testing the new OpenRISC port and getting all of the tests > > > passing. I am having an issue with generating localedata on my board running > > > out of memory. The process is getting killed by the linux oom-killer. The > > > board has 256mb ram and it seems to not be enough. Lot's of ram is being used > > > by my rootfs. > > > > I usually ran a full make check on a sh4 board that has 512MB of ram and it is > > shared with debian buildd process and I haven't see any issue. The machine does > > have 8GBi of swap enabled, so it might be mitigated by this. > > > > I checked the memory requirements of localedata generation (using valgrind massig) > > for tests and it seems that only one (cmw_TW.UTF8) consumes about ~190MB, mostly > > uses about ~150MB (as below). So I think using a small swap should help you here. > > > > az_AZ.UTF-8 : 143.4 MB > > bg_BG.UTF-8 : 145.29 MB > > am_ET.UTF-8 : 145.62 MB > > ber_DZ.UTF-8 : 145.28 MB > > ber_MA.UTF-8 : 145.26 MB > > be_BY.UTF-8 : 145.27 MB > > br_FR.UTF-8 : 145.26 MB > > bs_BA.UTF-8 : 145.27 MB > > ckb_IQ.UTF-8 : 144.6 MB > > cmn_TW.UTF-8 : 190.72 MB > > Actually, I was able to get the locale generation working yesterday by shrinking > my rootfs as much as possible. Next I found that the sort-test is failing > (localedata/xfrm-test with -nocache). The -nocache option makes the program > allocate 4096 byte per each locale line to avoid strxfrm from using a cache > path. Oh boy! > > > > > > > Previously I was running tests on a simulator where I could adjust the ram to > > > anything I like. But now I am running tests on my ARTY fpga dev board. > > > > > > Is it OK to skip locale generation in tests results? Or any suggestions? > > > > You will need the locales to get a full result, otherwise some tests will > > fail. I think it should be possible to build a native localedef and > > generate the locales in the host for a cross-compilation. You will need to > > handle the possible endianess mismatch though and there is the potential > > issue of a conversion library not support by the host that test might use. > > I think that I'm going to need to solve the ram issue by giving it some swap. > > I tried swap over NFS but that resulted in write errors (Write error on dio > swapfile): > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/mm/page_io.c?h=v5.11-rc6#n351 > > I will try to use an SD card. So, this took a bit longer than I expected but I have swap working now over SD card and I don't have a memory issue any more generating locale data. Interesting chain of events: - Had to order SD card pmod for my board which took a few days - Next, I found my SoC had broken hardware (ok its FPGA, but it took about a month to fix.) - After fixing we found some issues with the kernel driver, but now those are fixed and everyting works fine. - My workstation died so I had to rebuild it, now I have a fancy 6 core processor. Anyway, another question. Do most glibc embedded developers build and run tests on embedded boards using a cross-compiler and 'scripts/cross-test-ssh.sh'? I have been running tests like this using a buildroot rootfs on my board. I now have 58 failures, but looking into them some seem related to handling of stdout and it seems like maybe its related to running over SSH. If this is what other are doing then I will continue with SSH and figure out these issues. But if other do another way, i.e. with a native compiler on the board then I will look into that. In that case I will probably have to start building debian for fedora as buildroot doesn't allow for creating a native compiler. -Stafford