GLIBC Localedata out of memory

GLIBC Localedata out of memory

[Stafford Horne]

Hello,

I am working on testing the new OpenRISC port and getting all of the tests
passing.  I am having an issue with generating localedata on my board running
out of memory.  The process is getting killed by the linux oom-killer.  The
board has 256mb ram and it seems to not be enough.  Lot's of ram is being used
by my rootfs.

Previously I was running tests on a simulator where I could adjust the ram to
anything I like.  But now I am running tests on my ARTY fpga dev board.

Is it OK to skip locale generation in tests results?  Or any suggestions?

-Stafford

Re: GLIBC Localedata out of memory

[Stafford Horne]

On Sun, Jan 31, 2021 at 08:50:20AM +0900, Stafford Horne wrote:
> Hello,
> 
> I am working on testing the new OpenRISC port and getting all of the tests
> passing.  I am having an issue with generating localedata on my board running
> out of memory.  The process is getting killed by the linux oom-killer.  The
> board has 256mb ram and it seems to not be enough.  Lot's of ram is being used
> by my rootfs.
> 
> Previously I was running tests on a simulator where I could adjust the ram to
> anything I like.  But now I am running tests on my ARTY fpga dev board.
> 
> Is it OK to skip locale generation in tests results?  Or any suggestions?

I am thinking swap of NFS.  I'll give it a try.  Also, I can move more of my
rootfs (python, etc) over NFS too to free up memory.

Just some initial idea's, I would like to get some tips of how other people run
the glibc test suite on embedded systems.  Do most developers just use
simulators?

-Staford

Re: GLIBC Localedata out of memory

[Adhemerval Zanella]

On 30/01/2021 20:50, Stafford Horne via Libc-alpha wrote:
> Hello,
> 
> I am working on testing the new OpenRISC port and getting all of the tests
> passing.  I am having an issue with generating localedata on my board running
> out of memory.  The process is getting killed by the linux oom-killer.  The
> board has 256mb ram and it seems to not be enough.  Lot's of ram is being used
> by my rootfs.

I usually ran a full make check on a sh4 board that has 512MB of ram and it is
shared with debian buildd process and I haven't see any issue.  The machine does
have 8GBi of swap enabled, so it might be mitigated by this.

I checked the memory requirements of localedata generation (using valgrind massig)
for tests and it seems that only one (cmw_TW.UTF8) consumes about ~190MB, mostly
uses about ~150MB (as below).  So I think using a small swap should help you here.

az_AZ.UTF-8         : 143.4 MB
bg_BG.UTF-8         : 145.29 MB
am_ET.UTF-8         : 145.62 MB
ber_DZ.UTF-8        : 145.28 MB
ber_MA.UTF-8        : 145.26 MB
be_BY.UTF-8         : 145.27 MB
br_FR.UTF-8         : 145.26 MB
bs_BA.UTF-8         : 145.27 MB
ckb_IQ.UTF-8        : 144.6 MB
cmn_TW.UTF-8        : 190.72 MB
crh_UA.UTF-8        : 143.39 MB
cs_CZ.UTF-8         : 145.24 MB
csb_PL.UTF-8        : 145.28 MB
cv_RU.UTF-8         : 145.26 MB
cy_GB.UTF-8         : 145.3 MB
da_DK.ISO-8859-1    : 68.19 MB
de_DE.ISO-8859-1    : 68.36 MB
de_DE.UTF-8         : 145.26 MB
dsb_DE.UTF-8        : 144.62 MB
dz_BT.UTF-8         : 145.98 MB
en_GB.UTF-8         : 145.26 MB
en_US.ANSI_X3.4-1968: 68.3 MB
en_US.ISO-8859-1    : 68.36 MB
en_US.UTF-8         : 145.26 MB
eo.UTF-8            : 144.61 MB
es_ES.UTF-8         : 145.26 MB
et_EE.UTF-8         : 145.27 MB
fa_IR.UTF-8         : 145.37 MB
fi_FI.UTF-8         : 145.27 MB
fil_PH.UTF-8        : 145.27 MB
fr_CA.UTF-8         : 145.26 MB
fr_FR.ISO-8859-1    : 68.36 MB
fr_FR.UTF-8         : 145.26 MB
fur_IT.UTF-8        : 145.26 MB
gez_ER.UTF-8@abegede: 145.69 MB
ha_NG.UTF-8         : 145.28 MB
hr_HR.ISO-8859-2    : 68.38 MB
hr_HR.UTF-8         : 145.27 MB
hsb_DE.UTF-8        : 144.62 MB
hu_HU.UTF-8         : 145.38 MB
ig_NG.UTF-8         : 145.31 MB
ik_CA.UTF-8         : 145.29 MB
is_IS.UTF-8         : 145.27 MB
ja_JP.EUC-JP        : 34.47 MB
ja_JP.SJIS          : 31.63 MB
ja_JP.UTF-8         : 96.93 MB
kk_KZ.UTF-8         : 145.26 MB
ku_TR.UTF-8         : 143.4 MB
ky_KG.UTF-8         : 145.26 MB
ln_CD.UTF-8         : 145.25 MB
lt_LT.UTF-8         : 145.28 MB
lv_LV.UTF-8         : 145.26 MB
mi_NZ.UTF-8         : 145.26 MB
ml_IN.UTF-8         : 145.31 MB
mn_MN.UTF-8         : 145.29 MB
mr_IN.UTF-8         : 145.3 MB
mt_MT.UTF-8         : 145.26 MB
nan_TW.UTF-8@latin  : 145.26 MB
nb_NO.ISO-8859-1    : 68.19 MB
nb_NO.UTF-8         : 145.28 MB
nl_NL.UTF-8         : 145.26 MB
nn_NO.ISO-8859-1    : 68.39 MB
om_KE.UTF-8         : 145.28 MB
or_IN.UTF-8         : 145.33 MB
os_RU.UTF-8         : 145.26 MB
pl_PL.UTF-8         : 145.28 MB
ps_AF.UTF-8         : 144.66 MB
ro_RO.UTF-8         : 145.26 MB
ru_RU.UTF-8         : 145.26 MB
sah_RU.UTF-8        : 145.28 MB
sc_IT.UTF-8         : 145.26 MB
se_NO.UTF-8         : 145.28 MB
si_LK.UTF-8         : 145.26 MB
sq_AL.UTF-8         : 145.3 MB
sr_RS.UTF-8         : 145.31 MB
sv_SE.ISO-8859-1    : 68.2 MB
sv_SE.UTF-8         : 145.27 MB
szl_PL.UTF-8        : 145.28 MB
tg_TJ.UTF-8         : 145.27 MB
tk_TM.UTF-8         : 145.31 MB
tr_TR.ISO-8859-9    : 66.72 MB
tr_TR.UTF-8         : 143.4 MB
tt_RU.UTF-8         : 145.27 MB
tt_RU.UTF-8@iqtelif : 143.4 MB
ug_CN.UTF-8         : 144.6 MB
uk_UA.UTF-8         : 144.67 MB
uz_UZ.UTF-8         : 145.31 MB
vi_VN.UTF-8         : 145.26 MB
yi_US.UTF-8         : 145.29 MB
yo_NG.UTF-8         : 145.28 MB
zh_CN.UTF-8         : 154.75 MB
zh_TW.EUC-TW        : 94.77 MB

> 
> Previously I was running tests on a simulator where I could adjust the ram to
> anything I like.  But now I am running tests on my ARTY fpga dev board.
> 
> Is it OK to skip locale generation in tests results?  Or any suggestions?

You will need the locales to get a full result, otherwise some tests will
fail.  I think it should be possible to build a native localedef and
generate the locales in the host for a cross-compilation. You will need to
handle the possible endianess mismatch though and there is the potential
issue of a conversion library not support by the host that test might use.

Re: GLIBC Localedata out of memory

[Stafford Horne]

On Tue, Feb 02, 2021 at 10:40:02AM -0300, Adhemerval Zanella wrote:
> 
> 
> On 30/01/2021 20:50, Stafford Horne via Libc-alpha wrote:
> > Hello,
> > 
> > I am working on testing the new OpenRISC port and getting all of the tests
> > passing.  I am having an issue with generating localedata on my board running
> > out of memory.  The process is getting killed by the linux oom-killer.  The
> > board has 256mb ram and it seems to not be enough.  Lot's of ram is being used
> > by my rootfs.
> 
> I usually ran a full make check on a sh4 board that has 512MB of ram and it is
> shared with debian buildd process and I haven't see any issue.  The machine does
> have 8GBi of swap enabled, so it might be mitigated by this.
> 
> I checked the memory requirements of localedata generation (using valgrind massig)
> for tests and it seems that only one (cmw_TW.UTF8) consumes about ~190MB, mostly
> uses about ~150MB (as below).  So I think using a small swap should help you here.
> 
> az_AZ.UTF-8         : 143.4 MB
> bg_BG.UTF-8         : 145.29 MB
> am_ET.UTF-8         : 145.62 MB
> ber_DZ.UTF-8        : 145.28 MB
> ber_MA.UTF-8        : 145.26 MB
> be_BY.UTF-8         : 145.27 MB
> br_FR.UTF-8         : 145.26 MB
> bs_BA.UTF-8         : 145.27 MB
> ckb_IQ.UTF-8        : 144.6 MB
> cmn_TW.UTF-8        : 190.72 MB

Actually, I was able to get the locale generation working yesterday by shrinking
my rootfs as much as possible.  Next I found that the sort-test is failing
(localedata/xfrm-test with -nocache).  The -nocache option makes the program
allocate 4096 byte per each locale line to avoid strxfrm from using a cache
path.  Oh boy!

> > 
> > Previously I was running tests on a simulator where I could adjust the ram to
> > anything I like.  But now I am running tests on my ARTY fpga dev board.
> > 
> > Is it OK to skip locale generation in tests results?  Or any suggestions?
> 
> You will need the locales to get a full result, otherwise some tests will
> fail.  I think it should be possible to build a native localedef and
> generate the locales in the host for a cross-compilation. You will need to
> handle the possible endianess mismatch though and there is the potential
> issue of a conversion library not support by the host that test might use.

I think that I'm going to need to solve the ram issue by giving it some swap.

I tried swap over NFS but that resulted in write errors (Write error on dio
swapfile):
  https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/mm/page_io.c?h=v5.11-rc6#n351

I will try to use an SD card.

-Stafford

Re: GLIBC Localedata out of memory

[Stafford Horne]

On Wed, Feb 03, 2021 at 06:40:34AM +0900, Stafford Horne wrote:
> On Tue, Feb 02, 2021 at 10:40:02AM -0300, Adhemerval Zanella wrote:
> > 
> > 
> > On 30/01/2021 20:50, Stafford Horne via Libc-alpha wrote:
> > > Hello,
> > > 
> > > I am working on testing the new OpenRISC port and getting all of the tests
> > > passing.  I am having an issue with generating localedata on my board running
> > > out of memory.  The process is getting killed by the linux oom-killer.  The
> > > board has 256mb ram and it seems to not be enough.  Lot's of ram is being used
> > > by my rootfs.
> > 
> > I usually ran a full make check on a sh4 board that has 512MB of ram and it is
> > shared with debian buildd process and I haven't see any issue.  The machine does
> > have 8GBi of swap enabled, so it might be mitigated by this.
> > 
> > I checked the memory requirements of localedata generation (using valgrind massig)
> > for tests and it seems that only one (cmw_TW.UTF8) consumes about ~190MB, mostly
> > uses about ~150MB (as below).  So I think using a small swap should help you here.
> > 
> > az_AZ.UTF-8         : 143.4 MB
> > bg_BG.UTF-8         : 145.29 MB
> > am_ET.UTF-8         : 145.62 MB
> > ber_DZ.UTF-8        : 145.28 MB
> > ber_MA.UTF-8        : 145.26 MB
> > be_BY.UTF-8         : 145.27 MB
> > br_FR.UTF-8         : 145.26 MB
> > bs_BA.UTF-8         : 145.27 MB
> > ckb_IQ.UTF-8        : 144.6 MB
> > cmn_TW.UTF-8        : 190.72 MB
> 
> Actually, I was able to get the locale generation working yesterday by shrinking
> my rootfs as much as possible.  Next I found that the sort-test is failing
> (localedata/xfrm-test with -nocache).  The -nocache option makes the program
> allocate 4096 byte per each locale line to avoid strxfrm from using a cache
> path.  Oh boy!
> 
> > > 
> > > Previously I was running tests on a simulator where I could adjust the ram to
> > > anything I like.  But now I am running tests on my ARTY fpga dev board.
> > > 
> > > Is it OK to skip locale generation in tests results?  Or any suggestions?
> > 
> > You will need the locales to get a full result, otherwise some tests will
> > fail.  I think it should be possible to build a native localedef and
> > generate the locales in the host for a cross-compilation. You will need to
> > handle the possible endianess mismatch though and there is the potential
> > issue of a conversion library not support by the host that test might use.
> 
> I think that I'm going to need to solve the ram issue by giving it some swap.
> 
> I tried swap over NFS but that resulted in write errors (Write error on dio
> swapfile):
>   https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/mm/page_io.c?h=v5.11-rc6#n351
> 
> I will try to use an SD card.

So, this took a bit longer than I expected but I have swap working now over SD
card and I don't have a memory issue any more generating locale data.

Interesting chain of events:
  - Had to order SD card pmod for my board which took a few days
  - Next, I found my SoC had broken hardware (ok its FPGA, but it took about
    a month to fix.)
  - After fixing we found some issues with the kernel driver, but now those
    are fixed and everyting works fine.
  - My workstation died so I had to rebuild it, now I have a fancy 6 core
    processor.

Anyway, another question.  Do most glibc embedded developers build and run tests
on embedded boards using a cross-compiler and 'scripts/cross-test-ssh.sh'?

I have been running tests like this using a buildroot rootfs on my board.

I now have 58 failures, but looking into them some seem related to handling of
stdout and it seems like maybe its related to running over SSH.

If this is what other are doing then I will continue with SSH and figure out
these issues.  But if other do another way, i.e. with a native compiler on
the board then I will look into that.  In that case I will probably have to
start building debian for fedora as buildroot doesn't allow for creating
a native compiler.

-Stafford

Re: GLIBC Localedata out of memory

[Adhemerval Zanella]

On 29/05/2021 07:08, Stafford Horne wrote:
> On Wed, Feb 03, 2021 at 06:40:34AM +0900, Stafford Horne wrote:
>> On Tue, Feb 02, 2021 at 10:40:02AM -0300, Adhemerval Zanella wrote:
>>>
>>>
>>> On 30/01/2021 20:50, Stafford Horne via Libc-alpha wrote:
>>>> Hello,
>>>>
>>>> I am working on testing the new OpenRISC port and getting all of the tests
>>>> passing.  I am having an issue with generating localedata on my board running
>>>> out of memory.  The process is getting killed by the linux oom-killer.  The
>>>> board has 256mb ram and it seems to not be enough.  Lot's of ram is being used
>>>> by my rootfs.
>>>
>>> I usually ran a full make check on a sh4 board that has 512MB of ram and it is
>>> shared with debian buildd process and I haven't see any issue.  The machine does
>>> have 8GBi of swap enabled, so it might be mitigated by this.
>>>
>>> I checked the memory requirements of localedata generation (using valgrind massig)
>>> for tests and it seems that only one (cmw_TW.UTF8) consumes about ~190MB, mostly
>>> uses about ~150MB (as below).  So I think using a small swap should help you here.
>>>
>>> az_AZ.UTF-8         : 143.4 MB
>>> bg_BG.UTF-8         : 145.29 MB
>>> am_ET.UTF-8         : 145.62 MB
>>> ber_DZ.UTF-8        : 145.28 MB
>>> ber_MA.UTF-8        : 145.26 MB
>>> be_BY.UTF-8         : 145.27 MB
>>> br_FR.UTF-8         : 145.26 MB
>>> bs_BA.UTF-8         : 145.27 MB
>>> ckb_IQ.UTF-8        : 144.6 MB
>>> cmn_TW.UTF-8        : 190.72 MB
>>
>> Actually, I was able to get the locale generation working yesterday by shrinking
>> my rootfs as much as possible.  Next I found that the sort-test is failing
>> (localedata/xfrm-test with -nocache).  The -nocache option makes the program
>> allocate 4096 byte per each locale line to avoid strxfrm from using a cache
>> path.  Oh boy!
>>
>>>>
>>>> Previously I was running tests on a simulator where I could adjust the ram to
>>>> anything I like.  But now I am running tests on my ARTY fpga dev board.
>>>>
>>>> Is it OK to skip locale generation in tests results?  Or any suggestions?
>>>
>>> You will need the locales to get a full result, otherwise some tests will
>>> fail.  I think it should be possible to build a native localedef and
>>> generate the locales in the host for a cross-compilation. You will need to
>>> handle the possible endianess mismatch though and there is the potential
>>> issue of a conversion library not support by the host that test might use.
>>
>> I think that I'm going to need to solve the ram issue by giving it some swap.
>>
>> I tried swap over NFS but that resulted in write errors (Write error on dio
>> swapfile):
>>   https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/mm/page_io.c?h=v5.11-rc6#n351
>>
>> I will try to use an SD card.
> 
> So, this took a bit longer than I expected but I have swap working now over SD
> card and I don't have a memory issue any more generating locale data.
> 
> Interesting chain of events:
>   - Had to order SD card pmod for my board which took a few days
>   - Next, I found my SoC had broken hardware (ok its FPGA, but it took about
>     a month to fix.)
>   - After fixing we found some issues with the kernel driver, but now those
>     are fixed and everyting works fine.
>   - My workstation died so I had to rebuild it, now I have a fancy 6 core
>     processor.
> 
> Anyway, another question.  Do most glibc embedded developers build and run tests
> on embedded boards using a cross-compiler and 'scripts/cross-test-ssh.sh'?

I usually run it natively, but I think both recent ports for arc and riscv32 
were done using the scripts/cross-test-ssh.sh.

> 
> I have been running tests like this using a buildroot rootfs on my board.
> 
> I now have 58 failures, but looking into them some seem related to handling of
> stdout and it seems like maybe its related to running over SSH.
> 
> If this is what other are doing then I will continue with SSH and figure out
> these issues.  But if other do another way, i.e. with a native compiler on
> the board then I will look into that.  In that case I will probably have to
> start building debian for fedora as buildroot doesn't allow for creating
> a native compiler.

Usually the issues reported for bring up and newer architectures are:

  - Tests timeout.  A lot of tests are build and set to run on faster
    machine, so they might timeout on slower ones.  I would recommend
    to run the testsuite with TIMEOUTFACTOR environment variable set
    to 10 or something higher (it might extend the make check if the
    test is stuck due some failure).

  - Missing installed library, either libgcc_s.so or libstdc++.so.
    The former is required for stack unwind required for some pthread
    routines while the latter for testing.  I would recommend to
    copy both of them to the build directory prior ran the testsuite.

  - Issues with file permission or synchronization issues depending of
    the filesystem.  If I recall correctly NFS might not implement all
    the required facilities depending of the version used.  I am not
    sure how to mitigate it.

Re: GLIBC Localedata out of memory

[Joseph Myers]

On Tue, 1 Jun 2021, Adhemerval Zanella via Libc-alpha wrote:

>   - Issues with file permission or synchronization issues depending of
>     the filesystem.  If I recall correctly NFS might not implement all
>     the required facilities depending of the version used.  I am not
>     sure how to mitigate it.

Mounting with acdirmax=0,acdirmin=0 may help with some synchronization 
issues.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: GLIBC Localedata out of memory

[Stafford Horne]

On Tue, Jun 01, 2021 at 06:01:53PM +0000, Joseph Myers wrote:
> On Tue, 1 Jun 2021, Adhemerval Zanella via Libc-alpha wrote:
> 
> >   - Issues with file permission or synchronization issues depending of
> >     the filesystem.  If I recall correctly NFS might not implement all
> >     the required facilities depending of the version used.  I am not
> >     sure how to mitigate it.
> 
> Mounting with acdirmax=0,acdirmin=0 may help with some synchronization 
> issues.

Thanks, I think this is sorted out for me.  I haven't seen NFS sync related
issues.

Re: GLIBC Localedata out of memory

[Stafford Horne]

On Tue, Jun 01, 2021 at 02:45:47PM -0300, Adhemerval Zanella wrote:
> 
> 
> On 29/05/2021 07:08, Stafford Horne wrote:
> > On Wed, Feb 03, 2021 at 06:40:34AM +0900, Stafford Horne wrote:
> >> On Tue, Feb 02, 2021 at 10:40:02AM -0300, Adhemerval Zanella wrote:
> >>>
> >>>
> >>> On 30/01/2021 20:50, Stafford Horne via Libc-alpha wrote:
> >>>> Hello,
> >>>>
> >>>> I am working on testing the new OpenRISC port and getting all of the tests
> >>>> passing.  I am having an issue with generating localedata on my board running
> >>>> out of memory.  The process is getting killed by the linux oom-killer.  The
> >>>> board has 256mb ram and it seems to not be enough.  Lot's of ram is being used
> >>>> by my rootfs.
> >>>
> >>> I usually ran a full make check on a sh4 board that has 512MB of ram and it is
> >>> shared with debian buildd process and I haven't see any issue.  The machine does
> >>> have 8GBi of swap enabled, so it might be mitigated by this.
> >>>
> >>> I checked the memory requirements of localedata generation (using valgrind massig)
> >>> for tests and it seems that only one (cmw_TW.UTF8) consumes about ~190MB, mostly
> >>> uses about ~150MB (as below).  So I think using a small swap should help you here.
> >>>
> >>> az_AZ.UTF-8         : 143.4 MB
> >>> bg_BG.UTF-8         : 145.29 MB
> >>> am_ET.UTF-8         : 145.62 MB
> >>> ber_DZ.UTF-8        : 145.28 MB
> >>> ber_MA.UTF-8        : 145.26 MB
> >>> be_BY.UTF-8         : 145.27 MB
> >>> br_FR.UTF-8         : 145.26 MB
> >>> bs_BA.UTF-8         : 145.27 MB
> >>> ckb_IQ.UTF-8        : 144.6 MB
> >>> cmn_TW.UTF-8        : 190.72 MB
> >>
> >> Actually, I was able to get the locale generation working yesterday by shrinking
> >> my rootfs as much as possible.  Next I found that the sort-test is failing
> >> (localedata/xfrm-test with -nocache).  The -nocache option makes the program
> >> allocate 4096 byte per each locale line to avoid strxfrm from using a cache
> >> path.  Oh boy!
> >>
> >>>>
> >>>> Previously I was running tests on a simulator where I could adjust the ram to
> >>>> anything I like.  But now I am running tests on my ARTY fpga dev board.
> >>>>
> >>>> Is it OK to skip locale generation in tests results?  Or any suggestions?
> >>>
> >>> You will need the locales to get a full result, otherwise some tests will
> >>> fail.  I think it should be possible to build a native localedef and
> >>> generate the locales in the host for a cross-compilation. You will need to
> >>> handle the possible endianess mismatch though and there is the potential
> >>> issue of a conversion library not support by the host that test might use.
> >>
> >> I think that I'm going to need to solve the ram issue by giving it some swap.
> >>
> >> I tried swap over NFS but that resulted in write errors (Write error on dio
> >> swapfile):
> >>   https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/mm/page_io.c?h=v5.11-rc6#n351
> >>
> >> I will try to use an SD card.
> > 
> > So, this took a bit longer than I expected but I have swap working now over SD
> > card and I don't have a memory issue any more generating locale data.
> > 
> > Interesting chain of events:
> >   - Had to order SD card pmod for my board which took a few days
> >   - Next, I found my SoC had broken hardware (ok its FPGA, but it took about
> >     a month to fix.)
> >   - After fixing we found some issues with the kernel driver, but now those
> >     are fixed and everyting works fine.
> >   - My workstation died so I had to rebuild it, now I have a fancy 6 core
> >     processor.
> > 
> > Anyway, another question.  Do most glibc embedded developers build and run tests
> > on embedded boards using a cross-compiler and 'scripts/cross-test-ssh.sh'?
> 
> I usually run it natively, but I think both recent ports for arc and riscv32 
> were done using the scripts/cross-test-ssh.sh.

Thanks, its good to know.  I will update the wiki with these details:

 - https://sourceware.org/glibc/wiki/Testing/Testsuite#Testing_with_a_cross-compiler

> > 
> > I have been running tests like this using a buildroot rootfs on my board.
> > 
> > I now have 58 failures, but looking into them some seem related to handling of
> > stdout and it seems like maybe its related to running over SSH.
> > 
> > If this is what other are doing then I will continue with SSH and figure out
> > these issues.  But if other do another way, i.e. with a native compiler on
> > the board then I will look into that.  In that case I will probably have to
> > start building debian for fedora as buildroot doesn't allow for creating
> > a native compiler.
> 
> Usually the issues reported for bring up and newer architectures are:
> 
>   - Tests timeout.  A lot of tests are build and set to run on faster
>     machine, so they might timeout on slower ones.  I would recommend
>     to run the testsuite with TIMEOUTFACTOR environment variable set
>     to 10 or something higher (it might extend the make check if the
>     test is stuck due some failure).

I have sored this one, but it could be that some of the 58 failures are still
just timeouts.

>   - Missing installed library, either libgcc_s.so or libstdc++.so.
>     The former is required for stack unwind required for some pthread
>     routines while the latter for testing.  I would recommend to
>     copy both of them to the build directory prior ran the testsuite.

Yup, I have sorted this one out.

>   - Issues with file permission or synchronization issues depending of
>     the filesystem.  If I recall correctly NFS might not implement all
>     the required facilities depending of the version used.  I am not
>     sure how to mitigate it.

Thanks.

Thanks for the notes.