gdb moxie build failure

gdb moxie build failure

[Joel Sherrill]

Hi

gdb on the head has moxie-rtems fail like this:

echo "/* generated by Makefile */" > tmp-hw.h
/bin/sh ../../../binutils-gdb/sim/moxie/../common/create-version.sh
../../../binutils-gdb/sim/moxie/.. x86_64-unknown-linux-gnu
moxie-rtems4.11 version.c
sim_hw=""; \
for hw in $sim_hw ; do \
  echo "extern const struct hw_descriptor dv_${hw}_descriptor[];" ; \
done >> tmp-hw.h
dtc -O dtb -o moxie-gdb.dtb ../../../binutils-gdb/sim/moxie/moxie-gdb.dts
make[3]: dtc: Command not found
echo "const struct hw_descriptor *hw_descriptors[] = {" >> tmp-hw.h
make[3]: *** [moxie-gdb.dtb] Error 127
make[3]: *** Waiting for unfinished jobs....
sim_hw=""; \
for hw in $sim_hw ; do \

Is dtc a required tool that is not checked for? Is the output in the tree
but not timestamped? maintainer only tool?

Thanks.

-- 
Joel Sherrill, Ph.D.             Director of Research & Development
joel.sherrill@OARcorp.com        On-Line Applications Research
Ask me about RTEMS: a free RTOS  Huntsville AL 35805
Support Available                (256) 722-9985

integrating dtc into the sim/ tree

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 2483 bytes --]

On Mon 18 Aug 2014 17:14:52 Joel Sherrill wrote:
> gdb on the head has moxie-rtems fail like this:
> 
> echo "/* generated by Makefile */" > tmp-hw.h
> /bin/sh ../../../binutils-gdb/sim/moxie/../common/create-version.sh
> ../../../binutils-gdb/sim/moxie/.. x86_64-unknown-linux-gnu
> moxie-rtems4.11 version.c
> sim_hw=""; \
> for hw in $sim_hw ; do \
>   echo "extern const struct hw_descriptor dv_${hw}_descriptor[];" ; \
> done >> tmp-hw.h
> dtc -O dtb -o moxie-gdb.dtb ../../../binutils-gdb/sim/moxie/moxie-gdb.dts
> make[3]: dtc: Command not found
> echo "const struct hw_descriptor *hw_descriptors[] = {" >> tmp-hw.h
> make[3]: *** [moxie-gdb.dtb] Error 127
> make[3]: *** Waiting for unfinished jobs....
> sim_hw=""; \
> for hw in $sim_hw ; do \
> 
> Is dtc a required tool that is not checked for? Is the output in the tree
> but not timestamped? maintainer only tool?

yeah, you need dtc for moxie.  it's been this way forever ?  for people 
building from git, i think it's fine.  it also leads me to a topic i've been 
meaning to start for a while, so let's fork the thread.

what issues are there with making the sim use the existing dtc project ?  the 
dtc tool is licensed under the GPL-2 or later, and the libfdt library is dual 
licensed (BSD-2 & GPL-2 or later).
http://www.devicetree.org/Device_Tree_Compiler

the reason i ask is that the existing sim code uses device tree style logic 
heavily as it comes from the ppc sim import/commonization logic.  afaik, this 
codebase is somewhat a predecessor of the dtc codebase -- they both originated 
at IBM, and the ppc sim is dated mid-1990's while dtc is dated mid-2000.  i'm 
not saying they necessarily share actual code, but they certainly have shared 
lineage above that.  at this point though, the syntax has diverged quite a 
bit, so i'd love to just gut all the logic in sim and use the common dtc 
package as that's what the rest of the world is using now.  it'd also make it 
easier to share device tree definitions between projects (sim/u-
boot/linux/etc...).

importing dtc in directly would be my preference, but iiuc, people would 
prefer to only import FSF owned projects.  i know that ship sailed long ago 
with the sim codebase (large portions of it are not owned by the FSF), but 
still the FSF seems to want to keep that from expanding further.

so i guess that leaves us with linking against a system copy.  i'm fine with 
that too.  thoughts ?
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: integrating dtc into the sim/ tree

[Joel Brobecker]

> so i guess that leaves us with linking against a system copy.  i'm fine with 
> that too.  thoughts ?

Do I understand correctly that this is the status quo?

-- 
Joel

Re: integrating dtc into the sim/ tree

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 883 bytes --]

On Tue 19 Aug 2014 08:25:57 Joel Brobecker wrote:
> > so i guess that leaves us with linking against a system copy.  i'm fine
> > with that too.  thoughts ?
> 
> Do I understand correctly that this is the status quo?

gdb links against system libs which the FSF has no control over:
	- python
	- xz-utils (lzma)
	- expat
	- zlib

this last one is also included directly in the gcc tree (also the sourceware 
tree in the past iirc)

gcc links to non-FSF 3rd party libs, albeit doesn't bundle them

so after talking to myself, i guess the issue isn't so much with using the 
system copy of dtc as it is with making it a hard requirement (which is 
different from how the other packages use 3rd party libs -- they're all 
optional).  might be able to reduce the requirement so only the hw layers hard 
depend on it, but i'm not sure if even that is possible.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: integrating dtc into the sim/ tree

[Joel Brobecker]

> so after talking to myself, i guess the issue isn't so much with using
> the system copy of dtc as it is with making it a hard requirement
> (which is different from how the other packages use 3rd party libs --
> they're all optional).  might be able to reduce the requirement so
> only the hw layers hard depend on it, but i'm not sure if even that is
> possible.

OK. Is that dependency on dtc something new? Or was it there already
for certain sims and not others?

Dependency requirements like these should be discussed on a
case-per-case basis, I think. And the answers to the questions
you ask above (can it be made optional, what functionality gets
lost, etc) definitely influence the decision.

-- 
Joel

Re: integrating dtc into the sim/ tree

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 1762 bytes --]

On Wed 20 Aug 2014 11:53:47 Joel Brobecker wrote:
> > so after talking to myself, i guess the issue isn't so much with using
> > the system copy of dtc as it is with making it a hard requirement
> > (which is different from how the other packages use 3rd party libs --
> > they're all optional).  might be able to reduce the requirement so
> > only the hw layers hard depend on it, but i'm not sure if even that is
> > possible.
> 
> OK. Is that dependency on dtc something new? Or was it there already
> for certain sims and not others?
> 
> Dependency requirements like these should be discussed on a
> case-per-case basis, I think. And the answers to the questions
> you ask above (can it be made optional, what functionality gets
> lost, etc) definitely influence the decision.

moxie is the only one that hard requires dtc (it might be limited to 
maintainer mode).  but the larger point is to delete a large body of custom 
code that the sim has today for parsing its device tree like data format and 
convert over to the standard format that the rest of the world is using now.  
and longer term, make it so we can share dtc files between linux/u-boot/qemu 
such that you can feed a fdt to the sim and it'll automatically bring up 
hardware in the same way as the kernel would have found it.

atm, you have to basically write two different device trees with different 
syntax and names, then feed one to the sim and the other to the kernel.  and 
hope they don't get out of sync :).

there's basically no chance of people rewriting the existing sim code so that 
it gains all the same functionality as the public dtc, and then keeping it in 
sync.  i'd rather just gut it and be done, and get the dtc updates for free.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: integrating dtc into the sim/ tree

[Joel Brobecker]

> moxie is the only one that hard requires dtc (it might be limited to
> maintainer mode).  but the larger point is to delete a large body of
> custom code that the sim has today for parsing its device tree like
> data format and convert over to the standard format that the rest of
> the world is using now.  and longer term, make it so we can share dtc
> files between linux/u-boot/qemu such that you can feed a fdt to the
> sim and it'll automatically bring up hardware in the same way as the
> kernel would have found it.
> 
> atm, you have to basically write two different device trees with
> different syntax and names, then feed one to the sim and the other to
> the kernel.  and hope they don't get out of sync :).
> 
> there's basically no chance of people rewriting the existing sim code
> so that it gains all the same functionality as the public dtc, and
> then keeping it in sync.  i'd rather just gut it and be done, and get
> the dtc updates for free.

My 2 cents: This sounds interesting, but on the other hand, I have
this feeling that requiring dtc might be a big ask. I'm not sure
how portable the dtc project is, and how easy it is to get it
installed. I went to the "Device Tree Compiler" page you referenced,
and it doesn't give at all the impression of being a mature and
widespread project... For instance, I was looking for the documentation
in order to check for things like installation, OS support,
requirements, etc. I ended up looking inside the source tree itself,
and found Documentation/manual.txt and README, but none of them answered
any of these questions. I am also wondering about releases and such,
but couldn't really find much about it.

I hope this explains why I would personally feel a little more
comfortable if that dependency remained optional.

Now, if the project was really super easy to install and completely
portable (think Linux & Windows, of course, but also Darwin,
Solaris...), I would consider making it mandatory.

-- 
Joel

Re: integrating dtc into the sim/ tree

[Joel Sherrill]

On 8/20/2014 11:06 AM, Joel Brobecker wrote:
>> moxie is the only one that hard requires dtc (it might be limited to
>> maintainer mode).  but the larger point is to delete a large body of
>> custom code that the sim has today for parsing its device tree like
>> data format and convert over to the standard format that the rest of
>> the world is using now.  and longer term, make it so we can share dtc
>> files between linux/u-boot/qemu such that you can feed a fdt to the
>> sim and it'll automatically bring up hardware in the same way as the
>> kernel would have found it.
>>
>> atm, you have to basically write two different device trees with
>> different syntax and names, then feed one to the sim and the other to
>> the kernel.  and hope they don't get out of sync :).
>>
>> there's basically no chance of people rewriting the existing sim code
>> so that it gains all the same functionality as the public dtc, and
>> then keeping it in sync.  i'd rather just gut it and be done, and get
>> the dtc updates for free.
> My 2 cents: This sounds interesting, but on the other hand, I have
> this feeling that requiring dtc might be a big ask. I'm not sure
> how portable the dtc project is, and how easy it is to get it
> installed. I went to the "Device Tree Compiler" page you referenced,
> and it doesn't give at all the impression of being a mature and
> widespread project... For instance, I was looking for the documentation
> in order to check for things like installation, OS support,
> requirements, etc. I ended up looking inside the source tree itself,
> and found Documentation/manual.txt and README, but none of them answered
> any of these questions. I am also wondering about releases and such,
> but couldn't really find much about it.
>
> I hope this explains why I would personally feel a little more
> comfortable if that dependency remained optional.
>
> Now, if the project was really super easy to install and completely
> portable (think Linux & Windows, of course, but also Darwin,
> Solaris...), I would consider making it mandatory.
>
I would have been happy if the generated files were in the tree and
only maintainer mode would result in the dependencies being checked.

This has precedence for other tools.

-- 
Joel Sherrill, Ph.D.             Director of Research & Development
joel.sherrill@OARcorp.com        On-Line Applications Research
Ask me about RTEMS: a free RTOS  Huntsville AL 35805
Support Available                (256) 722-9985

Re: integrating dtc into the sim/ tree

[Joel Brobecker]

> I would have been happy if the generated files were in the tree and
> only maintainer mode would result in the dependencies being checked.
> 
> This has precedence for other tools.

Is it just generated files? I was under the impression that you had
to link against a library as well. If that wasn't the case, then it
would certainly mitigate most of my concerns!

-- 
Joel

Re: integrating dtc into the sim/ tree

[Joel Sherrill]

On 8/20/2014 11:19 AM, Joel Brobecker wrote:
>> I would have been happy if the generated files were in the tree and
>> only maintainer mode would result in the dependencies being checked.
>>
>> This has precedence for other tools.
> Is it just generated files? I was under the impression that you had
> to link against a library as well. If that wasn't the case, then it
> would certainly mitigate most of my concerns!
>
Inspecting the Makefile.in, I believe that is the case. This is all
there is:

moxie-gdb.dtb: moxie-gdb.dts
    dtc -O dtb -o moxie-gdb.dtb ${srcdir}/moxie-gdb.dts

install-dtb: moxie-gdb.dtb
    $(SHELL) $(srcdir)/../../mkinstalldirs $(DESTDIR)$(dtbdir)
    $(INSTALL_DATA) moxie-gdb.dtb $(DESTDIR)$(dtbdir)/moxie-gdb.dtb

No reference to libraries. I checked the generated moxie-gdb.dtb and it
shows this:

$ file moxie-gdb.dtb
moxie-gdb.dtb: Device Tree Blob version 17, size=519, boot CPU=0, string
block size=75, DT structure block size=388

I am nearly 100% sure this could be a maintainer only option.

FWIW we have been looking into using dtc with RTEMS. It is a nice
way to abstract away the particulars of a SoC and let the BSP/drivers
find their properties from a configuration file/blob during initialization.
But I don't know how this package is distributed.

-- 
Joel Sherrill, Ph.D.             Director of Research & Development
joel.sherrill@OARcorp.com        On-Line Applications Research
Ask me about RTEMS: a free RTOS  Huntsville AL 35805
Support Available                (256) 722-9985

Re: integrating dtc into the sim/ tree

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 635 bytes --]

On Wed 20 Aug 2014 18:19:21 Joel Brobecker wrote:
> > I would have been happy if the generated files were in the tree and
> > only maintainer mode would result in the dependencies being checked.
> > 
> > This has precedence for other tools.
> 
> Is it just generated files? I was under the impression that you had
> to link against a library as well. If that wasn't the case, then it
> would certainly mitigate most of my concerns!

currently it's just a generated file for moxie.  the compiled file is a 
standalone blob and treated as data.  but i want to move the sim past that 
point, hence the forked thread ;).
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: integrating dtc into the sim/ tree

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 2755 bytes --]

On Wed 20 Aug 2014 18:06:35 Joel Brobecker wrote:
> > moxie is the only one that hard requires dtc (it might be limited to
> > maintainer mode).  but the larger point is to delete a large body of
> > custom code that the sim has today for parsing its device tree like
> > data format and convert over to the standard format that the rest of
> > the world is using now.  and longer term, make it so we can share dtc
> > files between linux/u-boot/qemu such that you can feed a fdt to the
> > sim and it'll automatically bring up hardware in the same way as the
> > kernel would have found it.
> > 
> > atm, you have to basically write two different device trees with
> > different syntax and names, then feed one to the sim and the other to
> > the kernel.  and hope they don't get out of sync :).
> > 
> > there's basically no chance of people rewriting the existing sim code
> > so that it gains all the same functionality as the public dtc, and
> > then keeping it in sync.  i'd rather just gut it and be done, and get
> > the dtc updates for free.
> 
> My 2 cents: This sounds interesting, but on the other hand, I have
> this feeling that requiring dtc might be a big ask. I'm not sure
> how portable the dtc project is, and how easy it is to get it
> installed. I went to the "Device Tree Compiler" page you referenced,
> and it doesn't give at all the impression of being a mature and
> widespread project... For instance, I was looking for the documentation
> in order to check for things like installation, OS support,
> requirements, etc. I ended up looking inside the source tree itself,
> and found Documentation/manual.txt and README, but none of them answered
> any of these questions. I am also wondering about releases and such,
> but couldn't really find much about it.

it's pretty mature imo.  lemme phrase it this way: it's a hard requirement 
nowadays for ARM on Linux, so it's def viable.  i think a lot of the docs 
you're referring to is because the library aims to be used literally 
everywhere -- vendor BIOS, vendor kernels, etc...  the license readme explains 
this a bit more:
https://git.kernel.org/cgit/utils/dtc/dtc.git/tree/README.license

> I hope this explains why I would personally feel a little more
> comfortable if that dependency remained optional.
> 
> Now, if the project was really super easy to install and completely
> portable (think Linux & Windows, of course, but also Darwin,
> Solaris...), I would consider making it mandatory.

i'd be willing to make sure it builds everywhere.  the external dependencies 
in libfdt are extremely light (by design -- it wants to work in your typical 
BIOS).  basically it needs str/mem funcs and not ancient stdint.h.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: integrating dtc into the sim/ tree

[Joel Brobecker]

> it's pretty mature imo.  lemme phrase it this way: it's a hard
> requirement nowadays for ARM on Linux, so it's def viable.  i think a
> lot of the docs you're referring to is because the library aims to be
> used literally everywhere -- vendor BIOS, vendor kernels, etc...  the
> license readme explains this a bit more:
> https://git.kernel.org/cgit/utils/dtc/dtc.git/tree/README.license
[...]
> i'd be willing to make sure it builds everywhere.  the external
> dependencies in libfdt are extremely light (by design -- it wants to
> work in your typical BIOS).  basically it needs str/mem funcs and not
> ancient stdint.h.

You may actually have some issues with ancient stdint.h, but that
should be reasonably easy to take care of (eg. via gnulib) and
it seems to me like you have the answers to all my concerns, so
I no longer have any reservation towards that move. I would perhaps
poll the other GDB GMs to make sure they also have no objection
to it. So, to summarize:

  - Require dtc compiler in maintainer mode or when modifying
    some of the description files. Already required today for
    the moxie simulator.

    In non-maintainer mode, the generated files are already
    checked in.

  - Next step: Require and link against libdtc for all simulators
    instead of just moxie.

-- 
Joel

Re: integrating dtc into the sim/ tree

[Pedro Alves]

Hi guys,

On 08/21/2014 08:32 AM, Joel Brobecker wrote:
> I would perhaps poll the other GDB GMs to make sure they also have
> no objection to it.

I have no objections.  Good points were raised and addressed, and Mike's
reasoning is very convincing.

Thanks,
Pedro Alves

Re: integrating dtc into the sim/ tree

[Joel Sherrill]

On 8/21/2014 4:04 AM, Pedro Alves wrote:
> Hi guys,
>
> On 08/21/2014 08:32 AM, Joel Brobecker wrote:
>> I would perhaps poll the other GDB GMs to make sure they also have
>> no objection to it.
> I have no objections.  Good points were raised and addressed, and Mike's
> reasoning is very convincing.
I have no issues with it. Hopefully it will lead to more development
being on on the simulators. :)
> Thanks,
> Pedro Alves
>

-- 
Joel Sherrill, Ph.D.             Director of Research & Development
joel.sherrill@OARcorp.com        On-Line Applications Research
Ask me about RTEMS: a free RTOS  Huntsville AL 35805
Support Available                (256) 722-9985

Re: integrating dtc into the sim/ tree

[Richard Henderson]

On 08/20/2014 09:06 AM, Joel Brobecker wrote:
> Now, if the project was really super easy to install and completely
> portable (think Linux & Windows, of course, but also Darwin,
> Solaris...), I would consider making it mandatory.

It is.

It's required by the qemu project, so it's appearing in most linux
distributions.  So in that case one wouldn't even have to compile it yourself.


r~