RISC-V ELF multilibs

RISC-V ELF multilibs

[Sebastian Huber]

Hello,

I built a riscv64-rtems5 GCC (it uses gcc/config/riscv/t-elf-multilib). The following multilibs are built:

riscv64-rtems5-gcc -print-multi-lib
.;
rv32i/ilp32;@march=rv32i@mabi=ilp32
rv32im/ilp32;@march=rv32im@mabi=ilp32
rv32iac/ilp32;@march=rv32iac@mabi=ilp32
rv32imac/ilp32;@march=rv32imac@mabi=ilp32
rv32imafc/ilp32f;@march=rv32imafc@mabi=ilp32f
rv64imac/lp64;@march=rv64imac@mabi=lp64
rv64imafdc/lp64d;@march=rv64imafdc@mabi=lp64d

If I print out the builtin defines and search paths for the default settings and the -march=rv64imafdc and compare the results I get:

riscv64-rtems5-gcc -E -P -v -dD empty.c > def.txt 2>&1
riscv64-rtems5-gcc -E -P -v -dD empty.c -march=rv64imafdc > rv64imafdc.txt 2>&1
diff -u def.txt rv64imafdc.txt 
--- def.txt     2018-05-26 14:53:26.277760090 +0200
+++ rv64imafdc.txt      2018-05-26 14:53:47.705638409 +0200
@@ -4,8 +4,8 @@
 Configured with: ../gcc-7.3.0/configure --prefix=/opt/rtems/5 --bindir=/opt/rtems/5/bin --exec_prefix=/opt/rtems/5 --includedir=/opt/rtems/5/include --libdir=/opt/rtems/5/lib --libexecdir=/opt/rtems/5/libexec --mandir=/opt/rtems/5/share/man --infodir=/opt/rtems/5/share/info --datadir=/opt/rtems/5/share --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=riscv64-rtems5 --disable-libstdcxx-pch --with-gnu-as --with-gnu-ld --verbose --with-newlib --disable-nls --without-included-gettext --disable-win32-registry --enable-version-specific-runtime-libs --disable-lto --enable-newlib-io-c99-formats --enable-newlib-iconv --enable-newlib-iconv-encodings=big5,cp775,cp850,cp852,cp855,cp866,euc_jp,euc_kr,euc_tw,iso_8859_1,iso_8859_10,iso_8859_11,iso_8859_13,iso_8859_14,iso_8859_15,iso_8859_2,iso_8859_3,iso_8859_4,iso_8859_5,iso_8859_6,iso_8859_7,iso_8859_8,iso_8859_9,iso_ir_111,koi8_r,koi8_ru,koi8_u,koi8_uni,ucs_2,ucs_2_internal,ucs_2be,ucs_2le,ucs_4,ucs_4_internal,ucs_4be,ucs_4le,us_ascii,utf_16,utf_16be,utf_16le,utf_8,win_1250,win_1251,win_1252,win_1253,win_1254,win_1255,win_1256,win_1257,win_1258 --enable-threads --disable-plugin --enable-libgomp --enable-languages=c,c++,ada
 Thread model: rtems
 gcc version 7.3.0 20180125 (RTEMS 5, RSB a3a6c34c150a357e57769a26a460c475e188438f, Newlib 3.0.0) (GCC) 
-COLLECT_GCC_OPTIONS='-E' '-P' '-v' '-dD' '-march=rv64gc' '-mabi=lp64d'
- /opt/rtems/5/libexec/gcc/riscv64-rtems5/7.3.0/cc1 -E -quiet -v -P -imultilib rv64imafdc/lp64d empty.c -march=rv64gc -mabi=lp64d -dD
+COLLECT_GCC_OPTIONS='-E' '-P' '-v' '-dD' '-march=rv64imafdc' '-mabi=lp64d'
+ /opt/rtems/5/libexec/gcc/riscv64-rtems5/7.3.0/cc1 -E -quiet -v -P -imultilib rv64imafdc/lp64d empty.c -march=rv64imafdc -mabi=lp64d -dD
 ignoring nonexistent directory "/opt/rtems/5/lib/gcc/riscv64-rtems5/7.3.0/../../../../riscv64-rtems5/sys-include"
 #include "..." search starts here:
 #include <...> search starts here:
@@ -338,4 +338,4 @@
 #define __ELF__ 1
 COMPILER_PATH=/opt/rtems/5/libexec/gcc/riscv64-rtems5/7.3.0/:/opt/rtems/5/libexec/gcc/riscv64-rtems5/7.3.0/:/opt/rtems/5/libexec/gcc/riscv64-rtems5/:/opt/rtems/5/lib/gcc/riscv64-rtems5/7.3.0/:/opt/rtems/5/lib/gcc/riscv64-rtems5/:/opt/rtems/5/lib/gcc/riscv64-rtems5/7.3.0/../../../../riscv64-rtems5/bin/
 LIBRARY_PATH=/opt/rtems/5/lib/gcc/riscv64-rtems5/7.3.0/rv64imafdc/lp64d/:/opt/rtems/5/lib/gcc/riscv64-rtems5/7.3.0/../../../../riscv64-rtems5/lib/rv64imafdc/lp64d/:/opt/rtems/5/lib/gcc/riscv64-rtems5/7.3.0/:/opt/rtems/5/lib/gcc/riscv64-rtems5/7.3.0/../../../../riscv64-rtems5/lib/:/lib/:/usr/lib/
-COLLECT_GCC_OPTIONS='-E' '-P' '-v' '-dD' '-march=rv64gc' '-mabi=lp64d'
+COLLECT_GCC_OPTIONS='-E' '-P' '-v' '-dD' '-march=rv64imafdc' '-mabi=lp64d'

This looks pretty much the same and the documentation says that G == IMAFD.

Why is the default multilib and a variant identical?

Most variants include the C extension. Would it be possible to add -march=rv32g and -march=rv64g variants?

-- 
Sebastian Huber, embedded brains GmbH

Address : Dornierstr. 4, D-82178 Puchheim, Germany
Phone   : +49 89 189 47 41-16
Fax     : +49 89 189 47 41-09

Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.

Re: RISC-V ELF multilibs

[Jim Wilson]

On 05/26/2018 06:04 AM, Sebastian Huber wrote:
> Why is the default multilib and a variant identical?

This is supposed to be a single multilib, with two names.  We use 
MULTILIB_REUSE to map the two names to a single multilib.

rohan:1030$ ./xgcc -B./ -march=rv64imafdc -mabi=lp64d --print-libgcc
./rv64imafdc/lp64d/libgcc.a
rohan:1031$ ./xgcc -B./ -march=rv64gc -mabi=lp64d --print-libgcc
./rv64imafdc/lp64d/libgcc.a
rohan:1032$ ./xgcc -B./ --print-libgcc
./libgcc.a
rohan:1033$

So this is working right when the -march option is given, but not when 
no -march is given.  I'd suggest a bug report so I can track this, if 
you haven't already filed one.

> Most variants include the C extension. Would it be possible to add -march=rv32g and -march=rv64g variants?

The expectation is that most implementations will include the C 
extension.  It reduces code size, improves performance, and I think I 
read somewhere that it takes only 400 gates to implement.

It isn't practical to try to support every possible combination of 
architecture and ABI here, as there are too many possible combinations. 
But if there is a major RISC-V target that is rv32g or rv64g then we 
should consider it.  You can of course define your own set of multilibs.

Jim

Re: RISC-V ELF multilibs

[Palmer Dabbelt]

On Tue, 29 May 2018 11:02:58 PDT (-0700), Jim Wilson wrote:
> On 05/26/2018 06:04 AM, Sebastian Huber wrote:
>> Why is the default multilib and a variant identical?
>
> This is supposed to be a single multilib, with two names.  We use
> MULTILIB_REUSE to map the two names to a single multilib.
>
> rohan:1030$ ./xgcc -B./ -march=rv64imafdc -mabi=lp64d --print-libgcc
> ./rv64imafdc/lp64d/libgcc.a
> rohan:1031$ ./xgcc -B./ -march=rv64gc -mabi=lp64d --print-libgcc
> ./rv64imafdc/lp64d/libgcc.a
> rohan:1032$ ./xgcc -B./ --print-libgcc
> ./libgcc.a
> rohan:1033$
>
> So this is working right when the -march option is given, but not when
> no -march is given.  I'd suggest a bug report so I can track this, if
> you haven't already filed one.

IIRC this is actually a limit of the GCC build system: there needs to be some 
default multilib, and it has to be unprefixed.  I wanted to keep the library 
paths orthogonal (ie, not bake in a default that rv64gc/lp64d lives at /lib), 
so I chose to just build a redundant multilib.

It'd be great to get rid of this, but I'm afraid it's way past my level of 
understanding as to how all this works.

>> Most variants include the C extension. Would it be possible to add -march=rv32g and -march=rv64g variants?
>
> The expectation is that most implementations will include the C
> extension.  It reduces code size, improves performance, and I think I
> read somewhere that it takes only 400 gates to implement.
>
> It isn't practical to try to support every possible combination of
> architecture and ABI here, as there are too many possible combinations.
> But if there is a major RISC-V target that is rv32g or rv64g then we
> should consider it.  You can of course define your own set of multilibs.

While that's the standard answer, note that Sebastian added the RISC-V RTEMS 
target in the first place so as far as I'm concerned he can add additional 
multilibs to it if he wants.  While I'm not opposed to RTEMS multilibs for 
rv32g/ilp32d and rv64g/lp64d, note that we have made the decision in Linux land 
that the C extension will be common and thus I expect most RISC-V 
implementations with virtual memory to also have the C extension.  If you go 
down this route then you should move RTEMS to its own multilib target fragment 
(t-rtems-multilib or something similar).

If you need help figuring out the patch feel free to ask.  I wrote a blog that 
might be useful

    https://www.sifive.com/blog/2017/09/18/all-aboard-part-5-risc-v-multilib/

RE: RISC-V ELF multilibs

[Matthew Fortune]

Palmer Dabbelt <palmer@sifive.com> writes:
> On Tue, 29 May 2018 11:02:58 PDT (-0700), Jim Wilson wrote:
> > On 05/26/2018 06:04 AM, Sebastian Huber wrote:
> >> Why is the default multilib and a variant identical?
> >
> > This is supposed to be a single multilib, with two names.  We use
> > MULTILIB_REUSE to map the two names to a single multilib.
> >
> > rohan:1030$ ./xgcc -B./ -march=rv64imafdc -mabi=lp64d --print-libgcc
> > ./rv64imafdc/lp64d/libgcc.a
> > rohan:1031$ ./xgcc -B./ -march=rv64gc -mabi=lp64d --print-libgcc
> > ./rv64imafdc/lp64d/libgcc.a
> > rohan:1032$ ./xgcc -B./ --print-libgcc
> > ./libgcc.a
> > rohan:1033$
> >
> > So this is working right when the -march option is given, but not
> when
> > no -march is given.  I'd suggest a bug report so I can track this, if
> > you haven't already filed one.
> 
> IIRC this is actually a limit of the GCC build system: there needs to
> be some
> default multilib, and it has to be unprefixed.  I wanted to keep the
> library
> paths orthogonal (ie, not bake in a default that rv64gc/lp64d lives at
> /lib),
> so I chose to just build a redundant multilib.
> 
> It'd be great to get rid of this, but I'm afraid it's way past my level
> of
> understanding as to how all this works.

I do actually have a solution for this but it is not submitted upstream.
MIPS has basically the same set of problems that RISC-V does in this area
and in an ideal world there would be no 'fallback' multilib such that if
you use compiler options that map to a library variant that does not
exist then the linker just fails to find any libraries at all rather than
using the default multilib.

I can share the raw patch for this and try to give you some idea about how
it works. I am struggling to find time to do much open source support at
the moment so may not be able to do all the due diligence to get it
committed. Would you be willing to take a look and do some of the work to
get it in tree?

Matthew

Re: RISC-V ELF multilibs

[Jim Wilson]

On Thu, May 31, 2018 at 7:23 AM, Matthew Fortune
<Matthew.Fortune@mips.com> wrote:
> I do actually have a solution for this but it is not submitted upstream.
> MIPS has basically the same set of problems that RISC-V does in this area
> and in an ideal world there would be no 'fallback' multilib such that if
> you use compiler options that map to a library variant that does not
> exist then the linker just fails to find any libraries at all rather than
> using the default multilib.
>
> I can share the raw patch for this and try to give you some idea about how
> it works. I am struggling to find time to do much open source support at
> the moment so may not be able to do all the due diligence to get it
> committed. Would you be willing to take a look and do some of the work to
> get it in tree?

I have a long list of things on my to do list.  RISC-V is a new
target, and there is lots of stuff that needs to be bug fixed,
finished, or added.  I can't make any guarantees.

But if you file a bug report and then attach a patch to it, someone
might volunteer to help finish it.  Or if it is too big to be
reasonably attached to a bug report (like the nano mips work) you
could put it on a branch, and mention the branch name as unfinished
work in a bug report.

Jim

Re: RISC-V ELF multilibs

[Sebastian Huber]

On 29/05/18 20:02, Jim Wilson wrote:
>> Most variants include the C extension. Would it be possible to add 
>> -march=rv32g and -march=rv64g variants?
>
> The expectation is that most implementations will include the C 
> extension.  It reduces code size, improves performance, and I think I 
> read somewhere that it takes only 400 gates to implement.
>
> It isn't practical to try to support every possible combination of 
> architecture and ABI here, as there are too many possible 
> combinations. But if there is a major RISC-V target that is rv32g or 
> rv64g then we should consider it.  You can of course define your own 
> set of multilibs.

I am not a hardware developer, but I heard that the C extension is not 
easy to implement in out of order machines. For example:

https://www2.eecs.berkeley.edu/Pubs/TechRpts/2017/EECS-2017-157.html

-- 
Sebastian Huber, embedded brains GmbH

Address : Dornierstr. 4, D-82178 Puchheim, Germany
Phone   : +49 89 189 47 41-16
Fax     : +49 89 189 47 41-09
E-Mail  : sebastian.huber@embedded-brains.de
PGP     : Public key available on request.

Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.

Re: RISC-V ELF multilibs

[Sebastian Huber]

On 31/05/18 11:08, Palmer Dabbelt wrote:
> On Tue, 29 May 2018 11:02:58 PDT (-0700), Jim Wilson wrote:
>> On 05/26/2018 06:04 AM, Sebastian Huber wrote:
>>> Why is the default multilib and a variant identical?
>>
>> This is supposed to be a single multilib, with two names.  We use
>> MULTILIB_REUSE to map the two names to a single multilib.
>>
>> rohan:1030$ ./xgcc -B./ -march=rv64imafdc -mabi=lp64d --print-libgcc
>> ./rv64imafdc/lp64d/libgcc.a
>> rohan:1031$ ./xgcc -B./ -march=rv64gc -mabi=lp64d --print-libgcc
>> ./rv64imafdc/lp64d/libgcc.a
>> rohan:1032$ ./xgcc -B./ --print-libgcc
>> ./libgcc.a
>> rohan:1033$
>>
>> So this is working right when the -march option is given, but not when
>> no -march is given.  I'd suggest a bug report so I can track this, if
>> you haven't already filed one.
>
> IIRC this is actually a limit of the GCC build system: there needs to 
> be some default multilib, and it has to be unprefixed.  I wanted to 
> keep the library paths orthogonal (ie, not bake in a default that 
> rv64gc/lp64d lives at /lib), so I chose to just build a redundant 
> multilib.
>
> It'd be great to get rid of this, but I'm afraid it's way past my 
> level of understanding as to how all this works.

Ok, I just wanted to know if this was intentional or some sort of a bug.

Some way to disable the default multilib would be nice in general. 
Ending up with the default multilib usually indicates some build system 
misconfiguration. It is not easy for the average user to figure out what 
is going on especially on targets where the linker silently links ABI 
incompatible objects together.

>
>>> Most variants include the C extension. Would it be possible to add 
>>> -march=rv32g and -march=rv64g variants?
>>
>> The expectation is that most implementations will include the C
>> extension.  It reduces code size, improves performance, and I think I
>> read somewhere that it takes only 400 gates to implement.
>>
>> It isn't practical to try to support every possible combination of
>> architecture and ABI here, as there are too many possible combinations.
>> But if there is a major RISC-V target that is rv32g or rv64g then we
>> should consider it.  You can of course define your own set of multilibs.
>
> While that's the standard answer, note that Sebastian added the RISC-V 
> RTEMS target in the first place so as far as I'm concerned he can add 
> additional multilibs to it if he wants.  While I'm not opposed to 
> RTEMS multilibs for rv32g/ilp32d and rv64g/lp64d, note that we have 
> made the decision in Linux land that the C extension will be common 
> and thus I expect most RISC-V implementations with virtual memory to 
> also have the C extension.  If you go down this route then you should 
> move RTEMS to its own multilib target fragment (t-rtems-multilib or 
> something similar).
>
> If you need help figuring out the patch feel free to ask.  I wrote a 
> blog that might be useful
>
> https://www.sifive.com/blog/2017/09/18/all-aboard-part-5-risc-v-multilib/

I will probably add a special multilib set for RTEMS.

-- 
Sebastian Huber, embedded brains GmbH

Address : Dornierstr. 4, D-82178 Puchheim, Germany
Phone   : +49 89 189 47 41-16
Fax     : +49 89 189 47 41-09
E-Mail  : sebastian.huber@embedded-brains.de
PGP     : Public key available on request.

Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.

Re: RISC-V ELF multilibs

[Michael Clark]

> On 1/06/2018, at 6:16 PM, Sebastian Huber <sebastian.huber@embedded-brains.de> wrote:
> 
> On 29/05/18 20:02, Jim Wilson wrote:
>>> Most variants include the C extension. Would it be possible to add -march=rv32g and -march=rv64g variants?
>> 
>> The expectation is that most implementations will include the C extension.  It reduces code size, improves performance, and I think I read somewhere that it takes only 400 gates to implement.
>> 
>> It isn't practical to try to support every possible combination of architecture and ABI here, as there are too many possible combinations. But if there is a major RISC-V target that is rv32g or rv64g then we should consider it.  You can of course define your own set of multilibs.
> 
> I am not a hardware developer, but I heard that the C extension is not easy to implement in out of order machines.

Easy is relative.

The RISC-V ISA encoding has been designed so that wide fetch is relatively easy, possibly an order of magnitude easier than an ISA with a complex variable length encoding like x86. 

RV64GC instruction lengths can be derived from the 2 least significant bits in the first half-word packet of each instruction for mixed 16/32 bit instructions. It does not require an attempt to parse multiple prefixes with instructions ranging from 1 byte up to 15 bytes, before arriving at an instruction length (x86). From that perspective RV64GC is decidedly simple. That said, RV64GC is a little more complex than regular 32-bit encodings like RV64G, PowerPC, AArch64, or SPARC.

I don’t think the paper you have linked to draws the conclusion you are arriving at. I spoke to Chris Celio about RVC and he said he just just didn’t get around to implementing RVC in BOOM, not necessarily that it’s absence is a statement of its difficulty rather the academic objectives of BOOM, one of a very small number of Open Source OoO processors. Sure, if it was “trivial” then BOOM would include it, so it’s certainly non trivial.

For BOOM, RVC requires an instruction buffer that handles unaligned fetches and the way the PC is derived further down the pipeline needs to be modified as the same micro-op may have a different width depending on whether it was expanded from an RVC encoding. It may or may not an extra pre-decoder pipe stage. I have a fair understanding of what’s requested. The misaligned fetch is probably the most complex but BOOM already needs to have complex cases like the fetch buffer spanning cache lines and page boundaries. Unaligned instruction fetches add to this.

> For example:
> 
> https://www2.eecs.berkeley.edu/Pubs/TechRpts/2017/EECS-2017-157.html
> 
> -- 
> Sebastian Huber, embedded brains GmbH
> 
> Address : Dornierstr. 4, D-82178 Puchheim, Germany
> Phone   : +49 89 189 47 41-16
> Fax     : +49 89 189 47 41-09
> E-Mail  : sebastian.huber@embedded-brains.de
> PGP     : Public key available on request.
> 
> Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.
> 

Re: RISC-V ELF multilibs

[Sebastian Huber]

On 01/06/18 10:23, Michael Clark wrote:
>
>> On 1/06/2018, at 6:16 PM, Sebastian Huber <sebastian.huber@embedded-brains.de> wrote:
>>
>> On 29/05/18 20:02, Jim Wilson wrote:
>>>> Most variants include the C extension. Would it be possible to add -march=rv32g and -march=rv64g variants?
>>> The expectation is that most implementations will include the C extension.  It reduces code size, improves performance, and I think I read somewhere that it takes only 400 gates to implement.
>>>
>>> It isn't practical to try to support every possible combination of architecture and ABI here, as there are too many possible combinations. But if there is a major RISC-V target that is rv32g or rv64g then we should consider it.  You can of course define your own set of multilibs.
>> I am not a hardware developer, but I heard that the C extension is not easy to implement in out of order machines.
> Easy is relative.
>
> The RISC-V ISA encoding has been designed so that wide fetch is relatively easy, possibly an order of magnitude easier than an ISA with a complex variable length encoding like x86.
>
> RV64GC instruction lengths can be derived from the 2 least significant bits in the first half-word packet of each instruction for mixed 16/32 bit instructions. It does not require an attempt to parse multiple prefixes with instructions ranging from 1 byte up to 15 bytes, before arriving at an instruction length (x86). From that perspective RV64GC is decidedly simple. That said, RV64GC is a little more complex than regular 32-bit encodings like RV64G, PowerPC, AArch64, or SPARC.
>
> I don’t think the paper you have linked to draws the conclusion you are arriving at.

Yes, please use my words with care. I am not a hardware developer. 
Reading "A.3 The Compressed (“C”) ISA Extension" in

https://github.com/ccelio/riscv-boom-doc/

lead me to this statement that it is not easy from my inexperienced 
point of view.

-- 
Sebastian Huber, embedded brains GmbH

Address : Dornierstr. 4, D-82178 Puchheim, Germany
Phone   : +49 89 189 47 41-16
Fax     : +49 89 189 47 41-09
E-Mail  : sebastian.huber@embedded-brains.de
PGP     : Public key available on request.

Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.

RE: RISC-V ELF multilibs

[Palmer Dabbelt]

On Thu, 31 May 2018 07:23:22 PDT (-0700), Matthew.Fortune@mips.com wrote:
> Palmer Dabbelt <palmer@sifive.com> writes:
>> On Tue, 29 May 2018 11:02:58 PDT (-0700), Jim Wilson wrote:
>> > On 05/26/2018 06:04 AM, Sebastian Huber wrote:
>> >> Why is the default multilib and a variant identical?
>> >
>> > This is supposed to be a single multilib, with two names.  We use
>> > MULTILIB_REUSE to map the two names to a single multilib.
>> >
>> > rohan:1030$ ./xgcc -B./ -march=rv64imafdc -mabi=lp64d --print-libgcc
>> > ./rv64imafdc/lp64d/libgcc.a
>> > rohan:1031$ ./xgcc -B./ -march=rv64gc -mabi=lp64d --print-libgcc
>> > ./rv64imafdc/lp64d/libgcc.a
>> > rohan:1032$ ./xgcc -B./ --print-libgcc
>> > ./libgcc.a
>> > rohan:1033$
>> >
>> > So this is working right when the -march option is given, but not
>> when
>> > no -march is given.  I'd suggest a bug report so I can track this, if
>> > you haven't already filed one.
>> 
>> IIRC this is actually a limit of the GCC build system: there needs to
>> be some
>> default multilib, and it has to be unprefixed.  I wanted to keep the
>> library
>> paths orthogonal (ie, not bake in a default that rv64gc/lp64d lives at
>> /lib),
>> so I chose to just build a redundant multilib.
>> 
>> It'd be great to get rid of this, but I'm afraid it's way past my level
>> of
>> understanding as to how all this works.
> 
> I do actually have a solution for this but it is not submitted upstream.
> MIPS has basically the same set of problems that RISC-V does in this area
> and in an ideal world there would be no 'fallback' multilib such that if
> you use compiler options that map to a library variant that does not
> exist then the linker just fails to find any libraries at all rather than
> using the default multilib.
> 
> I can share the raw patch for this and try to give you some idea about how
> it works. I am struggling to find time to do much open source support at
> the moment so may not be able to do all the due diligence to get it
> committed. Would you be willing to take a look and do some of the work to
> get it in tree?

If you have a rough patch that'd be great, it'll be at least a starting point.  
Last time I poked around I wasn't ever sure where to look.

Thanks!