Re: [PATCH] Fix call functions command bug in 64-bit programs for AIX

[Ulrich Weigand <Ulrich.Weigand@de.ibm.com>]

Aditya Kamath1 <Aditya.Kamath1@ibm.com> wrote:

>The issue is that when a user attempts to test the return type or print
>statements via the call "FUNC_NAME (parameter A, parameter B)" command,
any input be it parameter A or B is taken in little endian format from
>the GDB cache. But AIX is using Big endian format. 

>For example, if we have a value 21 as type long then the higher 32 bits
>[ which is 0 in number 21] were stored in lower 32 bits and lower 32 bits
>[ represent 21 in the number 21] is stored in higher 32 bits.

This is not how the GDB register cache is working.  Can you explain in
more detail what exactly you're refering to here?

The register cache holds a sequence of bytes for each register.  The
number of bytes in the cache depends on the size of that register as
defined by the architecture.  The contents of those bytes are *always*
assumed to be in big-endian byte order on AIX.

For 64-bit inferior processes, GDB assumes that the ptrace call
always uses a 64-bit buffer, even if the register size is only 4.
To handle these cases, the current code will convert between the
two formats, which looks correct to me: in the fetch_register
case, it will retrieve an 8-byte value from ptrace, convert that
value (correctly, also for big-endian) to an 4-byte value, and
pass that 4-byte value to raw_supply, which should expect 4 bytes
in this case because the register_size is 4.

To understand what's going on exactly I need more details:
- what does ARCH64() return?
- what does register_size(...) return for those registers?
  (for GPRs with a 64-bit inferior it should return 8, if
  it doesn't that's probably the root cause of the problem)
- what value does ptrace return in your case?


In any case, explicitly listing specific register numbers by ABI
is definitely wrong here.  This routine must correctly handle
any register, it does not matter what the ABI usage is.


Bye,
Ulrich