[PATCH] Fix 64 bit red zone frame size in AIX

[PATCH] Fix 64 bit red zone frame size in AIX

[Aditya Kamath1]

[-- Attachment #1.1: Type: text/plain, Size: 2764 bytes --]

Respected Ulrich and GDB community members,

Hi,

Please find attached a patch. See: 0001-Fix-64-bit-red-zone-frame-size-in-AIX.patch.

The idea of this patch is to keep the concept behind the 32-bit mode red zone frame size and 64-bit mode red zone frame size the same.

So, we were experimenting around this red zone concept where if we call an internal function from a leaf or a non-leaf function whether we can safely create a frame for using the call feature without corrupting any red zone nonvolatile registers. As we went through the code, we realized that for 64-bit mode we do not set any space for the redzone. This cannot be true is what I feel. In 32-bit mode if we are leaving a space of 19 * 4 + 18 * 8 bytes for the 64-bit case as well I feel we should do the same.

So, we have 18 GPRS and 18 FPRS as per the document https://www.ibm.com/docs/en/aix/7.2?topic=overview-register-usage-conventions..

So, we should leave 18*8 + 18*8 = 288 bytes in 64-bit mode.. This patch does this..

Having said that when I tried to see if I can corrupt something while the red zone frame size was 0 in 64-bit mode I was not successful. I did try passing lots of arrays or numbers like in example 1 pasted below this email where I tried call function a with the call feature while I was in the main, but I was not successful in doing so. The attempt was to check when a’s execution is done , is the nonvolatile register contents of main retrieved?. And surprisingly it did even if I pass large structures or arrays. This makes me wonder if having red zone frame size 0 was correct or having 224 bytes in 32-bit mode is wrong.

Kindly give me a test case or let me know if there is any test suite that I can check, to see if 0 is the right frame size or what it should be so that we can do the right thing. It would be great if you can tell me if I analyzed this wrong as well.

Have a nice day ahead.

Thanks and regards,
Aditya.
Example 1:-
cat ~/gdb_tests/nine_parameter_func.c
#include <stdio.h>

int b ()
{
  const float register f1 = 1.0;
  const float register f2 = 2.0;
  float register f3 = 2.0;
  int register i1 = 900;
  return printf("%f %f\n", f1 + i1, f2 + f3);
}

int a (int b, int c, int d, int e, int f, int g, int h, int i, int j, int k, int l)
{
  const float register f3 = 12.0;
  const int register i1 = 800;
  printf("%f \n", f3 + i1);
  static int var = 123;
  b++;
  c++;
  d = e + f + g + h + i + j + k + l;
  printf ("9th para = %d , 10th para = %d\n", j, k);
  printf ("j = %d \n", j);
  return (int)(d);
}
int main ()
{
  const float register f3 = 19.0;
  const int register i1 = 700;
  printf("%f \n", f3 + i1);
  b ();
  a (1, 2, 3, 4, 5, 6, 7, 8, 9, 983, 19);
  return 0;
}


[-- Attachment #2: 0001-Fix-64-bit-red-zone-frame-size-in-AIX.patch --]
[-- Type: application/octet-stream, Size: 968 bytes --]

From f7b0cc04284027eefa02632d8b88664f90ae22c2 Mon Sep 17 00:00:00 2001
From: Aditya Vidyadhar Kamath <Aditya.Kamath1@ibm.com>
Date: Fri, 25 Aug 2023 09:12:57 -0500
Subject: [PATCH] Fix 64 bit red zone frame size in AIX

---
 gdb/rs6000-aix-tdep.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/gdb/rs6000-aix-tdep.c b/gdb/rs6000-aix-tdep.c
index 829f55981ca..419142bdb2d 100644
--- a/gdb/rs6000-aix-tdep.c
+++ b/gdb/rs6000-aix-tdep.c
@@ -1391,7 +1391,9 @@ rs6000_aix_init_osabi (struct gdbarch_info info, struct gdbarch *gdbarch)
        224.  */
     set_gdbarch_frame_red_zone_size (gdbarch, 224);
   else
-    set_gdbarch_frame_red_zone_size (gdbarch, 0);
+    /* In 64 bit mode the red zone should have 18 8 byte GPRS + 18 8 byte
+       FPRS making it 288 bytes. This is 16 byte aligned as well.  */  
+    set_gdbarch_frame_red_zone_size (gdbarch, 288);
 
   if (tdep->wordsize == 8)
     set_gdbarch_wchar_bit (gdbarch, 32);
-- 
2.38.3

Re: [PATCH] Fix 64 bit red zone frame size in AIX

[Ulrich Weigand]

Aditya Kamath1 <Aditya.Kamath1@ibm.com> wrote:

>So, we should leave 18*8 + 18*8 = 288 bytes in 64-bit mode.. This patch does this..

I think this is right - GCC assumes a 288 byte red zone on 64-bit AIX as well.

I've committed the patch now.
 
>Having said that when I tried to see if I can corrupt something while the red zone
>frame size was 0 in 64-bit mode I was not successful.

In order to trigger corruption, you need to interrupt a function that uses
the red zone.  It's up the compiler whether or not (and how) it uses that zone,
so this is not something you can reliably trigger.

If you want to experiment, I'd suggest using a leaf function that requires
some (but not too much) space on the stack.

Bye,
Ulrich

Re: [PATCH] Fix 64 bit red zone frame size in AIX

[Aditya Kamath1]

[-- Attachment #1: Type: text/plain, Size: 1699 bytes --]

Thank you :)

>In order to trigger corruption, you need to interrupt a function that uses
>the red zone.  It's up the compiler whether or not (and how) it uses that zone,
>so this is not something you can reliably trigger.

I realised this. I was not able to control this. For example if I have a function a () called from main and I am in a  () and then I use the call feature of GDB to call b (), GDB is safely creating a frame, whether b () needs an integer array of 1, 100 or 1000 and then manages to reproduce the volatile register contents of a () as it was before the call b ().. So yeah.. But a nice concept to learn. Thanks again.

From: Ulrich Weigand <Ulrich.Weigand@de.ibm.com>
Date: Friday, 25 August 2023 at 9:20 PM
To: gdb-patches@sourceware.org <gdb-patches@sourceware.org>, Aditya Kamath1 <Aditya.Kamath1@ibm.com>
Cc: Sangamesh Mallayya <sangamesh.swamy@in.ibm.com>
Subject: Re: [PATCH] Fix 64 bit red zone frame size in AIX
Aditya Kamath1 <Aditya.Kamath1@ibm.com> wrote:

>So, we should leave 18*8 + 18*8 = 288 bytes in 64-bit mode.. This patch does this..

I think this is right - GCC assumes a 288 byte red zone on 64-bit AIX as well.

I've committed the patch now.

>Having said that when I tried to see if I can corrupt something while the red zone
>frame size was 0 in 64-bit mode I was not successful.

In order to trigger corruption, you need to interrupt a function that uses
the red zone.  It's up the compiler whether or not (and how) it uses that zone,
so this is not something you can reliably trigger.

If you want to experiment, I'd suggest using a leaf function that requires
some (but not too much) space on the stack.

Bye,
Ulrich