[PATCH] Fix 64 bit red zone frame size in AIX

[Aditya Kamath1 <Aditya.Kamath1@ibm.com>]

[-- Attachment #1.1: Type: text/plain, Size: 2764 bytes --]

Respected Ulrich and GDB community members,

Hi,

Please find attached a patch. See: 0001-Fix-64-bit-red-zone-frame-size-in-AIX.patch.

The idea of this patch is to keep the concept behind the 32-bit mode red zone frame size and 64-bit mode red zone frame size the same.

So, we were experimenting around this red zone concept where if we call an internal function from a leaf or a non-leaf function whether we can safely create a frame for using the call feature without corrupting any red zone nonvolatile registers. As we went through the code, we realized that for 64-bit mode we do not set any space for the redzone. This cannot be true is what I feel. In 32-bit mode if we are leaving a space of 19 * 4 + 18 * 8 bytes for the 64-bit case as well I feel we should do the same.

So, we have 18 GPRS and 18 FPRS as per the document https://www.ibm.com/docs/en/aix/7.2?topic=overview-register-usage-conventions..

So, we should leave 18*8 + 18*8 = 288 bytes in 64-bit mode.. This patch does this..

Having said that when I tried to see if I can corrupt something while the red zone frame size was 0 in 64-bit mode I was not successful. I did try passing lots of arrays or numbers like in example 1 pasted below this email where I tried call function a with the call feature while I was in the main, but I was not successful in doing so. The attempt was to check when a’s execution is done , is the nonvolatile register contents of main retrieved?. And surprisingly it did even if I pass large structures or arrays. This makes me wonder if having red zone frame size 0 was correct or having 224 bytes in 32-bit mode is wrong.

Kindly give me a test case or let me know if there is any test suite that I can check, to see if 0 is the right frame size or what it should be so that we can do the right thing. It would be great if you can tell me if I analyzed this wrong as well.

Have a nice day ahead.

Thanks and regards,
Aditya.
Example 1:-
cat ~/gdb_tests/nine_parameter_func.c
#include <stdio.h>

int b ()
{
  const float register f1 = 1.0;
  const float register f2 = 2.0;
  float register f3 = 2.0;
  int register i1 = 900;
  return printf("%f %f\n", f1 + i1, f2 + f3);
}

int a (int b, int c, int d, int e, int f, int g, int h, int i, int j, int k, int l)
{
  const float register f3 = 12.0;
  const int register i1 = 800;
  printf("%f \n", f3 + i1);
  static int var = 123;
  b++;
  c++;
  d = e + f + g + h + i + j + k + l;
  printf ("9th para = %d , 10th para = %d\n", j, k);
  printf ("j = %d \n", j);
  return (int)(d);
}
int main ()
{
  const float register f3 = 19.0;
  const int register i1 = 700;
  printf("%f \n", f3 + i1);
  b ();
  a (1, 2, 3, 4, 5, 6, 7, 8, 9, 983, 19);
  return 0;
}


[-- Attachment #2: 0001-Fix-64-bit-red-zone-frame-size-in-AIX.patch --]
[-- Type: application/octet-stream, Size: 968 bytes --]

From f7b0cc04284027eefa02632d8b88664f90ae22c2 Mon Sep 17 00:00:00 2001
From: Aditya Vidyadhar Kamath <Aditya.Kamath1@ibm.com>
Date: Fri, 25 Aug 2023 09:12:57 -0500
Subject: [PATCH] Fix 64 bit red zone frame size in AIX

---
 gdb/rs6000-aix-tdep.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/gdb/rs6000-aix-tdep.c b/gdb/rs6000-aix-tdep.c
index 829f55981ca..419142bdb2d 100644
--- a/gdb/rs6000-aix-tdep.c
+++ b/gdb/rs6000-aix-tdep.c
@@ -1391,7 +1391,9 @@ rs6000_aix_init_osabi (struct gdbarch_info info, struct gdbarch *gdbarch)
        224.  */
     set_gdbarch_frame_red_zone_size (gdbarch, 224);
   else
-    set_gdbarch_frame_red_zone_size (gdbarch, 0);
+    /* In 64 bit mode the red zone should have 18 8 byte GPRS + 18 8 byte
+       FPRS making it 288 bytes. This is 16 byte aligned as well.  */  
+    set_gdbarch_frame_red_zone_size (gdbarch, 288);
 
   if (tdep->wordsize == 8)
     set_gdbarch_wchar_bit (gdbarch, 32);
-- 
2.38.3