[binutils-gdb] x86-64: refuse "high" 8-bit regs with .insn and VEX/XOP/EVEX encodings

[Jan Beulich <jbeulich@sourceware.org>]

https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=3e4a511bee874d73f9f749cc8cf3bc748b4d47b5

commit 3e4a511bee874d73f9f749cc8cf3bc748b4d47b5
Author: Jan Beulich <jbeulich@suse.com>
Date:   Fri Dec 22 09:34:10 2023 +0100

    x86-64: refuse "high" 8-bit regs with .insn and VEX/XOP/EVEX encodings
    
    Much like REX, those encodings - if permitting 8-bit regs at all, i.e.
    only starting with APX - permit use of "new" 8-bit registers only. %ah,
    %ch, %dh, and %bh cannot be encoded and hence should be rejected.
    
    Permit their use outside of 64-bit code though, as "new" registers
    simply don't exist there.

Diff:
---
 gas/config/tc-i386.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/gas/config/tc-i386.c b/gas/config/tc-i386.c
index 6e2f6e51457..cdd3b55c655 100644
--- a/gas/config/tc-i386.c
+++ b/gas/config/tc-i386.c
@@ -11485,6 +11485,16 @@ s_insn (int dummy ATTRIBUTE_UNUSED)
 
       for (j = i.imm_operands; j < i.operands; ++j)
 	{
+	  /* Look for 8-bit operands that use old registers.  */
+	  if (i.vec_encoding != vex_encoding_default
+	      && flag_code == CODE_64BIT
+	      && i.types[j].bitfield.class == Reg
+	      && i.types[j].bitfield.byte
+	      && !(i.op[j].regs->reg_flags & RegRex64)
+	      && i.op[j].regs->reg_num > 3)
+	    as_bad (_("can't encode register '%s%s' with VEX/XOP/EVEX"),
+		    register_prefix, i.op[j].regs->reg_name);
+
 	  i.types[j].bitfield.instance = InstanceNone;
 
 	  if (operand_type_check (i.types[j], disp))