[Bug tree-optimization/112752] New: `~a - MIN<MIN<~a, ~b>, ~c>` is not optimized to `MAX<MAX<a,b>,c> - a`

[Bug tree-optimization/112752] New: `~a - MIN<MIN<~a, ~b>, ~c>` is not optimized to `MAX<MAX<a,b>,c> - a`

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112752

            Bug ID: 112752
           Summary: `~a - MIN<MIN<~a, ~b>, ~c>` is not optimized to
                    `MAX<MAX<a,b>,c> - a`
           Product: gcc
           Version: 14.0
            Status: UNCONFIRMED
          Keywords: missed-optimization
          Severity: enhancement
          Priority: P3
         Component: tree-optimization
          Assignee: unassigned at gcc dot gnu.org
          Reporter: pinskia at gcc dot gnu.org
                CC: evstupac at gmail dot com, kirill.yukhin at intel dot com,
                    pinskia at gcc dot gnu.org, rguenth at gcc dot gnu.org,
                    unassigned at gcc dot gnu.org
  Target Milestone: ---
            Target: x86_64-*-*

+++ This bug was initially created as a clone of Bug #52252 +++

This is an example of byte conversion from RGB (Red Green Blue) to CMYK (Cyan
Magenta Yellow blacK):
```
#define byte unsigned char
#define MIN(a, b) ((a) > (b)?(b):(a))

void convert_image(byte *in, byte *out, int size) {
    int i;
    for(i = 0; i < size; i++) {
        byte r = in[0];
        byte g = in[1];
        byte b = in[2];
        byte c, m, y, k, tmp;
        c = 255 - r;
        m = 255 - g;
        y = 255 - b;
        tmp = MIN(m, y);
        k = MIN(c, tmp);
        out[0] = c - k;
        out[1] = m - k;
        out[2] = y - k;
        out[3] = k;
        in += 3;
        out += 4;
    }
}
```

For the scalar (and vectorized versions) we should get instead:
```
#define byte unsigned char
#define MIN(a, b) ((a) > (b)?(b):(a))
#define MAX(a, b) ((a) < (b)?(b):(a))

void convert_image_1(byte *in, byte *out, int size) {
    int i;
    for(i = 0; i < size; i++) {
        byte r = in[0];
        byte g = in[1];
        byte b = in[2];
        byte c, m, y, k, tmp;
        tmp = MIN(r, g);
        k = MIN(b, tmp);
        out[0] = k - r;
        out[1] = k - g;
        out[2] = k - b;
        out[3] = 255 - k;
        in += 3;
        out += 4;
    }
}
```

See bug 52252 comment #10, 11, and 12 also.

[Bug tree-optimization/112752] `~a - MIN<MIN<~a, ~b>, ~c>` is not optimized to `MAX<MAX<a,b>,c> - a`

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112752

--- Comment #1 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
I made a small mistake into what it should be optimized to:
The MIN should be MAX (oops):
that is:
        tmp = MAX(r, g);
        k = MAX(b, tmp);

[Bug tree-optimization/112752] `~a - MIN<MIN<~a, ~b>, ~c>` is not optimized to `MAX<MAX<a,b>,c> - a`

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112752

--- Comment #2 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
Here is a semi-reduced testcase:
```
#define byte unsigned char
#define MIN(a, b) ((a) > (b)?(b):(a))

byte hh(byte r, byte g, byte b) {
  byte c = 255 - r;
  byte m = 255 - g;
  byte y = 255 - b;
  byte tmp = MIN(m, y);
  byte k = MIN(c, tmp);
  return m - k;
}
```
Note matching this directly, the above might not fix the original testcase.