[Bug target/64384] New: mingw-w64: stdcall function returning an aggregate is incompatible with MS ABI

[Bug target/64384] New: mingw-w64: stdcall function returning an aggregate is incompatible with MS ABI

[mity at morous dot org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64384

            Bug ID: 64384
           Summary: mingw-w64: stdcall function returning an aggregate is
                    incompatible with MS ABI
           Product: gcc
           Version: 4.9.2
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: target
          Assignee: unassigned at gcc dot gnu.org
          Reporter: mity at morous dot org

When using COM interface (i.e. calling stdcall function/method), aggregates are
returned through an implicit parameter after the this/interface pointer,
according to MS ABI. However it is not so when using gcc.

This issue manifests itself when e.g. calling ID2D1RenderTarget::GetSize() from
<d2d1.h> as discussed here:

https://sourceforge.net/p/mingw-w64/mailman/mingw-w64-public/?style=threaded&viewmonth=201412&viewday=23


Wine team encountered the issue too, they seem to "solve" the issue on their
side by ugly hacks and changing prototypes of the relevant functions for now:

https://www.winehq.org/pipermail/wine-patches/2014-September/134351.html

[Bug target/64384] mingw-w64: stdcall function returning an aggregate is incompatible with MS ABI

[jacek at codeweavers dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64384

Jacek Caban <jacek at codeweavers dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jacek at codeweavers dot com

--- Comment #1 from Jacek Caban <jacek at codeweavers dot com> ---
FWIW the issue is not specific to COM interfaces. It affects all stdcall
functions.

[Bug target/64384] mingw-w64: stdcall function returning an aggregate is incompatible with MS ABI

[ktietz at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64384

Kai Tietz <ktietz at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |ktietz at gcc dot gnu.org

--- Comment #2 from Kai Tietz <ktietz at gcc dot gnu.org> ---
The attribute 'callee_pop_aggregate_return' should fix this issue for you.
This is a general difference between sysv and ms ABIs.  It has in general
nothing to do with the stdcall-attribute.

[Bug target/64384] mingw-w64: stdcall function returning an aggregate is incompatible with MS ABI

[daniel.c.klauer at web dot de]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64384

daniel.c.klauer at web dot de changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |daniel.c.klauer at web dot de

--- Comment #3 from daniel.c.klauer at web dot de ---
It seems there is a difference between gcc and ms with regards to return in
register for C++ methods (regardless of stdcall/cdecl/thiscall).

Tested with i686-w64-mingw32 gcc 4.9.2 and cl.exe from VS 2010:

struct A {
    int field1;
};

struct A getA(void) {
    struct A a = {123};
    return a;
}

C function: both gcc and ms return in register (struct is small enough), ok.

struct A {
    int field1;
};

class Test {
public:
    A getA();
};

A Test::getA() {
    struct A a = {123};
    return a;
}

C++ method: ms returns in temp var on stack (as for big structs), but gcc
returns in register. 

The same happens for struct containing two floats (as in the original crash
issue reported on mingw-w64 mailing list).

Maybe (on 32bit) ms never uses return in register for C++ methods with
aggregate result?

[Bug target/64384] mingw-w64: stdcall function returning an aggregate is incompatible with MS ABI

[mity at morous dot org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64384

--- Comment #4 from mity <mity at morous dot org> ---
Daniel, COM interface should be, by definition, language agnostic. COM can be
called from C++ as well as from C, and also COM object may be implemented in
C++ as well as C. This implies that (at least for stdcall, as COM uses stdcall
convention) there shouldn't be any difference between C and C++.

[Bug target/64384] mingw-w64: stdcall function returning an aggregate is incompatible with MS ABI

[daniel.c.klauer at web dot de]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64384

--- Comment #5 from daniel.c.klauer at web dot de ---
I also think that COM is supposed to be usable from C and C++, but it seems
that here we have an exception to this rule.

Let's examine asm generated by the ms compiler for calls to GetSize(), C and
C++ versions respectively:

C++:
  ID2D1HwndRenderTarget *hwndRenderTarget;
  D2D1_SIZE_F size;
  size = hwndRenderTarget->GetSize();
Code generated by cl:
  lea   edx, DWORD PTR $T76363[ebp]                    # temp var for the
result
  push  edx                                            # hidden pointer arg
  [...push THIS arg and get function pointer from vtable into eax...]
  call  eax                     # returns pointer to temp var in eax
  mov   ecx, DWORD PTR [eax]    # reading result from temp var, 1st member
  mov   edx, DWORD PTR [eax+4]  # 2nd member of D2D1_SIZE_F
  mov   DWORD PTR _size$[ebp], ecx
  mov   DWORD PTR _size$[ebp+4], edx

C:
  ID2D1HwndRenderTarget *hwndRenderTarget;
  D2D1_SIZE_F size;
  size = hwndRenderTarget->lpVtbl->Base.GetSize(
      (ID2D1RenderTarget *)hwndRenderTarget);
Code generated by cl:
  [...push THIS arg and get function pointer into ecx...]
  call  ecx
  mov   DWORD PTR _size$[ebp], eax    # reading result from edx:eax registers
  mov   DWORD PTR _size$[ebp+4], edx

The C++ version works well; the C version crashes at the GetSize() call.
They're using incompatible aggregate return mechanisms (despite stdcall), but
since the d2d1.dll (which, I assume, implements this interface) will only use
one of the two mechanisms, the other won't work. Obviously d2d1.dll uses the
C++ version because that's what works.

Assuming this is part of MS ABI and not a bug in the MS compiler, then the
problem with gcc is that its aggregate return for C++ methods differs from MS
ABI (though it's correct for normal C-style functions).

[Bug target/64384] mingw-w64: stdcall function returning an aggregate is incompatible with MS ABI

[mizvekov at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64384

Matheus Izvekov <mizvekov at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |mizvekov at gmail dot com

--- Comment #7 from Matheus Izvekov <mizvekov at gmail dot com> ---
So this is still a problem in GCC 10.1.

Per
https://docs.microsoft.com/en-us/cpp/build/x64-calling-convention?view=vs-2019
What GCC does would be correct according to that spec. But in practice, MSVC
does different, and windows DLLs expect what MSVC does.

So I have stumbled on this problem with D3D12, calling
GetCPUDescriptorHandleForHeapStart method on ID3D12DescriptorHeap object.

Here is definition of this method from SDK:

```
virtual D3D12_CPU_DESCRIPTOR_HANDLE STDMETHODCALLTYPE 
GetCPUDescriptorHandleForHeapStart( void) = 0;
```

Returned aggregate is just a boxed SIZE_T:
```
typedef struct D3D12_CPU_DESCRIPTOR_HANDLE { SIZE_T ptr; }  
D3D12_CPU_DESCRIPTOR_HANDLE;
```

So it should be possible to return on register, but instead here is code from
that method (MASM syntax, taken from D3D12.dll):
```
CDescriptorHeap::GetCPUDescriptorHandleForHeapStart:
00007FF83AD6A3F0  mov         rax,qword ptr [rcx+0D0h]  
00007FF83AD6A3F7  mov         qword ptr [rdx],rax
00007FF83AD6A3FA  mov         rax,rdx  
00007FF83AD6A3FD  ret  
```

[Bug target/64384] mingw-w64: stdcall function returning an aggregate is incompatible with MS ABI

[alvinhochun at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64384

Alvin Wong <alvinhochun at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |alvinhochun at gmail dot com

--- Comment #8 from Alvin Wong <alvinhochun at gmail dot com> ---
According to https://devblogs.microsoft.com/oldnewthing/20220113-00/?p=106152,
Microsoft acknowledged that the behaviour is different between free function
and C++ member functions, and that it does cause breakage of the COM ABI
between C and C++. What they have done is to update the MIDL compiler to
generate C bindings compatible with the C++ ABI. (I filed
https://bugs.winehq.org/show_bug.cgi?id=54226 for implementing the same
function in widl.)

In this sense, the current behaviour of gcc and g++ regarding free functions
should be correct and does not need fixing. Only the ABI for C++ class member
functions are wrong.

[Bug target/64384] mingw-w64: stdcall function returning an aggregate is incompatible with MS ABI

[alvinhochun at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64384

--- Comment #9 from Alvin Wong <alvinhochun at gmail dot com> ---
P.S. related to https://gcc.gnu.org/bugzilla/show_bug.cgi?id=52792