Re: [bug] mingw64-*-w64-win-iconv: Cannot open handle; convert to UTF-8

Re: [bug] mingw64-*-w64-win-iconv: Cannot open handle; convert to UTF-8

[sdbenique]

Just a follow-up to this issue.
It appears my test program *was* invalid, but I discovered why SDL wouldn't load properly.

As you can see in my initial bug report,  SDL was attempting to convert a command line from UCS-2-INTERNAL to UTF-8 using win-iconv. "C" (as my test program had) is *definitely* invalid input! 

After going through the win-iconv source, I realized that win-iconv does not interpret the encoding name  "UCS-2-INTERNAL" as being UTF-16. (Source here: https://github.com/win-iconv/win-iconv/blob/master/win_iconv.c#L950) 

Luckily, around line 167 of that same file, there are mappings from codpages to name, which are used by a conversion function. 

So I have added the "UCS-2-INTERNAL" string in the places shown above,  and everything works so far... no more SDL errors.

I have filed a pull-request with upstream development, which has moved from Google Code to github at: https://github.com/win-iconv/win-iconv.

I've tried googling "UCS-2-INTERNAL", but I'm not sure why no one else has hit this issue. :(
Hopefully this patch solves the issue.

Regards,
sdbenique

On Sat, 12 Mar 2016 20:14:30 +0100 (CET), <sdbenique@runbox.com> wrote:

> Hello,
> 
> I'm writing because I'm encountering a strange bug in cygwin's distributed
> x86_64-w64-mingw32 libraries. Specifically a problem with win-iconv.
> 
> -=System Info=-
> OS(s): Windows 7 Professional, Windows 10 Professional Edition
> Package(s): mingw64-{i686,x86_64}-win-iconv version: 0.0.6-2
> Cygwin: Cygwin64, Setup 2.873
> 
> 
> I first stared experiencing an issue with an SDL2 application I am developing.
> I've been revamping my build toolkit to take advantage of the better mingw64
> support in newer releases of Cygwin.
> 
> When I finally got my build system work with the host, build, target triplets,
> I decided to try building some mingw32 binaries of the application.
> Everything built fine, but for some reason the application always immediately
> quit upon being ran, with the following error message:
> 
>       "Fatal Error: Out of memory aborting".
> 
> This happens even on a machine with 16GB of memory, 10GB of it being free.
> The i686 build as well as the x86_64 build encounters this issue as well.
> 
> I downloaded the source code for mingw64-SDL2 and compiled it with
> debuginfo.
> 
> I narrowed down the issue to some code in in SDL_windows_main.c,
> which calls SDL_iconv_string.
> 
> Stepping into that function, the following executes:
> Breakpoint 2, SDL_iconv_string (tocode=0x4052bc <__dyn_tls_init_callback+684> "UTF-8",
>     fromcode=0x4052ad <__dyn_tls_init_callback+669> "UCS-2-INTERNAL", inbuf=0x2e2dd2 "C", inbytesleft=116)
>     at /usr/src/debug/mingw64-x86_64-SDL2-2.0.1-1/src/stdlib/SDL_iconv.c:863
> 
> 863         size_t retCode = 0;
> 865         cd = SDL_iconv_open(tocode, fromcode);
> 866         if (cd == (SDL_iconv_t) - 1) {
> 868             if (!tocode || !*tocode) {
> 871             if (!fromcode || !*fromcode) {
> 874             cd = SDL_iconv_open(tocode, fromcode);
> 876         if (cd == (SDL_iconv_t) - 1) {
> 877             return NULL;
> 
> WinMain (hInst=0x400000, hPrev=0x0, szCmdLine=0x2e3859 "", sw=10)
>     at /usr/src/debug/mingw64-x86_64-SDL2-2.0.1-1/src/main/windows/SDL_windows_main.c:164
> 164         if (cmdline == NULL) {
> (gdb)
> 165             return OutOfMemory();
> 
> At first I thought it could be a bug with SDL, but to make sure, I created a very simple reproduction
> of the issue using only the iconv libraray, and a simple main() function. This "test" fails
> on every machine on which I have ran it, in both 32-bit and 64-bit builds.
> 
> #include <iconv.h>
> #include <stdio.h>
> #include <stdint.h>
> 
> // Check GCC
> #if __GNUC__
> #if __x86_64__ || __ppc64__
> #define PTR_T int64_t
> #else
> #define PTR_T int32_t
> #endif
> #endif
> 
> int main(int argc, char * args[])
> {
>     iconv_t handle = iconv_open("C", "UTF-8");
> 
>     if ((PTR_T) handle == -1)
>     {
>         printf("Could not open handle to iconv");
>     }
> 
>     return 0;
> }
> 
> I've uploaded a small repository with a Makefile that will conveniently set your $PATH
> correctly and launch the .exe when you run build the target 'run'.
> 
> For your convenience in recreating the issue, I have uploaded the repository on github,
> at the following URL.
> 
> git clone git://github.com/bittwiddler1/mingw64-iconv-test
> 
> I've never really submitted a bug report via mailing list, but please let me know if you need any
> other information and I will try my best to help out any way I can! 
> 
> I'll try and get some debug information out of the iconv.dll library, but no promises. I
> don't know much about text encoding, let alone unicode pages and whatnot. :)
> 
>  - sdbenique



--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: [bug] mingw64-*-w64-win-iconv: Cannot open handle; convert to UTF-8

[Yaakov Selkowitz]

On 2016-03-12 13:14, sdbenique@runbox.com wrote:
> At first I thought it could be a bug with SDL, but to make sure, I created a very simple reproduction
> of the issue using only the iconv libraray, and a simple main() function. This "test" fails
> on every machine on which I have ran it, in both 32-bit and 64-bit builds.
[snip]
>      iconv_t handle = iconv_open("C", "UTF-8");

Invalid code.  "C" is a locale, not an encoding.

-- 
Yaakov

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

[bug] mingw64-*-w64-win-iconv: Cannot open handle; convert to UTF-8

[sdbenique]

Hello,

I'm writing because I'm encountering a strange bug in cygwin's distributed
x86_64-w64-mingw32 libraries. Specifically a problem with win-iconv.

-=System Info=-
OS(s): Windows 7 Professional, Windows 10 Professional Edition
Package(s): mingw64-{i686,x86_64}-win-iconv version: 0.0.6-2
Cygwin: Cygwin64, Setup 2.873


I first stared experiencing an issue with an SDL2 application I am developing.
I've been revamping my build toolkit to take advantage of the better mingw64
support in newer releases of Cygwin.

When I finally got my build system work with the host, build, target triplets,
I decided to try building some mingw32 binaries of the application.
Everything built fine, but for some reason the application always immediately
quit upon being ran, with the following error message:

      "Fatal Error: Out of memory aborting".

This happens even on a machine with 16GB of memory, 10GB of it being free.
The i686 build as well as the x86_64 build encounters this issue as well.

I downloaded the source code for mingw64-SDL2 and compiled it with
debuginfo.

I narrowed down the issue to some code in in SDL_windows_main.c,
which calls SDL_iconv_string.

Stepping into that function, the following executes:
Breakpoint 2, SDL_iconv_string (tocode=0x4052bc <__dyn_tls_init_callback+684> "UTF-8",
    fromcode=0x4052ad <__dyn_tls_init_callback+669> "UCS-2-INTERNAL", inbuf=0x2e2dd2 "C", inbytesleft=116)
    at /usr/src/debug/mingw64-x86_64-SDL2-2.0.1-1/src/stdlib/SDL_iconv.c:863

863         size_t retCode = 0;
865         cd = SDL_iconv_open(tocode, fromcode);
866         if (cd == (SDL_iconv_t) - 1) {
868             if (!tocode || !*tocode) {
871             if (!fromcode || !*fromcode) {
874             cd = SDL_iconv_open(tocode, fromcode);
876         if (cd == (SDL_iconv_t) - 1) {
877             return NULL;

WinMain (hInst=0x400000, hPrev=0x0, szCmdLine=0x2e3859 "", sw=10)
    at /usr/src/debug/mingw64-x86_64-SDL2-2.0.1-1/src/main/windows/SDL_windows_main.c:164
164         if (cmdline == NULL) {
(gdb)
165             return OutOfMemory();

At first I thought it could be a bug with SDL, but to make sure, I created a very simple reproduction
of the issue using only the iconv libraray, and a simple main() function. This "test" fails
on every machine on which I have ran it, in both 32-bit and 64-bit builds.

#include <iconv.h>
#include <stdio.h>
#include <stdint.h>

// Check GCC
#if __GNUC__
#if __x86_64__ || __ppc64__
#define PTR_T int64_t
#else
#define PTR_T int32_t
#endif
#endif

int main(int argc, char * args[])
{
    iconv_t handle = iconv_open("C", "UTF-8");

    if ((PTR_T) handle == -1)
    {
        printf("Could not open handle to iconv");
    }

    return 0;
}

I've uploaded a small repository with a Makefile that will conveniently set your $PATH
correctly and launch the .exe when you run build the target 'run'.

For your convenience in recreating the issue, I have uploaded the repository on github,
at the following URL.

git clone git://github.com/bittwiddler1/mingw64-iconv-test

I've never really submitted a bug report via mailing list, but please let me know if you need any
other information and I will try my best to help out any way I can! 

I'll try and get some debug information out of the iconv.dll library, but no promises. I
don't know much about text encoding, let alone unicode pages and whatnot. :)

 - Saul Beniquez


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple