[Bug libc/6050] iconv(1) buffers all of stdin in memory

[Bug libc/6050] iconv(1) buffers all of stdin in memory

[schwab@linux-m68k.org]

http://sourceware.org/bugzilla/show_bug.cgi?id=6050

Andreas Schwab <schwab@linux-m68k.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         AssignedTo|drepper.fsp at gmail dot    |unassigned at sourceware
                   |com                         |dot org

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/6050] iconv(1) buffers all of stdin in memory

[bugdal at aerifal dot cx]

http://sourceware.org/bugzilla/show_bug.cgi?id=6050

Rich Felker <bugdal at aerifal dot cx> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |bugdal at aerifal dot cx

--- Comment #1 from Rich Felker <bugdal at aerifal dot cx> ---
Ping. This bug still exists. I traced it to a comment in the source:

/* we have a problem with reading from a desriptor since we must not
   provide the iconv() function an incomplete character or shift
   sequence at the end of the buffer.  Since we have to deal with
   arbitrary encodings we must read the whole text in a buffer and
   process it in one step.  */

See
http://sourceware.org/git/?p=glibc.git;a=blob;f=iconv/iconv_prog.c;h=1a1d0d0cf45c0d747a8090bc234addd9e49f1ba7;hb=HEAD#l561

The claims made in the comment are simply erroneous. Per POSIX, the iconv
function returns (size_t)-1 with errno set to EINVAL to indicate "Input
conversion stopped due to an incomplete character or shift sequence at the end
of the input buffer." This is a different condition from EILSEQ, and thus the
caller can detect and recover from it simply by moving the remaining bytes of
the input buffer to the beginning, re-filling the buffer, and calling iconv
again.

If glibc's iconv function does not support this behavior correctly, that's a
library-level bug which should be filed separately and fixed.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

Re: [Bug libc/6050] iconv(1) buffers all of stdin in memory

[Ondřej Bílka]

On Sat, Aug 10, 2013 at 04:01:22PM +0000, bugdal at aerifal dot cx wrote:
> http://sourceware.org/bugzilla/show_bug.cgi?id=6050
> 
> Rich Felker <bugdal at aerifal dot cx> changed:
> 
>            What    |Removed                     |Added
> ----------------------------------------------------------------------------
>                  CC|                            |bugdal at aerifal dot cx
> 
> --- Comment #1 from Rich Felker <bugdal at aerifal dot cx> ---
> Ping. This bug still exists. I traced it to a comment in the source:
> 
> /* we have a problem with reading from a desriptor since we must not
>    provide the iconv() function an incomplete character or shift
>    sequence at the end of the buffer.  Since we have to deal with
>    arbitrary encodings we must read the whole text in a buffer and
>    process it in one step.  */
> 
> See
> http://sourceware.org/git/?p=glibc.git;a=blob;f=iconv/iconv_prog.c;h=1a1d0d0cf45c0d747a8090bc234addd9e49f1ba7;hb=HEAD#l561
> 
> The claims made in the comment are simply erroneous. Per POSIX, the iconv
> function returns (size_t)-1 with errno set to EINVAL to indicate "Input
> conversion stopped due to an incomplete character or shift sequence at the end
> of the input buffer." This is a different condition from EILSEQ, and thus the
> caller can detect and recover from it simply by moving the remaining bytes of
> the input buffer to the beginning, re-filling the buffer, and calling iconv
> again.
>
This would work only for stateless encodings. You cannot do this with
ISO-2022-JP as you would need additional argument to save state.

[Bug libc/6050] iconv(1) buffers all of stdin in memory

[neleai at seznam dot cz]

http://sourceware.org/bugzilla/show_bug.cgi?id=6050

--- Comment #2 from Ondrej Bilka <neleai at seznam dot cz> ---
On Sat, Aug 10, 2013 at 04:01:22PM +0000, bugdal at aerifal dot cx wrote:
> http://sourceware.org/bugzilla/show_bug.cgi?id=6050
> 
> Rich Felker <bugdal at aerifal dot cx> changed:
> 
>            What    |Removed                     |Added
> ----------------------------------------------------------------------------
>                  CC|                            |bugdal at aerifal dot cx
> 
> --- Comment #1 from Rich Felker <bugdal at aerifal dot cx> ---
> Ping. This bug still exists. I traced it to a comment in the source:
> 
> /* we have a problem with reading from a desriptor since we must not
>    provide the iconv() function an incomplete character or shift
>    sequence at the end of the buffer.  Since we have to deal with
>    arbitrary encodings we must read the whole text in a buffer and
>    process it in one step.  */
> 
> See
> http://sourceware.org/git/?p=glibc.git;a=blob;f=iconv/iconv_prog.c;h=1a1d0d0cf45c0d747a8090bc234addd9e49f1ba7;hb=HEAD#l561
> 
> The claims made in the comment are simply erroneous. Per POSIX, the iconv
> function returns (size_t)-1 with errno set to EINVAL to indicate "Input
> conversion stopped due to an incomplete character or shift sequence at the end
> of the input buffer." This is a different condition from EILSEQ, and thus the
> caller can detect and recover from it simply by moving the remaining bytes of
> the input buffer to the beginning, re-filling the buffer, and calling iconv
> again.
>
This would work only for stateless encodings. You cannot do this with
ISO-2022-JP as you would need additional argument to save state.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/6050] iconv(1) buffers all of stdin in memory

[bugdal at aerifal dot cx]

http://sourceware.org/bugzilla/show_bug.cgi?id=6050

--- Comment #3 from Rich Felker <bugdal at aerifal dot cx> ---
On Mon, Oct 07, 2013 at 01:58:09PM +0000, neleai at seznam dot cz wrote:
> This would work only for stateless encodings. You cannot do this with
> ISO-2022-JP as you would need additional argument to save state.

No, the state is saved in the conversion descriptor, and iconv(3)
reports to the caller the exact point at which it stopped in the
input, so you simply resume from that point.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/6050] iconv(1) buffers all of stdin in memory

[neleai at seznam dot cz]

https://sourceware.org/bugzilla/show_bug.cgi?id=6050

Ondrej Bilka <neleai at seznam dot cz> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |neleai at seznam dot cz

--- Comment #4 from Ondrej Bilka <neleai at seznam dot cz> ---
ah, then it is ok, could you write a patch?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug locale/6050] iconv(1) buffers all of stdin in memory

[jsm28 at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=6050

Joseph Myers <jsm28 at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
          Component|libc                        |locale

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug locale/6050] iconv(1) buffers all of stdin in memory

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=6050

Florian Weimer <fweimer at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |FIXED
                 CC|                            |fweimer at redhat dot com
   Target Milestone|---                         |2.41
           Assignee|unassigned at sourceware dot org   |fweimer at redhat dot com
             Status|NEW                         |RESOLVED

--- Comment #6 from Florian Weimer <fweimer at redhat dot com> ---
Fixed for 2.41 via:

commit fa1b0d5e9f6e0353e16339430770a7a8824c0468
Author: Florian Weimer <fweimer@redhat.com>
Date:   Fri Sep 20 13:10:54 2024 +0200

    iconv: Input buffering for the iconv program (bug 6050)

    Do not read the entire input file into memory.

    Reviewed-by: DJ Delorie <dj@redhat.com>

-- 
You are receiving this mail because:
You are on the CC list for the bug.