issue when piping from a windows program

issue when piping from a windows program

[Leonid Mironov]

I am trying to feed the output of wmic.exe - a windows console program, to cygwin bash script. 
wmic.exe produces UTF16LE output with BOM and CR/LFs, so I am using dos2unix to convert it. 
The problem is that when I write wmic.exe output to a file and then use dos2unix to convert this file 
I get the expected result - ASCII file with LFs, I get the same result when I pipe this file to dos2unix, 
but when I pipe wmic.exe output directly to dos2unix I get ASCII file with CR/LFs and an extra empty line. 
Cygwin is up to date, windows 10. What gives?

Here are the hexdumps

of 'wmic /NAMESPACE:\\\\root\\WMI PATH BatteryStatus get charging,voltage,remainingcapacity,chargerate>file'

00000000  ff fe 43 00 68 00 61 00  72 00 67 00 65 00 52 00  |..C.h.a.r.g.e.R.|
00000010  61 00 74 00 65 00 20 00  20 00 43 00 68 00 61 00  |a.t.e. . .C.h.a.|
00000020  72 00 67 00 69 00 6e 00  67 00 20 00 20 00 52 00  |r.g.i.n.g. . .R.|
00000030  65 00 6d 00 61 00 69 00  6e 00 69 00 6e 00 67 00  |e.m.a.i.n.i.n.g.|
00000040  43 00 61 00 70 00 61 00  63 00 69 00 74 00 79 00  |C.a.p.a.c.i.t.y.|
00000050  20 00 20 00 56 00 6f 00  6c 00 74 00 61 00 67 00  | . .V.o.l.t.a.g.|
00000060  65 00 20 00 20 00 0d 00  0a 00 30 00 20 00 20 00  |e. . .....0. . .|
00000070  20 00 20 00 20 00 20 00  20 00 20 00 20 00 20 00  | . . . . . . . .|
00000080  20 00 46 00 41 00 4c 00  53 00 45 00 20 00 20 00  | .F.A.L.S.E. . .|
00000090  20 00 20 00 20 00 33 00  37 00 37 00 33 00 34 00  | . . .3.7.7.3.4.|
000000a0  20 00 20 00 20 00 20 00  20 00 20 00 20 00 20 00  | . . . . . . . .|
000000b0  20 00 20 00 20 00 20 00  20 00 20 00 31 00 32 00  | . . . . . .1.2.|
000000c0  37 00 34 00 30 00 20 00  20 00 20 00 20 00 0d 00  |7.4.0. . . . ...|
000000d0  0a 00                                             |..|
000000d2

of wimic>file followed by dos2unix<file or cat file|dos2unix

00000000  43 68 61 72 67 65 52 61  74 65 20 20 43 68 61 72  |ChargeRate  Char|
00000010  67 69 6e 67 20 20 52 65  6d 61 69 6e 69 6e 67 43  |ging  RemainingC|
00000020  61 70 61 63 69 74 79 20  20 56 6f 6c 74 61 67 65  |apacity  Voltage|
00000030  20 20 0a 30 20 20 20 20  20 20 20 20 20 20 20 46  |  .0           F|                                                              
00000040  41 4c 53 45 20 20 20 20  20 33 37 37 33 34 20 20  |ALSE     37734  |                                                              
00000050  20 20 20 20 20 20 20 20  20 20 20 20 31 32 37 34  |            1274|
00000060  30 20 20 20 20 0a                                 |0    .|
00000066

and of wimic|dos2unix

00000000  43 68 61 72 67 65 52 61  74 65 20 20 43 68 61 72  |ChargeRate  Char|
00000010  67 69 6e 67 20 20 52 65  6d 61 69 6e 69 6e 67 43  |ging  RemainingC|
00000020  61 70 61 63 69 74 79 20  20 56 6f 6c 74 61 67 65  |apacity  Voltage|
00000030  20 20 0d 0a 30 20 20 20  20 20 20 20 20 20 20 20  |  ..0           |
00000040  46 41 4c 53 45 20 20 20  20 20 33 37 37 33 34 20  |FALSE     37734 |
00000050  20 20 20 20 20 20 20 20  20 20 20 20 20 31 32 37  |             127|
00000060  34 30 20 20 20 20 0d 0a  0d 0a                    |40    ....|
0000006a

 
 

Re: issue when piping from a windows program

[Marco Atzeri]

On 11.04.2023 09:15, Leonid Mironov via Cygwin wrote:
> I am trying to feed the output of wmic.exe - a windows console program, to cygwin bash script.
> wmic.exe produces UTF16LE output with BOM and CR/LFs, so I am using dos2unix to convert it.
> The problem is that when I write wmic.exe output to a file and then use dos2unix to convert this file
> I get the expected result - ASCII file with LFs, I get the same result when I pipe this file to dos2unix,
> but when I pipe wmic.exe output directly to dos2unix I get ASCII file with CR/LFs and an extra empty line.
> Cygwin is up to date, windows 10. What gives?
> 
> Here are the hexdumps
> 
> of 'wmic /NAMESPACE:\\\\root\\WMI PATH BatteryStatus get charging,voltage,remainingcapacity,chargerate>file'
> 

use "iconv" to change the codification and than d2u for the line termination


$ iconv -f UTF16 -t UTF8 file
ChargeRate  Charging  RemainingCapacity  Voltage
0           FALSE     0                  0

$ iconv -f UTF16 -t UTF8 file |od -c
0000000   C   h   a   r   g   e   R   a   t   e           C   h   a   r
0000020   g   i   n   g           R   e   m   a   i   n   i   n   g   C
0000040   a   p   a   c   i   t   y           V   o   l   t   a   g   e
0000060          \r  \n   0
0000100   F   A   L   S   E                       0
0000120                                                       0
0000140                          \r  \n
0000150

$ iconv -f UTF16 -t UTF8 file |d2u | od -c
0000000   C   h   a   r   g   e   R   a   t   e           C   h   a   r
0000020   g   i   n   g           R   e   m   a   i   n   i   n   g   C
0000040   a   p   a   c   i   t   y           V   o   l   t   a   g   e
0000060          \n   0                                               F
0000100   A   L   S   E                       0
0000120                                                   0
0000140                      \n
0000146

Re: issue when piping from a windows program

[Andrey Repin]

Greetings, Leonid Mironov!

> I am trying to feed the output of wmic.exe - a windows console program, to cygwin bash script.
> wmic.exe produces UTF16LE output with BOM and CR/LFs, so I am using dos2unix to convert it.
> The problem is that when I write wmic.exe output to a file and then use dos2unix to convert this file
> I get the expected result - ASCII file with LFs, I get the same result when I pipe this file to dos2unix,
> but when I pipe wmic.exe output directly to dos2unix I get ASCII file with CR/LFs and an extra empty line.
> Cygwin is up to date, windows 10. What gives?

You don't need d2u. Use `… | iconv -f UTF16LE | tr -d '\r'`
(Don't even need -t for iconv - it assumes current locale settings as
default source/destination encoding.)


-- 
With best regards,
Andrey Repin
Tuesday, April 11, 2023 17:41:32

Sorry for my terrible english...

Re: issue when piping from a windows program

[Brian Inglis]

On 2023-04-11 08:43, Andrey Repin via Cygwin wrote:
> Greetings, Leonid Mironov!
> 
>> I am trying to feed the output of wmic.exe - a windows console program, to cygwin bash script.
>> wmic.exe produces UTF16LE output with BOM and CR/LFs, so I am using dos2unix to convert it.
>> The problem is that when I write wmic.exe output to a file and then use dos2unix to convert this file
>> I get the expected result - ASCII file with LFs, I get the same result when I pipe this file to dos2unix,
>> but when I pipe wmic.exe output directly to dos2unix I get ASCII file with CR/LFs and an extra empty line.
>> Cygwin is up to date, windows 10. What gives?
> 
> You don't need d2u. Use `… | iconv -f UTF16LE | tr -d '\r'`
> (Don't even need -t for iconv - it assumes current locale settings as
> default source/destination encoding.)

d2u -ul converts from utf-16le to utf-8

When I run wmic under mintty or | less it looks fine to me e.g.:

	$ wmic baseboard list full | less	# optional

but the output could benefut from \r stripping:

	$ wmic baseboard list full | tr -d '\r' | cat -A | less

https://learn.microsoft.com/en-us/windows/whats-new/deprecated-features#deprecated-features

The WMIC tool is deprecated in Windows 10, version 21H1 and the 21H1 General 
Availability Channel release of Windows Server. This tool is superseded by 
Windows PowerShell for WMI.

https://learn.microsoft.com/en-us/powershell/scripting/learn/ps101/07-working-with-wmi

-- 
Take care. Thanks, Brian Inglis              Calgary, Alberta, Canada

La perfection est atteinte                   Perfection is achieved
non pas lorsqu'il n'y a plus rien à ajouter  not when there is no more to add
mais lorsqu'il n'y a plus rien à retirer     but when there is no more to cut
                                 -- Antoine de Saint-Exupéry

Re: issue when piping from a windows program

[Takashi Yano]

On Tue, 11 Apr 2023 09:15:03 +0200
Leonid Mironov wrote:
> I am trying to feed the output of wmic.exe - a windows console program, to cygwin bash script. 
> wmic.exe produces UTF16LE output with BOM and CR/LFs, so I am using dos2unix to convert it. 
> The problem is that when I write wmic.exe output to a file and then use dos2unix to convert this file 
> I get the expected result - ASCII file with LFs, I get the same result when I pipe this file to dos2unix, 
> but when I pipe wmic.exe output directly to dos2unix I get ASCII file with CR/LFs and an extra empty line. 
> Cygwin is up to date, windows 10. What gives?
> 
> Here are the hexdumps
> 
> of 'wmic /NAMESPACE:\\\\root\\WMI PATH BatteryStatus get charging,voltage,remainingcapacity,chargerate>file'
> 
> 00000000  ff fe 43 00 68 00 61 00  72 00 67 00 65 00 52 00  |..C.h.a.r.g.e.R.|
> 00000010  61 00 74 00 65 00 20 00  20 00 43 00 68 00 61 00  |a.t.e. . .C.h.a.|
> 00000020  72 00 67 00 69 00 6e 00  67 00 20 00 20 00 52 00  |r.g.i.n.g. . .R.|
> 00000030  65 00 6d 00 61 00 69 00  6e 00 69 00 6e 00 67 00  |e.m.a.i.n.i.n.g.|
> 00000040  43 00 61 00 70 00 61 00  63 00 69 00 74 00 79 00  |C.a.p.a.c.i.t.y.|
> 00000050  20 00 20 00 56 00 6f 00  6c 00 74 00 61 00 67 00  | . .V.o.l.t.a.g.|
> 00000060  65 00 20 00 20 00 0d 00  0a 00 30 00 20 00 20 00  |e. . .....0. . .|
> 00000070  20 00 20 00 20 00 20 00  20 00 20 00 20 00 20 00  | . . . . . . . .|
> 00000080  20 00 46 00 41 00 4c 00  53 00 45 00 20 00 20 00  | .F.A.L.S.E. . .|
> 00000090  20 00 20 00 20 00 33 00  37 00 37 00 33 00 34 00  | . . .3.7.7.3.4.|
> 000000a0  20 00 20 00 20 00 20 00  20 00 20 00 20 00 20 00  | . . . . . . . .|
> 000000b0  20 00 20 00 20 00 20 00  20 00 20 00 31 00 32 00  | . . . . . .1.2.|
> 000000c0  37 00 34 00 30 00 20 00  20 00 20 00 20 00 0d 00  |7.4.0. . . . ...|
> 000000d0  0a 00                                             |..|
> 000000d2
> 
> of wimic>file followed by dos2unix<file or cat file|dos2unix
> 
> 00000000  43 68 61 72 67 65 52 61  74 65 20 20 43 68 61 72  |ChargeRate  Char|
> 00000010  67 69 6e 67 20 20 52 65  6d 61 69 6e 69 6e 67 43  |ging  RemainingC|
> 00000020  61 70 61 63 69 74 79 20  20 56 6f 6c 74 61 67 65  |apacity  Voltage|
> 00000030  20 20 0a 30 20 20 20 20  20 20 20 20 20 20 20 46  |  .0           F|                                                              
> 00000040  41 4c 53 45 20 20 20 20  20 33 37 37 33 34 20 20  |ALSE     37734  |                                                              
> 00000050  20 20 20 20 20 20 20 20  20 20 20 20 31 32 37 34  |            1274|
> 00000060  30 20 20 20 20 0a                                 |0    .|
> 00000066
> 
> and of wimic|dos2unix
> 
> 00000000  43 68 61 72 67 65 52 61  74 65 20 20 43 68 61 72  |ChargeRate  Char|
> 00000010  67 69 6e 67 20 20 52 65  6d 61 69 6e 69 6e 67 43  |ging  RemainingC|
> 00000020  61 70 61 63 69 74 79 20  20 56 6f 6c 74 61 67 65  |apacity  Voltage|
> 00000030  20 20 0d 0a 30 20 20 20  20 20 20 20 20 20 20 20  |  ..0           |
> 00000040  46 41 4c 53 45 20 20 20  20 20 33 37 37 33 34 20  |FALSE     37734 |
> 00000050  20 20 20 20 20 20 20 20  20 20 20 20 20 31 32 37  |             127|
> 00000060  34 30 20 20 20 20 0d 0a  0d 0a                    |40    ....|
> 0000006a

This is not cygwin problem.

Please try the following in command prompt without cygwin.

wmic > wmic_redirect.txt
wmic | more > wmic_pipe.txt

Then, open both text file using binary editor.
wmic_redirect.txt has \r\n in UTF16 encoding, however,
you will find wmic_pipe.txt has abnormal line endings.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>