Help with setting Cygwin's Emacs W32 TRAMP to WSL 1

Help with setting Cygwin's Emacs W32 TRAMP to WSL 1

[Oleksandr Gavenko]

I'm using Cygwin for two reasons: mintty + Emacs w32.

Nowadays WSL 1 has become important (vendors provide ready to work
.deb packages,
 I use: Ansible + Google Cloud SDK, if name any).

Still WSL 1 lacks UI and integrates less smoothly into my workflow
to replace Cygwin's amazing Emacs W32.

WSL1 files are "hidden" for regular access.

Emacs has TRAMP mode to access foreign environments:
https://www.gnu.org/software/tramp/

I tried to define configuration that should have allowed to access WSL 1
from Cygwin's Emacs via "wsl sh" jump (the way TRAMP support "su" &
"sudo" jumps):

* https://www.reddit.com/r/emacs/comments/jrkgmy/tramp_for_wsl_1_from_cygwin_emacsw32/
* https://github.com/microsoft/WSL/issues/6236
  - wsl.exe periodically sends control characters making impossible "piping"
  via interactive/terminal mode

The problem is that when Cygwin's Emacs invokes "wsl.exe" conhost or
wsl?? thinks
it is in a PTY mode and sends different "clearing" terminal escape
sequences (see WSL issue report).

I reproduce garbage with Elisp code:

(setq xxx (make-process :name "xxx" :buffer "xxx" :command '("wsl"
"echo" "OK") :connection-type 'pty))
(stop-process xxx)

Note that changing the last argument "pty" => "pipe" solves the problem,
but things are a bit complicated...

TRAMP mode has been implemented via call to "sh" by internal API "make-process".
And then TRAMP sends  "wsl ..." to the shell so WSL thinks it is in
PTY and I have no way
to influence that decision ((

I tried stupid tricks like passing everything related to "ws.exe" via
pipe to "cat":

  (with-eval-after-load 'tramp
    (setf (alist-get "wsl" tramp-methods nil nil #'equal)
          '((tramp-login-program "sh")
            (tramp-login-args (("-c") ("wsl") ("-u" "%u") ("-d" "%h")
("|" "cat")))
            (tramp-remote-shell "/bin/sh")
            (tramp-remote-shell-login ("-l"))
            (tramp-remote-shell-args ("-c"))
            (tramp-connection-timeout 10)
            (tramp-session-timeout 300))))

or define a  wrapper script that has "wsl ... | cat"  inside. Nothing helped.
Few experiments with Cygwin's Expect also failed as I forgot
everything about that obscure tech...

So I need to write some wrapper that will bidirectionally pipe to "wsl.exe"
putting wsl into piped instead TTY mode.

It can be that there is a ready solution.

If not please advise which API should I use to write such  a wrapper.

Should it be Cygwin's pipe/dup2/fork/execvp or popen?

Or should it be Windows native CreateChildProcess / WriteToPipe /
ReadFromPipe as here:
https://docs.microsoft.com/en-us/windows/win32/procthread/creating-a-child-process-with-redirected-input-and-output

Please CC me in replies, I'll check the mail archive after a week to
not lose any responses.

Re: Help with setting Cygwin's Emacs W32 TRAMP to WSL 1

[Eliot Moss]

On 11/26/2020 10:18 AM, Oleksandr Gavenko via Cygwin wrote:

 > Still WSL 1 lacks UI and integrates less smoothly into my workflow
 > to replace Cygwin's amazing Emacs W32.

Lacks UI?  You can run X windows applications in WSL if you have the Cygwin X server running :-) ... 
then you get the X UI.  I can certainly run xterm and emacs that way.

Maybe I am misunderstanding you?

Best wishes - Eliot Moss

Re: Help with setting Cygwin's Emacs W32 TRAMP to WSL 1

[Oleksandr Gavenko]

On 2020-11-26, Eliot Moss wrote:

> Lacks UI?  You can run X windows applications in WSL if you have the Cygwin X server running :-) ...
> then you get the X UI.  I can certainly run xterm and emacs that way.
>
I don't need 2 Emacs instances around. Cygwin's Emacs W32 (with native W32 GUI
widgets!) is the only my Emacs on Windows.

I run Emacs server so a visited file list is kept in a one place... It is a
part of my workflow (like quick switching to earlier visited files by fuzzy
name matching/etc).

And not to mention it is extra mental load to pass that DISPLAY=:0 or whatever
around...

I believe that X server under Windows is only to write cool blog posts, that
we are able to run "xterm" ))

Thanks god Cygwin is able to mount WSL's roots via P9 network file system.

PS. As you haven't CCed me I had to check GNUS + Gmane again. Haven't read
news for 3 years, found out that Lars Ingebrigtsen fucked up with domain
transfer.

-- 
http://defun.work/

Re: Help with setting Cygwin's Emacs W32 TRAMP to WSL 1

[Eliot Moss]

On 11/26/2020 3:38 PM, Oleksandr Gavenko via Cygwin wrote:

> I believe that X server under Windows is only to write cool blog posts, that
> we are able to run "xterm" ))

Not sure what you mean here.  I run Cygwin's X server and run xterm/bash windows and Emacs that way. 
  I was just indicating that (provided DISPLAY=:0.0 is set, as you pointed out) WSL X apps play fine 
with the Cygwin X server.

> Thanks god Cygwin is able to mount WSL's roots via P9 network file system.

Not sure what P9 is :-) ..

> PS. As you haven't CCed me I had to check GNUS + Gmane again. Haven't read
> news for 3 years, found out that Lars Ingebrigtsen fucked up with domain
> transfer.

??  When people post to Cygwin, I assume that are also subscribed and will get a response from 
there, and don't need _two_ emails.  But I am sending this one both places.  I also have not read 
net news in years - shouldn't be necessary if you're willing to receive Cygwin notes by email.

Cheers - Eliot

Re: Help with setting Cygwin's Emacs W32 TRAMP to WSL 1

[Oleksandr Gavenko]

On 2020-11-27, Eliot Moss wrote:

> On 11/26/2020 3:38 PM, Oleksandr Gavenko via Cygwin wrote:
>
>> I believe that X server under Windows is only to write cool blog posts, that
>> we are able to run "xterm" ))
>
> Not sure what you mean here.  I run Cygwin's X server and run xterm/bash windows and Emacs that way.

On daily basis? Why not mintty and emacs-w32?

Originally (2006-2012) I used native Windows Emacs + cygwin-mount.el. There
were compatibility issues, most struggles came from Emacs dependency on
external utilities to be fully functional (like search across files). Cygwin
environment gave those utilities.

Later in 2016 I had to work with Windows again and I discovered:

https://cygwin.com/packages/summary/emacs-w32.html
  This package provides emacs binaries that use the native Windows GUI for display.

Hooray! Good bye "cygwin-mount.el"!

I maintain "e" command to open everything in Emacs:
http://hg.defun.work/utils/file/tip/emacs

It simplifies jump from Windows CLI to Cygwin Emacs.

And recently Gradle 5.2 fixed escape sequence handling in Mintty, making it
great again:

https://docs.gradle.org/5.2/release-notes.html
  Gradle now detects when it is running from Mintty on Windows and enables the
  rich console. Mintty is a popular terminal emulator used by projects such as
  Cygwin and Git for Windows.

> I was just indicating that (provided DISPLAY=:0.0 is set, as you pointed out) WSL X apps play fine 
> with the Cygwin X server.
>
OK. No problem. I was surprised that you are using X Window from Cygwin for
xterm + Emacs when there are stronger alternatives. At least you don't need to
deal with env var DISPLAY.

For "mintty" there is an issue with TERM=mintty which is not supported on most
remote hosts. I fixed it in ~/.bashrc recently by:

case $TERM in
  mintty*) alias ssh='TERM=xterm ssh' ;;
esac

Note that I need TERM=mintty for Gradle to work properly. With TERM=xterm
Gradle sends unprocessible garbage to terminal...

>> Thanks god Cygwin is able to mount WSL's roots via P9 network file system.
>
> Not sure what P9 is :-) ..
>
Here it is:

https://devblogs.microsoft.com/commandline/whats-new-for-wsl-in-windows-10-version-1903/

>> PS. As you haven't CCed me I had to check GNUS + Gmane again. Haven't read
>> news for 3 years, found out that Lars Ingebrigtsen fucked up with domain
>> transfer.
>
> ??  When people post to Cygwin, I assume that are also subscribed and will get a response from
> there, and don't need _two_ emails.  But I am sending this one both places.  I also have not read 
> net news in years - shouldn't be necessary if you're willing to receive Cygwin notes by email.
>
I planned to read email archive week later to see responses as I didn't want to
subscribe to any mail list.

So:

* I will know about responses a week later.

* I wouldn't reply to any respond as I have to retrieve/inject Message-Id to
  maintain message chain. For this I used MarkMail WEB form or some other
  offer in the past but experience was confusing.

As I understand it is not possible to specify me as additional recipient via
headers To/Cc/Bcc/Replay-To if you are in a land of email lists.

I remember some articles recommended to ask about inclusion of my email in
addition to list address - the only solution we have ((

Now that I found Gmane bridge working again (I had to alter Gmane connection
endpoint to "news.gmane.io") at least I can reply without issues when not
subscribed to the list!

-- 
http://defun.work/

Re: Help with setting Cygwin's Emacs W32 TRAMP to WSL 1

[Henry S. Thompson]

Oleksandr Gavenko via Cygwin writes:

> ...
> WSL1 files are "hidden" for regular access.

I probably misunderstand, but I can see my WSL/Ubuntu files via this
path from Cygwin:

 /c/Users/ht/AppData/Local/Packages/CanonicalGroupLimited.../LocalState/rootfs

ht
-- 
       Henry S. Thompson, School of Informatics, University of Edinburgh
      10 Crichton Street, Edinburgh EH8 9AB, SCOTLAND -- (44) 131 650-4440
                Fax: (44) 131 650-4587, e-mail: ht@inf.ed.ac.uk
                       URL: http://www.ltg.ed.ac.uk/~ht/
 [mail from me _always_ has a .sig like this -- mail without it is forged spam]

The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.

Re: Help with setting Cygwin's Emacs W32 TRAMP to WSL 1

[Oleksandr Gavenko]

On Thu, Nov 26, 2020 at 8:42 PM Henry S. Thompson <ht@inf.ed.ac.uk> wrote:
>
> Oleksandr Gavenko via Cygwin writes:
>
> > ...
> > WSL1 files are "hidden" for regular access.
>
> I probably misunderstand, but I can see my WSL/Ubuntu files via this
> path from Cygwin:
>
>  /c/Users/ht/AppData/Local/Packages/CanonicalGroupLimited.../LocalState/rootfs
>
You are not allowed to edit them from the "Windows" process. Only from LXSS:

https://devblogs.microsoft.com/commandline/do-not-change-linux-files-using-windows-apps-and-tools/

> DO NOT, under ANY circumstances, access, create, and/or modify Linux files inside
> of your `%LOCALAPPDATA%` folder using Windows apps, tools, scripts, consoles, etc.

You'll face:

https://github.com/microsoft/WSL/issues/1524
changes made in Windows File System - not seen in Bash

Though I forgot about their announce of P9 fileserver (dated May 2019
Windows 10 version 1903):

> WSL hosts a new 9P fileserver, which exposes distro filesystems to Windows apps and tools via \\wsl$\<DistroName>\!

Details are here:

https://devblogs.microsoft.com/commandline/whats-new-for-wsl-in-windows-10-version-1903/

I tried in elevated cmd:

cmd# cd \\wsl$\debian\
'\\wsl$\debian\'
CMD does not support UNC paths as current directories.

cmd# net use x: \\wsl$\debian\
System error 67 has occurred.
The network name cannot be found.

It is only available in Explorer and some "cool" editors, like Code.

I do not know if Cygwin allows to "mount" that P9 fileserver...

There is an example of network mounting:
https://cygwin.com/cygwin-ug-net/mount.html

so I tried it:

bash# mount '//wsl$/debian' /mnt
mount: defaulting to 'notexec' mount option for speed since native path
       references a remote share.  Use '-f' option to override.

bash# ls /mnt
ls: cannot access '/mnt/initrd.img': Input/output error
ls: cannot access '/mnt/initrd.img.old': Input/output error
ls: cannot access '/mnt/vmlinuz': Input/output error
ls: cannot access '/mnt/vmlinuz.old': Input/output error
bin  boot  c  d  dev  etc  home  init  initrd.img  initrd.img.old  lib
 lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr
var  vmlinuz  vmlinuz.old

Cool! Let's make it permanent.

$ cat /etc/fstab

//wsl$/debian /wsl/debian      ntfs binary,noacl,posix=0 0 0
//wsl$/ubuntu /wsl/ubuntu      ntfs binary,noacl,posix=0 0 0
//wsl$/alpine /wsl/alpine      ntfs binary,noacl,posix=0 0 0

$ mkdir -p /wsl/{ubuntu,debian,alpine}
$ mount /wsl/debian
$ mount /wsl/ubuntu
$ mount /wsl/wsl

And the final step it testing with Cygwin's Emacs W32:

I can read files, but on save I've got:

Saving file /wsl/debian/opt/sa-batch-am-dbsync/run.env...
basic-save-buffer-2: Unlocking file: Invalid argument,
/wsl/debian/opt/sa-batch-am-dbsync/run.env

Trying in Cygwin's bash:

bash# cd /wsl/debian/home/user/
bash# touch test.txt
bash# ls -l test.txt
-rw-r--r-- 1 user user 0 2020-11-26 21:18 test.txt

bash# echo 123 >test.txt
bash# echo 456 >>test.txt
bash# cat test.txt
123
456

So Emacs tries to make some "smart" locking on dumb FS... Need to
waste another few hours to make Emacs work.

At least I have something!

Re: Help with setting Cygwin's Emacs W32 TRAMP to WSL 1

[Oleksandr Gavenko]

On Thu, Nov 26, 2020 at 9:22 PM Oleksandr Gavenko <gavenkoa@gmail.com> wrote:
> So Emacs tries to make some "smart" locking on dumb FS... Need to
> waste another few hours to make Emacs work.
>
Found the problem: Emacs fails on its "unlock-buffer" call from
emacs/src/filelock.c.

Feature described here:

https://www.gnu.org/software/emacs/manual/html_node/elisp/File-Locks.html

To prevent of special symlink creation I disabled that feature:

(setq create-lockfiles nil)

So no need to waste time on TRAMP mode if it is supported natively by Cygwin!

Thx to Henry for giving me the right spin!