* Windows Server 2012R2 64bit and 32bit Cygwin sshd @ 2014-08-18 17:26 Achim Gratz 2014-08-19 13:37 ` Achim Gratz 2014-09-02 11:52 ` Achim Gratz 0 siblings, 2 replies; 24+ messages in thread From: Achim Gratz @ 2014-08-18 17:26 UTC (permalink / raw) To: cygwin I'm not making progress on this, but I've come to the conclusion that it is highly unlikely that the existence of the second (64bit) Cygwin installation has anything to do with the problem, first reported here: http://thread.gmane.org/gmane.os.cygwin/147823 Meanwhile I've tried to make sure that the two installations don't interact by creating a separate privileged user for both of them, but that's not made a difference. So let me re-phrase the question, given: - Windows Server 2012R2 64bit running on hardware - Cygwin 32bit (latest snapshot from 2014-08-18 w/ AD integration) can anybody confirm they have gotten sshd successfully running? As far as I can debug this, the sshd I have installed goes through all the motions of checking and setting up the environment, but any process it then spawns to actually run my command or shell immediately exits. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Factory and User Sound Singles for Waldorf Blofeld: http://Synth.Stromeko.net/Downloads.html#WaldorfSounds -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Windows Server 2012R2 64bit and 32bit Cygwin sshd 2014-08-18 17:26 Windows Server 2012R2 64bit and 32bit Cygwin sshd Achim Gratz @ 2014-08-19 13:37 ` Achim Gratz 2014-08-19 17:02 ` Corinna Vinschen 2014-09-02 11:52 ` Achim Gratz 1 sibling, 1 reply; 24+ messages in thread From: Achim Gratz @ 2014-08-19 13:37 UTC (permalink / raw) To: cygwin Achim Gratz <Stromeko <at> nexgo.de> writes: > I'm not making progress on this, but I've come to the conclusion that it > is highly unlikely that the existence of the second (64bit) Cygwin > installation has anything to do with the problem, first reported here: > > http://thread.gmane.org/gmane.os.cygwin/147823 > > Meanwhile I've tried to make sure that the two installations don't > interact by creating a separate privileged user for both of them, but > that's not made a difference. > > So let me re-phrase the question, given: > > - Windows Server 2012R2 64bit running on hardware > - Cygwin 32bit (latest snapshot from 2014-08-18 w/ AD integration) > > can anybody confirm they have gotten sshd successfully running? As far > as I can debug this, the sshd I have installed goes through all the > motions of checking and setting up the environment, but any process it > then spawns to actually run my command or shell immediately exits. I've run strace on both the 32bit and the 64bit sshd and here's the difference between the two, right after sshd gets an exception 5 from the DC: 32bit --- Process 5820, exception 00000005 at 75BF1D4D 43239 4372793 [waitproc] sshd 2400 pinfo::maybe_set_exit_code_from_windows: pid 5820, exit value - old 0x0, windows 0x5, cygwin 0x8000500 64bit --- Process 484, exception 00000005 at 00007FFA4F5D5BF8 9907 4522411 [main] sshd 484 seterrno_from_win_error: .../cygwin-snapshot-20140819-1/winsup/cygwin/sec_auth.cc:263 windows error 5 36 4522447 [main] sshd 484 geterrno_from_win_error: windows error 5 == errno 13 The 64bit sshd then proceeds to fetch my credentials from the registry and forks the command, while the 32bit winds down and does an exit(5). Ideas? I have done a passwd -R in both environments just in case these would be stored in different registry keys. Regards, Achim. -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Windows Server 2012R2 64bit and 32bit Cygwin sshd 2014-08-19 13:37 ` Achim Gratz @ 2014-08-19 17:02 ` Corinna Vinschen 2014-08-19 17:21 ` Achim Gratz 0 siblings, 1 reply; 24+ messages in thread From: Corinna Vinschen @ 2014-08-19 17:02 UTC (permalink / raw) To: cygwin [-- Attachment #1: Type: text/plain, Size: 1683 bytes --] On Aug 19 13:37, Achim Gratz wrote: > I've run strace on both the 32bit and the 64bit sshd and here's the > difference between the two, right after sshd gets an exception 5 from the DC: > > 32bit > --- Process 5820, exception 00000005 at 75BF1D4D > 43239 4372793 [waitproc] sshd 2400 pinfo::maybe_set_exit_code_from_windows: > pid 5820, exit value - old 0x0, windows 0x5, cygwin 0x8000500 > > 64bit > --- Process 484, exception 00000005 at 00007FFA4F5D5BF8 > 9907 4522411 [main] sshd 484 seterrno_from_win_error: > .../cygwin-snapshot-20140819-1/winsup/cygwin/sec_auth.cc:263 > windows error 5 This is a call to the Win32 API call NetUserGetGroups creating an exception due to an "access denied". The difference between 32 and 64 bit is probably a result of the differences in exception handling, and given that an ExceptionCode 5 will be ignored by Cygwin's exception handler, it's probably the next handler in the chain. Are you running sshd under a local cyg_server account or under a domain cyg_server account? Is it possible that this is a local cyg_server account, and as a non-domain account actually gets an "access denied" when trying to request AD user information? If your cyg_server is a local account I'd suggest to try with an AD account per https://cygwin.com/faq/faq.html#faq.using.sshd-in-domain Other than that I have a funny idea how to workaround this problem from inside Cygwin. If you want to give it a try, I'll send you a pointer to a 32 bit DLL via PM. Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Maintainer cygwin AT cygwin DOT com Red Hat [-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --] ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Windows Server 2012R2 64bit and 32bit Cygwin sshd 2014-08-19 17:02 ` Corinna Vinschen @ 2014-08-19 17:21 ` Achim Gratz 2014-08-19 19:09 ` Corinna Vinschen 0 siblings, 1 reply; 24+ messages in thread From: Achim Gratz @ 2014-08-19 17:21 UTC (permalink / raw) To: cygwin Corinna Vinschen writes: > This is a call to the Win32 API call NetUserGetGroups creating an > exception due to an "access denied". The difference between 32 and 64 > bit is probably a result of the differences in exception handling, and > given that an ExceptionCode 5 will be ignored by Cygwin's exception > handler, it's probably the next handler in the chain. Well, as I said, Cygwin64 dutifully ignores the exception and then goes to fetch my password from the registry and forks off the command given to ssh. > Are you running sshd under a local cyg_server account or under a domain > cyg_server account? Is it possible that this is a local cyg_server > account, and as a non-domain account actually gets an "access denied" > when trying to request AD user information? It's a local account, actually I've got a separate account cyg_server32 for that. I've double checked that this is set up the same way as the original cyg_server account except for the home directory. > If your cyg_server is a local account I'd suggest to try with an > AD account per https://cygwin.com/faq/faq.html#faq.using.sshd-in-domain I can try but the AD admins are still sitting on their thumbs for the sshd domain account, so I don't have high hopes for fast enough turnaround. Plus, they have new rules that forbid administrative accounts that have unlimited activation, so I don't know how much use this is. > Other than that I have a funny idea how to workaround this problem from > inside Cygwin. If you want to give it a try, I'll send you a pointer > to a 32 bit DLL via PM. Yes please. As long as I don't get sshd to work, I can't use this installation for its intended purpose anyway. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Factory and User Sound Singles for Waldorf Q+, Q and microQ: http://Synth.Stromeko.net/Downloads.html#WaldorfSounds -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Windows Server 2012R2 64bit and 32bit Cygwin sshd 2014-08-19 17:21 ` Achim Gratz @ 2014-08-19 19:09 ` Corinna Vinschen 2014-08-19 19:21 ` Achim Gratz 0 siblings, 1 reply; 24+ messages in thread From: Corinna Vinschen @ 2014-08-19 19:09 UTC (permalink / raw) To: cygwin [-- Attachment #1: Type: text/plain, Size: 2492 bytes --] On Aug 19 19:21, Achim Gratz wrote: > Corinna Vinschen writes: > > This is a call to the Win32 API call NetUserGetGroups creating an > > exception due to an "access denied". The difference between 32 and 64 > > bit is probably a result of the differences in exception handling, and > > given that an ExceptionCode 5 will be ignored by Cygwin's exception > > handler, it's probably the next handler in the chain. > > Well, as I said, Cygwin64 dutifully ignores the exception and then goes > to fetch my password from the registry and forks off the command given > to ssh. Yeah, as I said. > > Are you running sshd under a local cyg_server account or under a domain > > cyg_server account? Is it possible that this is a local cyg_server > > account, and as a non-domain account actually gets an "access denied" > > when trying to request AD user information? > > It's a local account, actually I've got a separate account cyg_server32 > for that. I've double checked that this is set up the same way as the > original cyg_server account except for the home directory. It doesn't matter if it's the same. An exception is generated and 32 and 64 bit versions react differently for whatever reason. It's also really unnecessary to generate two accounts for this. > > If your cyg_server is a local account I'd suggest to try with an > > AD account per https://cygwin.com/faq/faq.html#faq.using.sshd-in-domain > > I can try but the AD admins are still sitting on their thumbs for the > sshd domain account, so I don't have high hopes for fast enough > turnaround. Plus, they have new rules that forbid administrative > accounts that have unlimited activation, so I don't know how much use > this is. Setting up sshd should be the job of an admin in the first place. The domain account is a service starter account, not just some arbitrary administrative account. That, and much easier maintainance domain-wide should be blatantly obvious to the admins. > > Other than that I have a funny idea how to workaround this problem from > > inside Cygwin. If you want to give it a try, I'll send you a pointer > > to a 32 bit DLL via PM. > > Yes please. As long as I don't get sshd to work, I can't use this > installation for its intended purpose anyway. Thanks, coming... Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Maintainer cygwin AT cygwin DOT com Red Hat [-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --] ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Windows Server 2012R2 64bit and 32bit Cygwin sshd 2014-08-19 19:09 ` Corinna Vinschen @ 2014-08-19 19:21 ` Achim Gratz 0 siblings, 0 replies; 24+ messages in thread From: Achim Gratz @ 2014-08-19 19:21 UTC (permalink / raw) To: cygwin Corinna Vinschen writes: > It doesn't matter if it's the same. An exception is generated and 32 > and 64 bit versions react differently for whatever reason. It's also > really unnecessary to generate two accounts for this. I know that now. But at one point I was wondering if somehow the fact that certain settings in the cyg_server user account point to the 64bit installation was responsible for my problem as it looked like there was some interaction. > Setting up sshd should be the job of an admin in the first place. The > domain account is a service starter account, not just some arbitrary > administrative account. That, and much easier maintainance domain-wide > should be blatantly obvious to the admins. I am the admin, but just of that single server and not all the time. As I said I can and will ask, but even if the guy I'm talking to agrees it makes sense that still doesn't mean it's going to happen once it's been filtered through their set of rules that don't have provisions for Cygwin in them. They've already asked why I don't convert that box to Linux (different rules again which at least know of sshd) and I actually might end up having to do so in the long run. > Thanks, coming... Appreciated. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ SD adaptations for Waldorf Q V3.00R3 and Q+ V3.54R2: http://Synth.Stromeko.net/Downloads.html#WaldorfSDada -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Windows Server 2012R2 64bit and 32bit Cygwin sshd 2014-08-18 17:26 Windows Server 2012R2 64bit and 32bit Cygwin sshd Achim Gratz 2014-08-19 13:37 ` Achim Gratz @ 2014-09-02 11:52 ` Achim Gratz 2014-09-02 14:07 ` Corinna Vinschen 1 sibling, 1 reply; 24+ messages in thread From: Achim Gratz @ 2014-09-02 11:52 UTC (permalink / raw) To: cygwin Achim Gratz <Stromeko <at> nexgo.de> writes: > - Windows Server 2012R2 64bit running on hardware > - domain member > - Cygwin 32bit (latest snapshot from 2014-08-31 w/ AD integration) > > can anybody confirm they have gotten sshd successfully running? As far > as I can debug this, the sshd I have installed goes through all the > motions of checking and setting up the environment, but any process it > then spawns to actually run my command or shell immediately exits. The server I was testing this on had to be re-installed for reasons unrelated to Cygwin, so I've had a fresh start. The issue remains the same however: I have successfully set up an sshd for both a productive 64bit Cygwin and a separate 64bit build environment and can use them both in parallel. The 32bit build environment works just fine in an RDP session, but I can't make the 32bit sshd actually start programs. As before, it validates the user logging in successfully and then fails to start the program requested. The failure is the same for a local account and a domain account and regardless of whether I try to log in from the outside or via localhost. Regards, Achim. -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Windows Server 2012R2 64bit and 32bit Cygwin sshd 2014-09-02 11:52 ` Achim Gratz @ 2014-09-02 14:07 ` Corinna Vinschen 2014-09-02 15:16 ` Achim Gratz 2014-09-02 16:25 ` Achim Gratz 0 siblings, 2 replies; 24+ messages in thread From: Corinna Vinschen @ 2014-09-02 14:07 UTC (permalink / raw) To: cygwin [-- Attachment #1: Type: text/plain, Size: 1862 bytes --] On Sep 2 11:51, Achim Gratz wrote: > Achim Gratz <Stromeko <at> nexgo.de> writes: > > - Windows Server 2012R2 64bit running on hardware > > - domain member > > - Cygwin 32bit (latest snapshot from 2014-08-31 w/ AD integration) > > > > can anybody confirm they have gotten sshd successfully running? As far > > as I can debug this, the sshd I have installed goes through all the > > motions of checking and setting up the environment, but any process it > > then spawns to actually run my command or shell immediately exits. > > The server I was testing this on had to be re-installed for reasons > unrelated to Cygwin, so I've had a fresh start. The issue remains the same > however: I have successfully set up an sshd for both a productive 64bit > Cygwin and a separate 64bit build environment and can use them both in > parallel. The 32bit build environment works just fine in an RDP session, > but I can't make the 32bit sshd actually start programs. As before, it > validates the user logging in successfully and then fails to start the > program requested. The failure is the same for a local account and a domain > account and regardless of whether I try to log in from the outside or via > localhost. This is strange, really. I just set up a Windows 8.1 machine with two services, one called sshd64 in 64 bit Cygwin, running under my usual AD account called cyg_server, listening on port 22 and one called sshd in 32 bit Cygwin, running under the same cyg_server account, listening on port 20022. Both services work fine in parallel, and password as wll as pubkey auth work as expected. I have no idea what could be wrong in your environment, sorry. Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Maintainer cygwin AT cygwin DOT com Red Hat [-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --] ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Windows Server 2012R2 64bit and 32bit Cygwin sshd 2014-09-02 14:07 ` Corinna Vinschen @ 2014-09-02 15:16 ` Achim Gratz 2014-09-02 15:38 ` Corinna Vinschen 2014-09-02 16:25 ` Achim Gratz 1 sibling, 1 reply; 24+ messages in thread From: Achim Gratz @ 2014-09-02 15:16 UTC (permalink / raw) To: cygwin Corinna Vinschen <corinna-cygwin <at> cygwin.com> writes: > I have no idea what could be wrong in your environment, sorry. Me neither. I've set all three Cygwin environments up exactly the same way, using identical users and command lines (just different service names, obviously). I still have to use a local cyg_server account and a local sshd account, the latter of which gets mapped from MACHINE+sshd to plain sshd via /etc/passwd to get sshd to start. Did the strace I sent you give any clue (I can make one again if that helps) because it seems that everything just works out as it should until the fork that should start the command given to ssh. Regards, Achim. -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Windows Server 2012R2 64bit and 32bit Cygwin sshd 2014-09-02 15:16 ` Achim Gratz @ 2014-09-02 15:38 ` Corinna Vinschen 2014-09-02 17:32 ` Achim Gratz 2014-09-03 7:17 ` Achim Gratz 0 siblings, 2 replies; 24+ messages in thread From: Corinna Vinschen @ 2014-09-02 15:38 UTC (permalink / raw) To: cygwin [-- Attachment #1: Type: text/plain, Size: 1395 bytes --] On Sep 2 15:16, Achim Gratz wrote: > Corinna Vinschen <corinna-cygwin <at> cygwin.com> writes: > > I have no idea what could be wrong in your environment, sorry. > > Me neither. I've set all three Cygwin environments up exactly the same way, > using identical users and command lines (just different service names, > obviously). > > I still have to use a local cyg_server account and a local sshd account, the > latter of which gets mapped from MACHINE+sshd to plain sshd via /etc/passwd > to get sshd to start. Don't use privilege separation, then the non-privileged sshd user won't matter at all. Privsep on Cygwin is only half-useful on Cygwin anyway, if at all. As for the local cyg_server account, I'm not sure. Usually, a local machine account has no or only limited access to AD information. As an account which needs AD to get user information it's a bit unfortunate if it doesn't have access. > Did the strace I sent you give any clue (I can make > one again if that helps) because it seems that everything just works out as > it should until the fork that should start the command given to ssh. The strace shows that it doesn't even *try* to start bash, but it's entirely unclear why. Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Maintainer cygwin AT cygwin DOT com Red Hat [-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --] ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Windows Server 2012R2 64bit and 32bit Cygwin sshd 2014-09-02 15:38 ` Corinna Vinschen @ 2014-09-02 17:32 ` Achim Gratz 2014-09-03 7:17 ` Achim Gratz 1 sibling, 0 replies; 24+ messages in thread From: Achim Gratz @ 2014-09-02 17:32 UTC (permalink / raw) To: cygwin Corinna Vinschen writes: > Don't use privilege separation, then the non-privileged sshd user won't > matter at all. Privsep on Cygwin is only half-useful on Cygwin anyway, > if at all. As for the local cyg_server account, I'm not sure. Usually, > a local machine account has no or only limited access to AD information. > As an account which needs AD to get user information it's a bit > unfortunate if it doesn't have access. Unless it's possible to set these rights depending on whether a 32bit or a 64bit process is run by the user I don't see how this would come into play. As I said, it's the exact same user and the exact same setup for both 64bit and 32bit. >> Did the strace I sent you give any clue (I can make >> one again if that helps) because it seems that everything just works out as >> it should until the fork that should start the command given to ssh. > > The strace shows that it doesn't even *try* to start bash, but it's > entirely unclear why. See my other mail, not that it elucidates the reason, but I think I've pinpointed where it happens now. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Factory and User Sound Singles for Waldorf rackAttack: http://Synth.Stromeko.net/Downloads.html#WaldorfSounds -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Windows Server 2012R2 64bit and 32bit Cygwin sshd 2014-09-02 15:38 ` Corinna Vinschen 2014-09-02 17:32 ` Achim Gratz @ 2014-09-03 7:17 ` Achim Gratz 2014-09-03 13:03 ` Achim Gratz 2014-09-03 13:26 ` Corinna Vinschen 1 sibling, 2 replies; 24+ messages in thread From: Achim Gratz @ 2014-09-03 7:17 UTC (permalink / raw) To: cygwin Corinna Vinschen <corinna-cygwin <at> cygwin.com> writes: > Don't use privilege separation, then the non-privileged sshd user won't > matter at all. Privsep on Cygwin is only half-useful on Cygwin anyway, > if at all. I've switched privilege separateion off completely, but no dice. The Access Denied comes from trying to switch from primary group "MACHINE+None" to "Domain Users". That is expected to happen, what I still don't get is why the parent process winds up with the exception instead of the originating process as on 64bit. > As for the local cyg_server account, I'm not sure. Usually, > a local machine account has no or only limited access to AD information. > As an account which needs AD to get user information it's a bit > unfortunate if it doesn't have access. When the process comes to this point it has already verified the user via AD. > The strace shows that it doesn't even *try* to start bash, but it's > entirely unclear why. Is it possible to run sshd in gdb? Regards, Achim. -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Windows Server 2012R2 64bit and 32bit Cygwin sshd 2014-09-03 7:17 ` Achim Gratz @ 2014-09-03 13:03 ` Achim Gratz 2014-09-03 13:37 ` Corinna Vinschen 2014-09-03 13:26 ` Corinna Vinschen 1 sibling, 1 reply; 24+ messages in thread From: Achim Gratz @ 2014-09-03 13:03 UTC (permalink / raw) To: cygwin Achim Gratz <Stromeko <at> NexGo.DE> writes: > > The strace shows that it doesn't even *try* to start bash, but it's > > entirely unclear why. > > Is it possible to run sshd in gdb? I can attach the debugger but I didn't manage to break into something useful (or anything at all, really). However, the bug must be triggered by a race because with the debugger attached once in a blue moon I'll get the command actually executed correctly. I'm running this on an 8-core machine, so it is very likely that each new thread hits a different core... Regards, Achim. -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Windows Server 2012R2 64bit and 32bit Cygwin sshd 2014-09-03 13:03 ` Achim Gratz @ 2014-09-03 13:37 ` Corinna Vinschen 2014-09-04 11:24 ` Achim Gratz 0 siblings, 1 reply; 24+ messages in thread From: Corinna Vinschen @ 2014-09-03 13:37 UTC (permalink / raw) To: cygwin [-- Attachment #1: Type: text/plain, Size: 2068 bytes --] On Sep 3 13:02, Achim Gratz wrote: > Achim Gratz <Stromeko <at> NexGo.DE> writes: > > > The strace shows that it doesn't even *try* to start bash, but it's > > > entirely unclear why. > > > > Is it possible to run sshd in gdb? > > I can attach the debugger but I didn't manage to break into something useful > (or anything at all, really). However, the bug must be triggered by a race > because with the debugger attached once in a blue moon I'll get the command > actually executed correctly. I'm running this on an 8-core machine, so it > is very likely that each new thread hits a different core... You already built your own Cygwin DLL, right? What you could do is to do some good old printf debugging. First let's try to find out if it's really one of the NetUser calls: Index: sec_auth.cc =================================================================== RCS file: /cvs/src/src/winsup/cygwin/sec_auth.cc,v retrieving revision 1.57 diff -u -p -r1.57 sec_auth.cc --- sec_auth.cc 22 May 2014 16:40:13 -0000 1.57 +++ sec_auth.cc 3 Sep 2014 13:37:11 -0000 @@ -256,8 +256,10 @@ get_user_groups (WCHAR *logonserver, cyg NET_API_STATUS ret; /* Look only on logonserver */ + debug_printf ("Before NetUserGetGroups"); ret = NetUserGetGroups (logonserver, user, 0, (LPBYTE *) &buf, MAX_PREFERRED_LENGTH, &cnt, &tot); + debug_printf ("After NetUserGetGroups"); if (ret) { __seterrno_from_win_error (ret); @@ -306,9 +308,11 @@ get_user_local_groups (PWCHAR logonserve DWORD cnt, tot; NET_API_STATUS ret; + debug_printf ("Before NetUserGetLocalGroups"); ret = NetUserGetLocalGroups (logonserver, user, 0, LG_INCLUDE_INDIRECT, (LPBYTE *) &buf, MAX_PREFERRED_LENGTH, &cnt, &tot); + debug_printf ("After NetUserGetLocalGroups"); if (ret) { __seterrno_from_win_error (ret); Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Maintainer cygwin AT cygwin DOT com Red Hat [-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --] ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Windows Server 2012R2 64bit and 32bit Cygwin sshd 2014-09-03 13:37 ` Corinna Vinschen @ 2014-09-04 11:24 ` Achim Gratz 2014-09-04 12:28 ` Corinna Vinschen 0 siblings, 1 reply; 24+ messages in thread From: Achim Gratz @ 2014-09-04 11:24 UTC (permalink / raw) To: cygwin Corinna Vinschen <corinna-cygwin <at> cygwin.com> writes: > You already built your own Cygwin DLL, right? What you could do is to > do some good old printf debugging. First let's try to find out if it's > really one of the NetUser calls: It looks like I need to install more than the DLL to make this work? I couldn't start cygserver as a service with (just) the built DLL in place. So I started it in debug mode from the command line (which makes it have less rights than it needs) and started the sshd in debug mode also. Due to presumably the missing rights mentioned I could only log in with an administrative account (domain account, but restricted to run on the server only). I didn't get any failure from the debug_printf instrumented functions. With my normal user account I got a "/bin/bash: Operation not permitted". The cygserver debug output also showed unfettered access to the AD. With the sshd running without privilege separation I've noticed some requests to the cygserver that seemed to indicate memory corruption: Early on in starting the daemon it would normally try to get account information for Administrators:544, but the debug output from cygserver was showing sshdrs as the account name being asked for. Also there are (probably unrelated since they are also present on x86_64) complaints about requests of illegal length (11). Going back to the original snapshot and using the same debugging setup the behaviour was still the same. Since I could now start the services again, I did that and am back to the original behaviour. I've asked our IT if there are restrictions specifically targetting 32bit services or processes, but got no answer so far (I'm not even sure this is possible). I'm not sure what to make of these results, but at the moment I've ran out of time anyway. Regards, Achim. -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Windows Server 2012R2 64bit and 32bit Cygwin sshd 2014-09-04 11:24 ` Achim Gratz @ 2014-09-04 12:28 ` Corinna Vinschen 2014-09-04 14:12 ` Achim Gratz 0 siblings, 1 reply; 24+ messages in thread From: Corinna Vinschen @ 2014-09-04 12:28 UTC (permalink / raw) To: cygwin [-- Attachment #1: Type: text/plain, Size: 2091 bytes --] On Sep 4 11:23, Achim Gratz wrote: > Corinna Vinschen <corinna-cygwin <at> cygwin.com> writes: > > You already built your own Cygwin DLL, right? What you could do is to > > do some good old printf debugging. First let's try to find out if it's > > really one of the NetUser calls: > > It looks like I need to install more than the DLL to make this work? No. > I couldn't start cygserver as a service with (just) the built DLL in place. No idea why. The patch just adds debug output to strace ouptput, nothing else. > So I started it in debug mode from the command line (which makes it have > less rights than it needs) and started the sshd in debug mode also. In a cyg_server GUI session? If so, you should have all rights required when starting this in an elevated shell. > Due to > presumably the missing rights mentioned I could only log in with an > administrative account (domain account, but restricted to run on the server > only). I didn't get any failure from the debug_printf instrumented > functions. With my normal user account I got a "/bin/bash: Operation not > permitted". The cygserver debug output also showed unfettered access to the > AD. With the sshd running without privilege separation I've noticed some > requests to the cygserver that seemed to indicate memory corruption: Early > on in starting the daemon it would normally try to get account information > for Administrators:544, but the debug output from cygserver was showing > sshdrs as the account name being asked for. Also there are (probably > unrelated since they are also present on x86_64) complaints about requests > of illegal length (11). Ok, I don't grok this. If you have trouble with cygserver, which is completely unrelated to this issue, please discuss this in a new thread and please describe detailed what you did to provoke the problem and paste what you see. Thanks, Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Maintainer cygwin AT cygwin DOT com Red Hat [-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --] ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Windows Server 2012R2 64bit and 32bit Cygwin sshd 2014-09-04 12:28 ` Corinna Vinschen @ 2014-09-04 14:12 ` Achim Gratz 2014-09-04 14:59 ` Achim Gratz 2014-09-05 11:16 ` Corinna Vinschen 0 siblings, 2 replies; 24+ messages in thread From: Achim Gratz @ 2014-09-04 14:12 UTC (permalink / raw) To: cygwin Corinna Vinschen <corinna-cygwin <at> cygwin.com> writes: > > I couldn't start cygserver as a service with (just) the built DLL in place. > > No idea why. The patch just adds debug output to strace ouptput, nothing > else. Whatever. I've installed all the binaries from that build and things work normally now. > > So I started it in debug mode from the command line (which makes it have > > less rights than it needs) and started the sshd in debug mode also. > > In a cyg_server GUI session? If so, you should have all rights required > when starting this in an elevated shell. Not the token privileges, I don't think so. But I'm not sure how to check. Here's the salient parts from the strace (attaching to the sshd running as a service in sandbox mode, running with no privilege separation produces a slightly different trace, but the events leading up to the error are the same): 262 1161585 [main] sshd 2044 getpid: 2044 = getpid() 10593 1172178 [main] sshd 2044 get_logon_server: DC: server: \\SC301 58 1172236 [main] sshd 2044 get_user_groups: Before NetUserGetGroups --- Process 560, exception 00000005 at 75511D4D 6543 1178779 [main] sshd 2044 get_user_groups: After NetUserGetGroups ret = 5 56 1178835 [main] sshd 2044 seterrno_from_win_error: ../../../../source/cygwin-snapshot-20140903-1/winsup/cygwin/sec_auth.cc:265 windows error 5 36 1178871 [main] sshd 2044 geterrno_from_win_error: windows error 5 == errno 13 33 1178904 [main] sshd 2044 get_user_local_groups: Before NetUserGetLocalGroups --- Process 560, exception 00000005 at 75511D4D 7964 1186868 [main] sshd 2044 get_user_local_groups: After NetUserGetLocalGroups ret = 5 50 1186918 [main] sshd 2044 seterrno_from_win_error: ../../../../source/cygwin-snapshot-20140903-1/winsup/cygwin/sec_auth.cc:318 windows error 5 38 1186956 [main] sshd 2044 geterrno_from_win_error: windows error 5 == errno 13 37 1186993 [main] sshd 2044 initgroups32: 0 = initgroups(gratz, 1049089) It then proceeds to log on via the token and mounts the entries from my personal fstab (that should fail if it was running as a different user for some of the entries). After checking for /etc/nologin this happens: 35 5023308 [main] sshd 2248 setegid32: new egid: 1049089 current: 197121 41 5023349 [main] sshd 2248 setegid32: NtSetInformationToken (hProcToken, TokenPrimaryGroup), 0xC000005B 3105 5026454 [main] sshd 2248 get_logon_server: DC: server: \\SC301 44 5026498 [main] sshd 2248 get_user_groups: Before NetUserGetGroups --- Process 2248, exception 00000005 at 75511D4D The process apparently gets killed while in the NetUserGetGroups call (much as you suspected). I'm not sure this tells us anything new, though. :-( Regards, Achim. -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Windows Server 2012R2 64bit and 32bit Cygwin sshd 2014-09-04 14:12 ` Achim Gratz @ 2014-09-04 14:59 ` Achim Gratz 2014-09-05 11:16 ` Corinna Vinschen 1 sibling, 0 replies; 24+ messages in thread From: Achim Gratz @ 2014-09-04 14:59 UTC (permalink / raw) To: cygwin Achim Gratz <Stromeko <at> NexGo.DE> writes: > Not the token privileges, I don't think so. But I'm not sure how to check. If I run the sshd as the user trying to log in, then it works; regardless of whether the user has administrative privileges or not or with or without privilege separation enabled for the sshd. Regards, Achim. -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Windows Server 2012R2 64bit and 32bit Cygwin sshd 2014-09-04 14:12 ` Achim Gratz 2014-09-04 14:59 ` Achim Gratz @ 2014-09-05 11:16 ` Corinna Vinschen 2014-09-05 11:56 ` Corinna Vinschen 1 sibling, 1 reply; 24+ messages in thread From: Corinna Vinschen @ 2014-09-05 11:16 UTC (permalink / raw) To: cygwin [-- Attachment #1: Type: text/plain, Size: 3115 bytes --] On Sep 4 14:12, Achim Gratz wrote: > Corinna Vinschen <corinna-cygwin <at> cygwin.com> writes: > > > I couldn't start cygserver as a service with (just) the built DLL in place. > > > > No idea why. The patch just adds debug output to strace ouptput, nothing > > else. > > Whatever. I've installed all the binaries from that build and things work > normally now. > > > > So I started it in debug mode from the command line (which makes it have > > > less rights than it needs) and started the sshd in debug mode also. > > > > In a cyg_server GUI session? If so, you should have all rights required > > when starting this in an elevated shell. > > Not the token privileges, I don't think so. But I'm not sure how to check. Windows whoami /all > Here's the salient parts from the strace (attaching to the sshd running as a > service in sandbox mode, running with no privilege separation produces a > slightly different trace, but the events leading up to the error are the same): > [...] > 44 5026498 [main] sshd 2248 get_user_groups: Before NetUserGetGroups > --- Process 2248, exception 00000005 at 75511D4D > > The process apparently gets killed while in the NetUserGetGroups call (much > as you suspected). I'm not sure this tells us anything new, though. :-( No, it just confirms it. Please remove the debug_printf patch and try this instead: Index: sec_auth.cc =================================================================== RCS file: /cvs/src/src/winsup/cygwin/sec_auth.cc,v retrieving revision 1.57 diff -u -p -r1.57 sec_auth.cc --- sec_auth.cc 22 May 2014 16:40:13 -0000 1.57 +++ sec_auth.cc 5 Sep 2014 11:12:01 -0000 @@ -20,6 +20,7 @@ details. */ #include "fhandler.h" #include "dtable.h" #include "cygheap.h" +#include "exception.h" #include "ntdll.h" #include "tls_pbuf.h" #include <lm.h> @@ -255,9 +256,13 @@ get_user_groups (WCHAR *logonserver, cyg DWORD cnt, tot, len; NET_API_STATUS ret; - /* Look only on logonserver */ - ret = NetUserGetGroups (logonserver, user, 0, (LPBYTE *) &buf, - MAX_PREFERRED_LENGTH, &cnt, &tot); + { + /* Experimental SEH */ + exception protect; + /* Look only on logonserver */ + ret = NetUserGetGroups (logonserver, user, 0, (LPBYTE *) &buf, + MAX_PREFERRED_LENGTH, &cnt, &tot); + } if (ret) { __seterrno_from_win_error (ret); @@ -306,9 +311,14 @@ get_user_local_groups (PWCHAR logonserve DWORD cnt, tot; NET_API_STATUS ret; - ret = NetUserGetLocalGroups (logonserver, user, 0, LG_INCLUDE_INDIRECT, - (LPBYTE *) &buf, MAX_PREFERRED_LENGTH, - &cnt, &tot); + { + /* Experimental SEH */ + exception protect; + + ret = NetUserGetLocalGroups (logonserver, user, 0, LG_INCLUDE_INDIRECT, + (LPBYTE *) &buf, MAX_PREFERRED_LENGTH, + &cnt, &tot); + } if (ret) { __seterrno_from_win_error (ret); Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Maintainer cygwin AT cygwin DOT com Red Hat [-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --] ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Windows Server 2012R2 64bit and 32bit Cygwin sshd 2014-09-05 11:16 ` Corinna Vinschen @ 2014-09-05 11:56 ` Corinna Vinschen 2014-09-05 18:17 ` Achim Gratz 0 siblings, 1 reply; 24+ messages in thread From: Corinna Vinschen @ 2014-09-05 11:56 UTC (permalink / raw) To: cygwin [-- Attachment #1: Type: text/plain, Size: 1855 bytes --] On Sep 5 13:16, Corinna Vinschen wrote: > On Sep 4 14:12, Achim Gratz wrote: > > Corinna Vinschen <corinna-cygwin <at> cygwin.com> writes: > > > > I couldn't start cygserver as a service with (just) the built DLL in place. > > > > > > No idea why. The patch just adds debug output to strace ouptput, nothing > > > else. > > > > Whatever. I've installed all the binaries from that build and things work > > normally now. > > > > > > So I started it in debug mode from the command line (which makes it have > > > > less rights than it needs) and started the sshd in debug mode also. > > > > > > In a cyg_server GUI session? If so, you should have all rights required > > > when starting this in an elevated shell. > > > > Not the token privileges, I don't think so. But I'm not sure how to check. > > Windows whoami /all > > > Here's the salient parts from the strace (attaching to the sshd running as a > > service in sandbox mode, running with no privilege separation produces a > > slightly different trace, but the events leading up to the error are the same): > > [...] > > 44 5026498 [main] sshd 2248 get_user_groups: Before NetUserGetGroups > > --- Process 2248, exception 00000005 at 75511D4D > > > > The process apparently gets killed while in the NetUserGetGroups call (much > > as you suspected). I'm not sure this tells us anything new, though. :-( > > No, it just confirms it. > > Please remove the debug_printf patch and try this instead: > > Index: sec_auth.cc > [...] For simplicity I added this experimental patch to the snapshot I just uploaded to https://cygwin.com/snapshots/. Please give it a try. Thanks, Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Maintainer cygwin AT cygwin DOT com Red Hat [-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --] ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Windows Server 2012R2 64bit and 32bit Cygwin sshd 2014-09-05 11:56 ` Corinna Vinschen @ 2014-09-05 18:17 ` Achim Gratz 0 siblings, 0 replies; 24+ messages in thread From: Achim Gratz @ 2014-09-05 18:17 UTC (permalink / raw) To: cygwin Corinna Vinschen writes: > For simplicity I added this experimental patch to the snapshot I just > uploaded to https://cygwin.com/snapshots/. I'm ooO so no updates from me on this until I get back. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Factory and User Sound Singles for Waldorf Blofeld: http://Synth.Stromeko.net/Downloads.html#WaldorfSounds -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Windows Server 2012R2 64bit and 32bit Cygwin sshd 2014-09-03 7:17 ` Achim Gratz 2014-09-03 13:03 ` Achim Gratz @ 2014-09-03 13:26 ` Corinna Vinschen 1 sibling, 0 replies; 24+ messages in thread From: Corinna Vinschen @ 2014-09-03 13:26 UTC (permalink / raw) To: cygwin [-- Attachment #1: Type: text/plain, Size: 2380 bytes --] On Sep 3 07:17, Achim Gratz wrote: > Corinna Vinschen <corinna-cygwin <at> cygwin.com> writes: > > Don't use privilege separation, then the non-privileged sshd user won't > > matter at all. Privsep on Cygwin is only half-useful on Cygwin anyway, > > if at all. > > I've switched privilege separateion off completely, but no dice. The Access > Denied comes from trying to switch from primary group "MACHINE+None" to > "Domain Users". That is expected to happen, what I still don't get is why > the parent process winds up with the exception instead of the originating > process as on 64bit. As I wrote, this is a red herring. A failing setgid is no error at all. It has nothing to do with the exception, except for the debug output preceeding the exception occurance. Note the get_logon_server: DC: server: \\SC301 between them. This, and the subsequent seterrno_from_win_error in the parent sshd are a pretty sure sign that the exception is triggered by the NetUserXXX calls. > > As for the local cyg_server account, I'm not sure. Usually, > > a local machine account has no or only limited access to AD information. > > As an account which needs AD to get user information it's a bit > > unfortunate if it doesn't have access. > > When the process comes to this point it has already verified the user via AD. Yes, but this has nothing to do with it. Before calling setuid (which it calls a couple of times during login), sshd calls initgroups for the new user, POSIX-like. Initgroups in turn has to call NetUserGetGroups and NetUserGetLocalGroups on the DC to fetch the full list of groups for a user. From the strace it *seems* that the call to NetUserGetGroups in the grand child sshd process results in simply terminating the process. The fact that there's no more output *at all* from the grand child points to Windows killing the process hard. Lacking any hint why this occurs, it's just an assumption, of course. > > The strace shows that it doesn't even *try* to start bash, but it's > > entirely unclear why. > > Is it possible to run sshd in gdb? Yes, but Windows/Cygwin gdb don't allow to follow the child process so it's very tricky. Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Maintainer cygwin AT cygwin DOT com Red Hat [-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --] ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Windows Server 2012R2 64bit and 32bit Cygwin sshd 2014-09-02 14:07 ` Corinna Vinschen 2014-09-02 15:16 ` Achim Gratz @ 2014-09-02 16:25 ` Achim Gratz 2014-09-02 19:14 ` Corinna Vinschen 1 sibling, 1 reply; 24+ messages in thread From: Achim Gratz @ 2014-09-02 16:25 UTC (permalink / raw) To: cygwin Corinna Vinschen <corinna-cygwin <at> cygwin.com> writes: > I have no idea what could be wrong in your environment, sorry. Some more information: up until the check for /etc/nologin everything works. Things fall apart when sshd tries to switch to the user logging on. I've traced this again, sshd apparently then tries to use the user token to set the primary group and gets the predictable error "0xC000005B -- STATUS_INVALID_PRIMARY_GROUP". The 64bit sshd catches the resulting exception in that same pid, sets errno and then goes on to switch the uid. In 32bit the exception winds up in the parent process instead and leads to the termination of the connection. The seteuid32 call is never initiated. Regards, Achim. -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Windows Server 2012R2 64bit and 32bit Cygwin sshd 2014-09-02 16:25 ` Achim Gratz @ 2014-09-02 19:14 ` Corinna Vinschen 0 siblings, 0 replies; 24+ messages in thread From: Corinna Vinschen @ 2014-09-02 19:14 UTC (permalink / raw) To: cygwin [-- Attachment #1: Type: text/plain, Size: 1470 bytes --] On Sep 2 16:24, Achim Gratz wrote: > Corinna Vinschen <corinna-cygwin <at> cygwin.com> writes: > > I have no idea what could be wrong in your environment, sorry. > > Some more information: up until the check for /etc/nologin everything works. > Things fall apart when sshd tries to switch to the user logging on. I've > traced this again, sshd apparently then tries to use the user token to set > the primary group and gets the predictable error "0xC000005B -- > STATUS_INVALID_PRIMARY_GROUP". This is a red herring. The 0xC000005B status code returned by NtSetInformationToken is not at all fatal, nor is it an exception. The only exceptions in this code occur right after figuring out the DC, when calling functions to request user or group information from the DC, NetUserGetGroups and NetUserGetLocalGroups. But these are first chance exceptions for the debugger. They don't occur if no debugger is present. However, for some reason, when this exception occurs in the sshd grand child which is (probably) supposed to execve the shell, this exception appears to be fatal suddenly. There's no hint at all why, given that it's a simple "access denied". This is really unexpected. We can try to workaround that, but it's puzzeling that this should be required at all. Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Maintainer cygwin AT cygwin DOT com Red Hat [-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --] ^ permalink raw reply [flat|nested] 24+ messages in thread
end of thread, other threads:[~2014-09-05 18:17 UTC | newest] Thread overview: 24+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2014-08-18 17:26 Windows Server 2012R2 64bit and 32bit Cygwin sshd Achim Gratz 2014-08-19 13:37 ` Achim Gratz 2014-08-19 17:02 ` Corinna Vinschen 2014-08-19 17:21 ` Achim Gratz 2014-08-19 19:09 ` Corinna Vinschen 2014-08-19 19:21 ` Achim Gratz 2014-09-02 11:52 ` Achim Gratz 2014-09-02 14:07 ` Corinna Vinschen 2014-09-02 15:16 ` Achim Gratz 2014-09-02 15:38 ` Corinna Vinschen 2014-09-02 17:32 ` Achim Gratz 2014-09-03 7:17 ` Achim Gratz 2014-09-03 13:03 ` Achim Gratz 2014-09-03 13:37 ` Corinna Vinschen 2014-09-04 11:24 ` Achim Gratz 2014-09-04 12:28 ` Corinna Vinschen 2014-09-04 14:12 ` Achim Gratz 2014-09-04 14:59 ` Achim Gratz 2014-09-05 11:16 ` Corinna Vinschen 2014-09-05 11:56 ` Corinna Vinschen 2014-09-05 18:17 ` Achim Gratz 2014-09-03 13:26 ` Corinna Vinschen 2014-09-02 16:25 ` Achim Gratz 2014-09-02 19:14 ` Corinna Vinschen
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).