Named pipes and multiple writers

Named pipes and multiple writers

[sten.kristian.ivarsson]

I'll apologize in advance if something is not appropriate, but this is the
first interaction with this project

We (another open source project) have bumped into some problems using named
pipes with multiple writers

As far as I can see, reading through history, this have been a known issue
for quite some time, but it seems like there have been some attempts to
solve it, e.g. in the branch topic/fifo (by Ken Brown)

I tried to build that branch and also tried to merge related commits to
master and build that, but all without success and I guess the changes need
some background info to understand how to manually incorporate the changes
into later versions of master

Does anyone have any knowledge about if this (topic/fifo branch) is working
and/or if it is somehow planned to make it into the master branch and end up
in a future release ?

Does anyone know of any other attempts to solve this issue ?

Kristian

Re: Named pipes and multiple writers

[Ken Brown]

On 3/25/2020 7:11 AM, Kristian Ivarsson via Cygwin wrote:
> I'll apologize in advance if something is not appropriate, but this is the
> first interaction with this project
> 
> We (another open source project) have bumped into some problems using named
> pipes with multiple writers
> 
> As far as I can see, reading through history, this have been a known issue
> for quite some time, but it seems like there have been some attempts to
> solve it, e.g. in the branch topic/fifo (by Ken Brown)
> 
> I tried to build that branch and also tried to merge related commits to
> master and build that, but all without success and I guess the changes need
> some background info to understand how to manually incorporate the changes
> into later versions of master
> 
> Does anyone have any knowledge about if this (topic/fifo branch) is working
> and/or if it is somehow planned to make it into the master branch and end up
> in a future release ?

That branch is obsolete.  Support for multiple writers was added to Cygwin as of 
release 3.1.0.

Ken

Re: Sv: Named pipes and multiple writers

[Ken Brown]

[Let's keep the discussion on the list in case others have suggestions.]

On 3/25/2020 9:41 AM, sten.kristian.ivarsson@gmail.com wrote:
> [snip]
>>> As far as I can see, reading through history, this have been a known
>>> issue for quite some time, but it seems like there have been some
>>> attempts to solve it, e.g. in the branch topic/fifo (by Ken Brown)
> 
> [snip]
>>> Does anyone have any knowledge about if this (topic/fifo branch) is
>>> working and/or if it is somehow planned to make it into the master
>>> branch and end up in a future release ?
> 
>> That branch is obsolete.  Support for multiple writers was added to Cygwin
> as of release 3.1.0.
> 
> Ok, thanks, but we're running 3.1.4 (and tested 3.1.5) but do still have
> problems (experiencing ENXIO (No such device or address)) but actually (as
> far as we see) with the 3:rd writer ?
> 
> We need to investigate the issue more thoroughly and might get back when we
> have more knowledge

Does the ENXIO come from fhandler_fifo::wait?  If so, it's quite possible that 
there's a bug involving read_ready in my code.

Ken

Re: Sv: Named pipes and multiple writers

[Ken Brown]

On 3/26/2020 10:06 AM, Ken Brown via Cygwin wrote:
> [Let's keep the discussion on the list in case others have suggestions.]
> 
> On 3/25/2020 9:41 AM, sten.kristian.ivarsson@gmail.com wrote:
>> [snip]
>>>> As far as I can see, reading through history, this have been a known
>>>> issue for quite some time, but it seems like there have been some
>>>> attempts to solve it, e.g. in the branch topic/fifo (by Ken Brown)
>>
>> [snip]
>>>> Does anyone have any knowledge about if this (topic/fifo branch) is
>>>> working and/or if it is somehow planned to make it into the master
>>>> branch and end up in a future release ?
>>
>>> That branch is obsolete.  Support for multiple writers was added to Cygwin
>> as of release 3.1.0.
>>
>> Ok, thanks, but we're running 3.1.4 (and tested 3.1.5) but do still have
>> problems (experiencing ENXIO (No such device or address)) but actually (as
>> far as we see) with the 3:rd writer ?
>>
>> We need to investigate the issue more thoroughly and might get back when we
>> have more knowledge
> 
> Does the ENXIO come from fhandler_fifo::wait?  If so, it's quite possible that 
> there's a bug involving read_ready in my code.

BTW, I've been working on adding support for multiple readers.  I expect to have 
a first cut ready within a week or two.  Would you have any use for that?  If 
so, I could revive the topic/fifo branch and push my patches there for you to test.

Ken

Re: Sv: Named pipes and multiple writers

[Norton Allen]

On 3/26/2020 11:11 AM, Ken Brown via Cygwin wrote:
>
> BTW, I've been working on adding support for multiple readers.  I 
> expect to have a first cut ready within a week or two.  Would you have 
> any use for that?  If so, I could revive the topic/fifo branch and 
> push my patches there for you to test.
>

Ken, what are the semantics for multiple readers? Do all readers see the 
same data,  or is it first come first served or something else?

-- 
=============================================================
Norton Allen (he/him/his)
Software Engineer
Harvard University School of Engineering and Applied Sciences
12 Oxford St., Link Bldg. (Office 282)
Cambridge, MA  02138
Phone: (617) 998-5553
=============================================================

Re: Sv: Named pipes and multiple writers

[Ken Brown]

On 3/26/2020 12:03 PM, Norton Allen wrote:
> On 3/26/2020 11:11 AM, Ken Brown via Cygwin wrote:
>>
>> BTW, I've been working on adding support for multiple readers.  I expect to 
>> have a first cut ready within a week or two.  Would you have any use for 
>> that?  If so, I could revive the topic/fifo branch and push my patches there 
>> for you to test.
>>
> 
> Ken, what are the semantics for multiple readers? Do all readers see the same 
> data,  or is it first come first served or something else?

It's first come, first served.  If two readers attempt to read simultaneously, 
it's possible that one will get some of the available input and the other will 
get some more.

The only use case for multiple readers that I've come across of is Midnight 
Commander running under tcsh.  I didn't dig into the code enough to know why 
they do it, or why only under tcsh.  See

   https://sourceware.org/pipermail/cygwin/2019-December/243317.html

and

   https://cygwin.com/pipermail/cygwin-apps/2019-December/039777.html

That's what got me interested in this.  It would be nice to know if there are 
other use cases.

Ken

Re: Sv: Named pipes and multiple writers

[Norton Allen]

On 3/26/2020 12:44 PM, Ken Brown via Cygwin wrote:
> On 3/26/2020 12:03 PM, Norton Allen wrote:
>> On 3/26/2020 11:11 AM, Ken Brown via Cygwin wrote:
>>>
>>> BTW, I've been working on adding support for multiple readers.  I 
>>> expect to have a first cut ready within a week or two.  Would you 
>>> have any use for that?  If so, I could revive the topic/fifo branch 
>>> and push my patches there for you to test.
>>>
>>
>> Ken, what are the semantics for multiple readers? Do all readers see 
>> the same data,  or is it first come first served or something else?
>
> It's first come, first served.  If two readers attempt to read 
> simultaneously, it's possible that one will get some of the available 
> input and the other will get some more.
>
> The only use case for multiple readers that I've come across of is 
> Midnight Commander running under tcsh.  I didn't dig into the code 
> enough to know why they do it, or why only under tcsh.  See
>
> https://sourceware.org/pipermail/cygwin/2019-December/243317.html
>
> and
>
> https://cygwin.com/pipermail/cygwin-apps/2019-December/039777.html
>
> That's what got me interested in this.  It would be nice to know if 
> there are other use cases.
>
I suppose it could be used as a simple approach to deploying jobs to 
worker processes, provided a process could guarantee that it received 
enough information to define a job and not more than one. I guess if the 
job definition were fixed length that could work.

Sv: Sv: Named pipes and multiple writers

[sten.kristian.ivarsson]

>> [snip]
>>>> As far as I can see, reading through history, this have been a known 
>>>> issue for quite some time, but it seems like there have been some 
>>>> attempts to solve it, e.g. in the branch topic/fifo (by Ken Brown)
>> 
>> [snip]
>>>> Does anyone have any knowledge about if this (topic/fifo branch) is 
>>>> working and/or if it is somehow planned to make it into the master 
>>>> branch and end up in a future release ?
>> 
>>> That branch is obsolete.  Support for multiple writers was added to 
>>> Cygwin
>> as of release 3.1.0.
>> 
>> Ok, thanks, but we're running 3.1.4 (and tested 3.1.5) but do still 
>> have problems (experiencing ENXIO (No such device or address)) but 
>> actually (as far as we see) with the 3:rd writer ?
>> 
>> We need to investigate the issue more thoroughly and might get back 
>> when we have more knowledge

>Does the ENXIO come from fhandler_fifo::wait?  If so, it's quite possible
that there's a bug involving read_ready in my code.

Our application is a bit complex and I have now narrowed it down

The ENIXIO occurs when parallel child-processes simultaneously using
O_NONBLOCK opening the descriptor. We're sometimes opening it blocking and
sometimes non-blocking and it seems like when the 2:nd non-blocking process
tries to open it gets ENIXIO. The child process open and closes the
fifo-descriptor for writing multiple times. I could provide a code-snippet
to reproduce it if wanted ?

Tnx for showing interest btw

Kristian

>Ken

Re: Sv: Sv: Named pipes and multiple writers

[Ken Brown]

On 3/26/2020 6:01 PM, sten.kristian.ivarsson@gmail.com wrote:
> The ENIXIO occurs when parallel child-processes simultaneously using
> O_NONBLOCK opening the descriptor.

This is consistent with my guess that the error is generated by 
fhandler_fifo::wait.  I have a feeling that read_ready should have been created 
as a manual-reset event, and that more care is needed to make sure it's set when 
it should be.

> I could provide a code-snippet
> to reproduce it if wanted ?

Yes, please!

Ken

Re: Sv: Sv: Named pipes and multiple writers

[Ken Brown]

[-- Attachment #1: Type: text/plain, Size: 683 bytes --]

On 3/26/2020 6:39 PM, Ken Brown via Cygwin wrote:
> On 3/26/2020 6:01 PM, sten.kristian.ivarsson@gmail.com wrote:
>> The ENIXIO occurs when parallel child-processes simultaneously using
>> O_NONBLOCK opening the descriptor.
> 
> This is consistent with my guess that the error is generated by 
> fhandler_fifo::wait.  I have a feeling that read_ready should have been created 
> as a manual-reset event, and that more care is needed to make sure it's set when 
> it should be.
> 
>> I could provide a code-snippet
>> to reproduce it if wanted ?
> 
> Yes, please!

That might not be necessary.  If you're able to build the git repo master 
branch, please try the attached patch.

Ken

[-- Attachment #2: 0001-Cygwin-FIFO-make-read_ready-a-manual-reset-event.patch --]
[-- Type: text/plain, Size: 1344 bytes --]

From 279591d91a13616957964256e02344a627b6f558 Mon Sep 17 00:00:00 2001
From: Ken Brown <kbrown@cornell.edu>
Date: Thu, 26 Mar 2020 19:02:16 -0400
Subject: [PATCH] Cygwin: FIFO: make read_ready a manual-reset event

If a FIFO is open for reading and an attempt is made to open it for
writing with O_NONBLOCK, read_ready must be set in order for open to
succeed.  When read_ready was an auto-reset event, there was a brief
period when read_ready was not set set after a writer opened.  If a
second writer attempted to open the FIFO with O_NONBLOCK during this
period, the attempt would fail.

Addresses: https://sourceware.org/pipermail/cygwin/2020-March/244201.html
---
 winsup/cygwin/fhandler_fifo.cc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/winsup/cygwin/fhandler_fifo.cc b/winsup/cygwin/fhandler_fifo.cc
index 19cd0e507..c05161099 100644
--- a/winsup/cygwin/fhandler_fifo.cc
+++ b/winsup/cygwin/fhandler_fifo.cc
@@ -516,7 +516,7 @@ fhandler_fifo::open (int flags, mode_t)
 
   char npbuf[MAX_PATH];
   __small_sprintf (npbuf, "r-event.%08x.%016X", get_dev (), get_ino ());
-  if (!(read_ready = CreateEvent (sa_buf, false, false, npbuf)))
+  if (!(read_ready = CreateEvent (sa_buf, true, false, npbuf)))
     {
       debug_printf ("CreateEvent for %s failed, %E", npbuf);
       res = error_set_errno;
-- 
2.21.0

Re: Sv: Sv: Named pipes and multiple writers

[Ken Brown]

[-- Attachment #1: Type: text/plain, Size: 775 bytes --]

On 3/26/2020 7:19 PM, Ken Brown via Cygwin wrote:
> On 3/26/2020 6:39 PM, Ken Brown via Cygwin wrote:
>> On 3/26/2020 6:01 PM, sten.kristian.ivarsson@gmail.com wrote:
>>> The ENIXIO occurs when parallel child-processes simultaneously using
>>> O_NONBLOCK opening the descriptor.
>>
>> This is consistent with my guess that the error is generated by 
>> fhandler_fifo::wait.  I have a feeling that read_ready should have been 
>> created as a manual-reset event, and that more care is needed to make sure 
>> it's set when it should be.
>>
>>> I could provide a code-snippet
>>> to reproduce it if wanted ?
>>
>> Yes, please!
> 
> That might not be necessary.  If you're able to build the git repo master 
> branch, please try the attached patch.

Here's a better patch.

Ken

[-- Attachment #2: 0001-Cygwin-FIFO-fix-a-problem-opening-nonblocking-writer.patch --]
[-- Type: text/plain, Size: 1828 bytes --]

From 3efd5a8cbff8d48b8cf9807070134bb79f591b7d Mon Sep 17 00:00:00 2001
From: Ken Brown <kbrown@cornell.edu>
Date: Thu, 26 Mar 2020 19:02:16 -0400
Subject: [PATCH] Cygwin: FIFO: fix a problem opening nonblocking writers

Make read_ready a manual-reset event.  Previously, when it was an
auto-reset event, there was a brief period when read_ready was not set
after a writer opened.  An attempt to open a second writer during this
period would fail with ENXIO if O_NONBLOCK was set, even if a reader
was open.

For the same reason, move ResetEvent(read_ready) from
listen_client_thread() to close().

Addresses: https://sourceware.org/pipermail/cygwin/2020-March/244201.html
---
 winsup/cygwin/fhandler_fifo.cc | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/winsup/cygwin/fhandler_fifo.cc b/winsup/cygwin/fhandler_fifo.cc
index 19cd0e507..c7e27e883 100644
--- a/winsup/cygwin/fhandler_fifo.cc
+++ b/winsup/cygwin/fhandler_fifo.cc
@@ -463,7 +463,6 @@ fhandler_fifo::listen_client_thread ()
 out:
   if (evt)
     CloseHandle (evt);
-  ResetEvent (read_ready);
   if (ret < 0)
     debug_printf ("exiting with error, %E");
   else
@@ -516,7 +515,7 @@ fhandler_fifo::open (int flags, mode_t)
 
   char npbuf[MAX_PATH];
   __small_sprintf (npbuf, "r-event.%08x.%016X", get_dev (), get_ino ());
-  if (!(read_ready = CreateEvent (sa_buf, false, false, npbuf)))
+  if (!(read_ready = CreateEvent (sa_buf, true, false, npbuf)))
     {
       debug_printf ("CreateEvent for %s failed, %E", npbuf);
       res = error_set_errno;
@@ -1016,6 +1015,8 @@ fhandler_fifo::close ()
      handler or another thread. */
   fifo_client_unlock ();
   int ret = stop_listen_client ();
+  if (reader && read_ready)
+    ResetEvent (read_ready);
   if (read_ready)
     CloseHandle (read_ready);
   if (write_ready)
-- 
2.21.0

Sv: Sv: Sv: Named pipes and multiple writers

[sten.kristian.ivarsson]

[-- Attachment #1: Type: text/plain, Size: 1042 bytes --]

>On 3/26/2020 7:19 PM, Ken Brown via Cygwin wrote:
>> On 3/26/2020 6:39 PM, Ken Brown via Cygwin wrote:
>>> On 3/26/2020 6:01 PM, sten.kristian.ivarsson@gmail.com wrote:
>>>> The ENIXIO occurs when parallel child-processes simultaneously using 
>>>> O_NONBLOCK opening the descriptor.
>>>
>>> This is consistent with my guess that the error is generated by 
>>> fhandler_fifo::wait.  I have a feeling that read_ready should have 
>>> been created as a manual-reset event, and that more care is needed to 
>>> make sure it's set when it should be.
>>>
>>>> I could provide a code-snippet
>>>> to reproduce it if wanted ?
>>>
>>> Yes, please!
>> 
>> That might not be necessary.  If you're able to build the git repo 
>> master branch, please try the attached patch.

>Here's a better patch.


I finally succeeded to build latest master (make is not my favourite tool)
and added the patch, but still no success in my little test-program (see
attachment) when creating a write-file-descriptor with O_NONBLOCK

 
>Ken

[-- Attachment #2: pipe.cpp --]
[-- Type: text/plain, Size: 1671 bytes --]

#include <cstring>

#include <fcntl.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <unistd.h>

#include <iostream>
#include <string>

namespace
{
   auto error(const int error)
   {
      std::cerr << getpid() << "\terror:\t" << error << '\t' << std::strerror(error) << std::endl;
      return error;
   }
}

int main()
{
   const auto name{"/tmp/my_name"};

   const auto result = mkfifo(name, 0666);

   if(result) return error(errno);

   constexpr auto writers{5};
   constexpr auto messages{4};

   for(auto idx = 0; idx < writers; ++idx)
   {
      const auto pid = fork();

      if(pid < 0) return error(errno);

      if(pid == 0)
      {
         std::cout << "child " << getpid() << std::endl;
         for(auto idx = 0; idx < messages; ++idx)
         {
            const auto wfd = open(name, O_WRONLY | O_NONBLOCK);
            if(wfd < 0) return error(errno);
            const auto msg{std::to_string(getpid())};
            if(write(wfd, msg.data(), msg.size() + 1) < 0) error(errno);
            close(wfd);
         }
         return 0;
      }
   }

   {
      std::cout << "parent" << std::endl;

      const auto rfd = open(name, O_RDONLY);
      const auto wfd = open(name, O_WRONLY);
      if(rfd < 0) return error(errno);
      for(auto idx = 0; idx < writers * messages; ++idx)
      {
         std::string buffer;
         buffer.resize(80);
         if(read(rfd, &buffer[0], buffer.size()) < 0) error(errno);
         std::cout << buffer << std::endl;
      }
      close(wfd);
      close(rfd);
   }

   if(unlink(name) < 0) return error(errno);

   return 0;
}

Re: Sv: Sv: Sv: Named pipes and multiple writers

[Ken Brown]

On 3/27/2020 10:53 AM, sten.kristian.ivarsson@gmail.com wrote:
>> On 3/26/2020 7:19 PM, Ken Brown via Cygwin wrote:
>>> On 3/26/2020 6:39 PM, Ken Brown via Cygwin wrote:
>>>> On 3/26/2020 6:01 PM, sten.kristian.ivarsson@gmail.com wrote:
>>>>> The ENIXIO occurs when parallel child-processes simultaneously using
>>>>> O_NONBLOCK opening the descriptor.
>>>>
>>>> This is consistent with my guess that the error is generated by
>>>> fhandler_fifo::wait.  I have a feeling that read_ready should have
>>>> been created as a manual-reset event, and that more care is needed to
>>>> make sure it's set when it should be.
>>>>
>>>>> I could provide a code-snippet
>>>>> to reproduce it if wanted ?
>>>>
>>>> Yes, please!
>>>
>>> That might not be necessary.  If you're able to build the git repo
>>> master branch, please try the attached patch.
> 
>> Here's a better patch.
> 
> 
> I finally succeeded to build latest master (make is not my favourite tool)
> and added the patch, but still no success in my little test-program (see
> attachment) when creating a write-file-descriptor with O_NONBLOCK

Your test program fails for me on Linux too.  Here's the output from one run:

child 657
657     error:  6       No such device or address
child 658
child 659
658659  error:  child 660
parent
child 661
         error:  66606661 661 661
                 error:          661
No such device or address6No such device or address

No such device or address

[I then killed it with control-C; the parent was blocked trying to open the FIFO.]

There's a race condition in your code.  The parent is trying to open the FIFO 
for reading (without O_NONBLOCK) while the child is trying to open it for 
writing (with O_NONBLOCK).  The parent is blocked waiting for the child, and the 
child's open fails with ENXIO; see

   https://pubs.opengroup.org/onlinepubs/9699919799/functions/open.html#tag_16_357

I think you need to rearrange things so that the FIFO is open for reading before 
you try a nonblocking open for writing.

I can work around the race by using a small positive 'wait' in 
fhandler_fifo::wait(), but I'm not sure this is the right thing to do, since 
Cygwin aims to emulate Linux.  Can you find a test case that works on Linux but 
fails on Cygwin?

Ken

Re: Sv: Sv: Sv: Named pipes and multiple writers

[Ken Brown]

On 3/27/2020 6:56 PM, Ken Brown via Cygwin wrote:
> On 3/27/2020 10:53 AM, sten.kristian.ivarsson@gmail.com wrote:
>>> On 3/26/2020 7:19 PM, Ken Brown via Cygwin wrote:
>>>> On 3/26/2020 6:39 PM, Ken Brown via Cygwin wrote:
>>>>> On 3/26/2020 6:01 PM, sten.kristian.ivarsson@gmail.com wrote:
>>>>>> The ENIXIO occurs when parallel child-processes simultaneously using
>>>>>> O_NONBLOCK opening the descriptor.
>>>>>
>>>>> This is consistent with my guess that the error is generated by
>>>>> fhandler_fifo::wait.  I have a feeling that read_ready should have
>>>>> been created as a manual-reset event, and that more care is needed to
>>>>> make sure it's set when it should be.
>>>>>
>>>>>> I could provide a code-snippet
>>>>>> to reproduce it if wanted ?
>>>>>
>>>>> Yes, please!
>>>>
>>>> That might not be necessary.  If you're able to build the git repo
>>>> master branch, please try the attached patch.
>>
>>> Here's a better patch.
>>
>>
>> I finally succeeded to build latest master (make is not my favourite tool)
>> and added the patch, but still no success in my little test-program (see
>> attachment) when creating a write-file-descriptor with O_NONBLOCK
> 
> Your test program fails for me on Linux too.  Here's the output from one run:
> 
> child 657
> 657     error:  6       No such device or address
> child 658
> child 659
> 658659  error:  child 660
> parent
> child 661
>          error:  66606661 661 661
>                  error:          661
> No such device or address6No such device or address
> 
> No such device or address
> 
> [I then killed it with control-C; the parent was blocked trying to open the FIFO.]
> 
> There's a race condition in your code.  The parent is trying to open the FIFO 
> for reading (without O_NONBLOCK) while the child is trying to open it for 
> writing (with O_NONBLOCK).  The parent is blocked waiting for the child, and the 
> child's open fails with ENXIO; see
> 
>    https://pubs.opengroup.org/onlinepubs/9699919799/functions/open.html#tag_16_357
> 
> I think you need to rearrange things so that the FIFO is open for reading before 
> you try a nonblocking open for writing.

For example, you could open it with O_RDWR instead of O_RDONLY.

Sv: Sv: Sv: Sv: Named pipes and multiple writers

[sten.kristian.ivarsson]

>On 3/27/2020 10:53 AM, sten.kristian.ivarsson@gmail.com wrote:
>>> On 3/26/2020 7:19 PM, Ken Brown via Cygwin wrote:
>>>> On 3/26/2020 6:39 PM, Ken Brown via Cygwin wrote:
>>>>> On 3/26/2020 6:01 PM, sten.kristian.ivarsson@gmail.com wrote:
>>>>>> The ENIXIO occurs when parallel child-processes simultaneously 
>>>>>> using O_NONBLOCK opening the descriptor.
>>>>>
>>>>> This is consistent with my guess that the error is generated by 
>>>>> fhandler_fifo::wait.  I have a feeling that read_ready should have 
>>>>> been created as a manual-reset event, and that more care is needed 
>>>>> to make sure it's set when it should be.
>>>>>
>>>>>> I could provide a code-snippet
>>>>>> to reproduce it if wanted ?
>>>>>
>>>>> Yes, please!
>>>>
>>>> That might not be necessary.  If you're able to build the git repo 
>>>> master branch, please try the attached patch.
>> 
>>> Here's a better patch.
>> 
>> 
>> I finally succeeded to build latest master (make is not my favourite 
>> tool) and added the patch, but still no success in my little 
>> test-program (see
>> attachment) when creating a write-file-descriptor with O_NONBLOCK

>Your test program fails for me on Linux too.  Here's the output from one
run:

You're right. That was extremely careless of me to not test this in Linux
first :-)

I can assure that we have a use case that works on Linux but not in Cygwin,
but it seems like I failed to narrow it down in the wrong way

I'll try to rearrange my code (that works in Linux) to mimic our application
but in a simple way (I'll be back)

[snip]

>Ken

Re: Sv: Sv: Sv: Sv: Named pipes and multiple writers

[Ken Brown]

On 3/28/2020 8:10 AM, sten.kristian.ivarsson@gmail.com wrote:
>> On 3/27/2020 10:53 AM, sten.kristian.ivarsson@gmail.com wrote:
>>>> On 3/26/2020 7:19 PM, Ken Brown via Cygwin wrote:
>>>>> On 3/26/2020 6:39 PM, Ken Brown via Cygwin wrote:
>>>>>> On 3/26/2020 6:01 PM, sten.kristian.ivarsson@gmail.com wrote:
>>>>>>> The ENIXIO occurs when parallel child-processes simultaneously
>>>>>>> using O_NONBLOCK opening the descriptor.
>>>>>>
>>>>>> This is consistent with my guess that the error is generated by
>>>>>> fhandler_fifo::wait.  I have a feeling that read_ready should have
>>>>>> been created as a manual-reset event, and that more care is needed
>>>>>> to make sure it's set when it should be.
>>>>>>
>>>>>>> I could provide a code-snippet
>>>>>>> to reproduce it if wanted ?
>>>>>>
>>>>>> Yes, please!
>>>>>
>>>>> That might not be necessary.  If you're able to build the git repo
>>>>> master branch, please try the attached patch.
>>>
>>>> Here's a better patch.
>>>
>>>
>>> I finally succeeded to build latest master (make is not my favourite
>>> tool) and added the patch, but still no success in my little
>>> test-program (see
>>> attachment) when creating a write-file-descriptor with O_NONBLOCK
> 
>> Your test program fails for me on Linux too.  Here's the output from one
> run:
> 
> You're right. That was extremely careless of me to not test this in Linux
> first :-)

No problem.

> I can assure that we have a use case that works on Linux but not in Cygwin,
> but it seems like I failed to narrow it down in the wrong way
> 
> I'll try to rearrange my code (that works in Linux) to mimic our application
> but in a simple way (I'll be back)

OK, I'll be waiting for you.  BTW, if it's not too hard to write your test case 
in plain C, or at least less modern C++, that would simplify things for me.  For 
example, your pipe.cpp failed to compile on one Linux machine I wanted to test 
it on, presumably because that machine had an older C++ compiler.

Ken

Re: Sv: Sv: Sv: Sv: Named pipes and multiple writers

[Ken Brown]

On 3/28/2020 11:43 AM, Ken Brown via Cygwin wrote:
> On 3/28/2020 8:10 AM, sten.kristian.ivarsson@gmail.com wrote:
>>> On 3/27/2020 10:53 AM, sten.kristian.ivarsson@gmail.com wrote:
>>>>> On 3/26/2020 7:19 PM, Ken Brown via Cygwin wrote:
>>>>>> On 3/26/2020 6:39 PM, Ken Brown via Cygwin wrote:
>>>>>>> On 3/26/2020 6:01 PM, sten.kristian.ivarsson@gmail.com wrote:
>>>>>>>> The ENIXIO occurs when parallel child-processes simultaneously
>>>>>>>> using O_NONBLOCK opening the descriptor.
>>>>>>>
>>>>>>> This is consistent with my guess that the error is generated by
>>>>>>> fhandler_fifo::wait.  I have a feeling that read_ready should have
>>>>>>> been created as a manual-reset event, and that more care is needed
>>>>>>> to make sure it's set when it should be.
>>>>>>>
>>>>>>>> I could provide a code-snippet
>>>>>>>> to reproduce it if wanted ?
>>>>>>>
>>>>>>> Yes, please!
>>>>>>
>>>>>> That might not be necessary.  If you're able to build the git repo
>>>>>> master branch, please try the attached patch.
>>>>
>>>>> Here's a better patch.
>>>>
>>>>
>>>> I finally succeeded to build latest master (make is not my favourite
>>>> tool) and added the patch, but still no success in my little
>>>> test-program (see
>>>> attachment) when creating a write-file-descriptor with O_NONBLOCK
>>
>>> Your test program fails for me on Linux too.  Here's the output from one
>> run:
>>
>> You're right. That was extremely careless of me to not test this in Linux
>> first :-)
> 
> No problem.
> 
>> I can assure that we have a use case that works on Linux but not in Cygwin,
>> but it seems like I failed to narrow it down in the wrong way
>>
>> I'll try to rearrange my code (that works in Linux) to mimic our application
>> but in a simple way (I'll be back)
> 
> OK, I'll be waiting for you.  BTW, if it's not too hard to write your test case 
> in plain C, or at least less modern C++, that would simplify things for me.  For 
> example, your pipe.cpp failed to compile on one Linux machine I wanted to test 
> it on, presumably because that machine had an older C++ compiler.

Never mind.  I was able to reproduce the problem and find the cause.  What 
happens is that when the first subprocess exits, fhandler_fifo::close resets 
read_ready.  That causes the second and subsequent subprocesses to think that 
there's no reader open, so their attempts to open a writer with O_NONBLOCK fail 
with ENXIO.

I should be able to fix this tomorrow.

Ken

Re: Sv: Sv: Sv: Sv: Named pipes and multiple writers

[Ken Brown]

[-- Attachment #1: Type: text/plain, Size: 3675 bytes --]

On 3/28/2020 10:19 PM, Ken Brown via Cygwin wrote:
> On 3/28/2020 11:43 AM, Ken Brown via Cygwin wrote:
>> On 3/28/2020 8:10 AM, sten.kristian.ivarsson@gmail.com wrote:
>>>> On 3/27/2020 10:53 AM, sten.kristian.ivarsson@gmail.com wrote:
>>>>>> On 3/26/2020 7:19 PM, Ken Brown via Cygwin wrote:
>>>>>>> On 3/26/2020 6:39 PM, Ken Brown via Cygwin wrote:
>>>>>>>> On 3/26/2020 6:01 PM, sten.kristian.ivarsson@gmail.com wrote:
>>>>>>>>> The ENIXIO occurs when parallel child-processes simultaneously
>>>>>>>>> using O_NONBLOCK opening the descriptor.
>>>>>>>>
>>>>>>>> This is consistent with my guess that the error is generated by
>>>>>>>> fhandler_fifo::wait.  I have a feeling that read_ready should have
>>>>>>>> been created as a manual-reset event, and that more care is needed
>>>>>>>> to make sure it's set when it should be.
>>>>>>>>
>>>>>>>>> I could provide a code-snippet
>>>>>>>>> to reproduce it if wanted ?
>>>>>>>>
>>>>>>>> Yes, please!
>>>>>>>
>>>>>>> That might not be necessary.  If you're able to build the git repo
>>>>>>> master branch, please try the attached patch.
>>>>>
>>>>>> Here's a better patch.
>>>>>
>>>>>
>>>>> I finally succeeded to build latest master (make is not my favourite
>>>>> tool) and added the patch, but still no success in my little
>>>>> test-program (see
>>>>> attachment) when creating a write-file-descriptor with O_NONBLOCK
>>>
>>>> Your test program fails for me on Linux too.  Here's the output from one
>>> run:
>>>
>>> You're right. That was extremely careless of me to not test this in Linux
>>> first :-)
>>
>> No problem.
>>
>>> I can assure that we have a use case that works on Linux but not in Cygwin,
>>> but it seems like I failed to narrow it down in the wrong way
>>>
>>> I'll try to rearrange my code (that works in Linux) to mimic our application
>>> but in a simple way (I'll be back)
>>
>> OK, I'll be waiting for you.  BTW, if it's not too hard to write your test 
>> case in plain C, or at least less modern C++, that would simplify things for 
>> me.  For example, your pipe.cpp failed to compile on one Linux machine I 
>> wanted to test it on, presumably because that machine had an older C++ compiler.
> 
> Never mind.  I was able to reproduce the problem and find the cause.  What 
> happens is that when the first subprocess exits, fhandler_fifo::close resets 
> read_ready.  That causes the second and subsequent subprocesses to think that 
> there's no reader open, so their attempts to open a writer with O_NONBLOCK fail 
> with ENXIO.
> 
> I should be able to fix this tomorrow.

I've pushed what I think is a fix to the topic/fifo branch.  I tested it with 
the attached program, which is a variant of the test case you sent last week. 
Please test it in your use case.

Note: If you've previously pulled the topic/fifo branch, then you will probably 
get a lot of conflicts when you pull again, because I did a forced push a few 
days ago.  If that happens, just do

   git reset --hard origin/topic/fifo

It turned out that the fix required some of the ideas that I've been working on 
in connection with allowing multiple readers.  Even though the code allows a 
FIFO to be *explicitly* opened for reading only once, there can still be several 
open file descriptors for readers because of dup and fork.  The existing code on 
git master doesn't handle those situations properly.

The code on topic/fifo doesn't completely fix that yet, but I think it should 
work under the following assumptions:

1. The FIFO is opened only once for reading.

2. The file descriptor obtained from this is the only one on which a read is 
attempted.

I'm working on removing both of these restrictions.

Ken

[-- Attachment #2: fifo_fork_nonblocking_writers.c --]
[-- Type: text/plain, Size: 1639 bytes --]

/* Adapted from
   https://sourceware.org/pipermail/cygwin/2020-March/244219.html */

#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <errno.h>
#include <sys/wait.h>
#include <string.h>

#define FIFO "/tmp/myfifo"
#define nsubproc  5
#define nmessages 4
#define pid_len   4

int
error (const int n, const char *name)
{
  fprintf (stderr, "\n%d\t%s:\t%d\t%s\n", getpid (), name, n, strerror (n));
  return n;
}

int
main ()
{
  if (mkfifo (FIFO, S_IRUSR | S_IWUSR | S_IWGRP) < 0
      && errno != EEXIST)
    return error (errno, "mkfifo");

  int rfd = open (FIFO, O_RDWR);
  if (rfd < 0)
    return error (errno, "open reader");

  for (int i = 0; i < nsubproc; i++)
    {
      pid_t pid = fork ();

      if (pid < 0)
	return error (errno, "fork");

      if (pid == 0)
	{
	  printf ("child %d\n", getpid ());
	  for (int j = 0; j < nmessages; j++)
	    {
	      char buf[pid_len + 2]; /* +1 for newline, +1 for nul */

	      int wfd = open (FIFO, O_WRONLY | O_NONBLOCK);
	      if (wfd < 0)
		_exit (error (errno, "open writer"));
	      sprintf (buf, "%d\n", getpid ());
	      ssize_t nwritten = write (wfd, buf, strlen (buf));
	      if (nwritten < 0)
		error (errno, "write");
	      /* printf ("i = %d, j = %d, nwritten = %zd\n", i, j, nwritten); */
	      close (wfd);
	    }
	  _exit (0);
	}
    }

  printf ("parent\n");
  char buf[pid_len + 2];
  for (int i = 0; i < nsubproc; i++)
    for (int j = 0; j < nmessages; j++)
      {
	if (read (rfd, buf, nsubproc) < 0)
	  error (errno, "read");
	buf[pid_len + 1] = '\0';
	printf ("%s", buf);
      }
  close (rfd);
}

Sv: Sv: Sv: Sv: Sv: Named pipes and multiple writers

[sten.kristian.ivarsson]

[-- Attachment #1: Type: text/plain, Size: 4305 bytes --]

>On 3/28/2020 10:19 PM, Ken Brown via Cygwin wrote:
>> On 3/28/2020 11:43 AM, Ken Brown via Cygwin wrote:
>>> On 3/28/2020 8:10 AM, sten.kristian.ivarsson@gmail.com wrote:
>>>>> On 3/27/2020 10:53 AM, sten.kristian.ivarsson@gmail.com wrote:
>>>>>>> On 3/26/2020 7:19 PM, Ken Brown via Cygwin wrote:
>>>>>>>> On 3/26/2020 6:39 PM, Ken Brown via Cygwin wrote:
>>>>>>>>> On 3/26/2020 6:01 PM, sten.kristian.ivarsson@gmail.com wrote:
>>>>>>>>>> The ENIXIO occurs when parallel child-processes simultaneously 
>>>>>>>>>> using O_NONBLOCK opening the descriptor.
>>>>>>>>>
>>>>>>>>> This is consistent with my guess that the error is generated by 
>>>>>>>>> fhandler_fifo::wait.  I have a feeling that read_ready should 
>>>>>>>>> have been created as a manual-reset event, and that more care 
>>>>>>>>> is needed to make sure it's set when it should be.
>>>>>>>>>
>>>>>>>>>> I could provide a code-snippet to reproduce it if wanted ?
>>>>>>>>>
>>>>>>>>> Yes, please!
>>>>>>>>
>>>>>>>> That might not be necessary.  If you're able to build the git 
>>>>>>>> repo master branch, please try the attached patch.
>>>>>>
>>>>>>> Here's a better patch.
>>>>>>
>>>>>>
>>>>>> I finally succeeded to build latest master (make is not my 
>>>>>> favourite
>>>>>> tool) and added the patch, but still no success in my little 
>>>>>> test-program (see
>>>>>> attachment) when creating a write-file-descriptor with O_NONBLOCK
>>>>
>>>>> Your test program fails for me on Linux too.  Here's the output 
>>>>> from one
>>>> run:
>>>>
>>>> You're right. That was extremely careless of me to not test this in 
>>>> Linux first :-)
>>>
>>> No problem.
>>>
>>>> I can assure that we have a use case that works on Linux but not in 
>>>> Cygwin, but it seems like I failed to narrow it down in the wrong 
>>>> way
>>>>
>>>> I'll try to rearrange my code (that works in Linux) to mimic our 
>>>> application but in a simple way (I'll be back)
>>>
>>> OK, I'll be waiting for you.  BTW, if it's not too hard to write your 
>>> test case in plain C, or at least less modern C++, that would 
>>> simplify things for me.  For example, your pipe.cpp failed to compile 
>>> on one Linux machine I wanted to test it on, presumably because that
machine had an older C++ compiler.
>> 
>> Never mind.  I was able to reproduce the problem and find the cause.  
>> What happens is that when the first subprocess exits, 
>> fhandler_fifo::close resets read_ready.  That causes the second and 
>> subsequent subprocesses to think that there's no reader open, so their 
>> attempts to open a writer with O_NONBLOCK fail with ENXIO.
>> 
>> I should be able to fix this tomorrow.

>I've pushed what I think is a fix to the topic/fifo branch.  I tested it
with the attached program, which is a variant of the test case you sent last
week. 
>Please test it in your use case.

>Note: If you've previously pulled the topic/fifo branch, then you will
probably get a lot of conflicts when you pull again, because I did a forced
push a few days ago.  If that happens, just do

>   git reset --hard origin/topic/fifo

>It turned out that the fix required some of the ideas that I've been
working on in connection with allowing multiple readers.  Even though the
code allows a FIFO to be *explicitly* opened for reading only once, there
can still be several open file descriptors for readers because of dup and
fork.  The existing code on git master doesn't handle those situations
properly.

>The code on topic/fifo doesn't completely fix that yet, but I think it
should work under the following assumptions:

>1. The FIFO is opened only once for reading.

>2. The file descriptor obtained from this is the only one on which a read
is attempted.

>I'm working on removing both of these restrictions.

>Ken 

We finally took the time to make some kind of a simplified "hack" that works
on Ubuntu and BSD/OSX but with latest on master newlib-cygwin gave "ENXIO"
now and then but with your previous patch attached, there was no ENXIO but
::read returns EAGIN (until exhausted) (with cygwin) almost every run

I will try your newest things tomorrow

See latest attatched test-program (starts to get bloated but this time more
C-compatible though:-)

Kristian


[-- Attachment #2: pipe.cpp --]
[-- Type: text/plain, Size: 2462 bytes --]

#include <string.h>
#include <fcntl.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <unistd.h>
#include <stdio.h>
#include <errno.h>


int print_error(const int error)
{
   printf("%s\n", strerror(error));
   return error;
}
struct Message
{
   int pid;
   int index;
};

int main()
{
   const char* name = "/tmp/my_pipe";
   const int result = mkfifo(name, 0666);
   if (result)
      return print_error(errno);
   const int writers{5};
   const int messages{5};
   printf("open parent pipe\n");
   const int rfd = open(name, O_RDONLY | O_NONBLOCK);
   const int wfd = open(name, O_WRONLY);
   if (rfd < 0)
      return print_error(errno);
   const int block_alternation[] = {0, O_NONBLOCK};
   int pids[writers];
   for (auto idx = 0; idx < writers; ++idx)
   {
      const auto pid = fork();
      if (pid < 0)
         return print_error(errno);
      if (pid == 0)
      {
         printf("child %d\n", getpid());
         for (int index = 0; index < messages; ++index)
         {
            const int wfd = open(name, O_WRONLY | block_alternation[index % 2]);
            if (wfd < 0)
               return print_error(errno);
            Message message{getpid(), index};
            short attempts{10000};
            while (write(wfd, &message, sizeof(message)) == -1 && errno == EAGAIN && attempts > 0)
               --attempts;
            if(attempts == 0)
               printf("child %d write - pipe busy\n", getpid());
            close(wfd);
         }
         return 0;
      }
      pids[idx] = pid;
   }
   {
      printf("parent read\n");
      for (int idx = 0; idx < writers * messages; ++idx)
      {
         short attempts{10000};
         Message message{};
         while (::read(rfd, &message, sizeof(message)) == -1 && errno == EAGAIN && attempts > 0)
            --attempts;

         if (attempts == 0)
            printf("parent %d read - pipe empty\n", getpid());
         else
            printf("message %d %d\n", message.index, message.pid);
      }
      // wait for children.
      for (int idx = 0; idx < writers; ++idx)
      {
         int status = 0;
         waitpid(pids[idx], &status, 0);
         if (WIFEXITED(status))
            printf("child exited %d\n", pids[idx]);
      }
      close(wfd);
      close(rfd);
   }
   if (unlink(name) < 0)
      return print_error(errno);
   return 0;
}

Re: Sv: Sv: Sv: Sv: Sv: Named pipes and multiple writers

[Ken Brown]

On 3/31/2020 5:10 PM, sten.kristian.ivarsson@gmail.com wrote:
>> On 3/28/2020 10:19 PM, Ken Brown via Cygwin wrote:
>>> On 3/28/2020 11:43 AM, Ken Brown via Cygwin wrote:
>>>> On 3/28/2020 8:10 AM, sten.kristian.ivarsson@gmail.com wrote:
>>>>>> On 3/27/2020 10:53 AM, sten.kristian.ivarsson@gmail.com wrote:
>>>>>>>> On 3/26/2020 7:19 PM, Ken Brown via Cygwin wrote:
>>>>>>>>> On 3/26/2020 6:39 PM, Ken Brown via Cygwin wrote:
>>>>>>>>>> On 3/26/2020 6:01 PM, sten.kristian.ivarsson@gmail.com wrote:
>>>>>>>>>>> The ENIXIO occurs when parallel child-processes simultaneously
>>>>>>>>>>> using O_NONBLOCK opening the descriptor.
>>>>>>>>>>
>>>>>>>>>> This is consistent with my guess that the error is generated by
>>>>>>>>>> fhandler_fifo::wait.  I have a feeling that read_ready should
>>>>>>>>>> have been created as a manual-reset event, and that more care
>>>>>>>>>> is needed to make sure it's set when it should be.
>>>>>>>>>>
>>>>>>>>>>> I could provide a code-snippet to reproduce it if wanted ?
>>>>>>>>>>
>>>>>>>>>> Yes, please!
>>>>>>>>>
>>>>>>>>> That might not be necessary.  If you're able to build the git
>>>>>>>>> repo master branch, please try the attached patch.
>>>>>>>
>>>>>>>> Here's a better patch.
>>>>>>>
>>>>>>>
>>>>>>> I finally succeeded to build latest master (make is not my
>>>>>>> favourite
>>>>>>> tool) and added the patch, but still no success in my little
>>>>>>> test-program (see
>>>>>>> attachment) when creating a write-file-descriptor with O_NONBLOCK
>>>>>
>>>>>> Your test program fails for me on Linux too.  Here's the output
>>>>>> from one
>>>>> run:
>>>>>
>>>>> You're right. That was extremely careless of me to not test this in
>>>>> Linux first :-)
>>>>
>>>> No problem.
>>>>
>>>>> I can assure that we have a use case that works on Linux but not in
>>>>> Cygwin, but it seems like I failed to narrow it down in the wrong
>>>>> way
>>>>>
>>>>> I'll try to rearrange my code (that works in Linux) to mimic our
>>>>> application but in a simple way (I'll be back)
>>>>
>>>> OK, I'll be waiting for you.  BTW, if it's not too hard to write your
>>>> test case in plain C, or at least less modern C++, that would
>>>> simplify things for me.  For example, your pipe.cpp failed to compile
>>>> on one Linux machine I wanted to test it on, presumably because that
> machine had an older C++ compiler.
>>>
>>> Never mind.  I was able to reproduce the problem and find the cause.
>>> What happens is that when the first subprocess exits,
>>> fhandler_fifo::close resets read_ready.  That causes the second and
>>> subsequent subprocesses to think that there's no reader open, so their
>>> attempts to open a writer with O_NONBLOCK fail with ENXIO.
>>>
>>> I should be able to fix this tomorrow.
> 
>> I've pushed what I think is a fix to the topic/fifo branch.  I tested it
> with the attached program, which is a variant of the test case you sent last
> week.
>> Please test it in your use case.
> 
>> Note: If you've previously pulled the topic/fifo branch, then you will
> probably get a lot of conflicts when you pull again, because I did a forced
> push a few days ago.  If that happens, just do
> 
>>    git reset --hard origin/topic/fifo
> 
>> It turned out that the fix required some of the ideas that I've been
> working on in connection with allowing multiple readers.  Even though the
> code allows a FIFO to be *explicitly* opened for reading only once, there
> can still be several open file descriptors for readers because of dup and
> fork.  The existing code on git master doesn't handle those situations
> properly.
> 
>> The code on topic/fifo doesn't completely fix that yet, but I think it
> should work under the following assumptions:
> 
>> 1. The FIFO is opened only once for reading.
> 
>> 2. The file descriptor obtained from this is the only one on which a read
> is attempted.
> 
>> I'm working on removing both of these restrictions.
> 
>> Ken
> 
> We finally took the time to make some kind of a simplified "hack" that works
> on Ubuntu and BSD/OSX but with latest on master newlib-cygwin gave "ENXIO"
> now and then but with your previous patch attached, there was no ENXIO but
> ::read returns EAGIN (until exhausted) (with cygwin) almost every run
> 
> I will try your newest things tomorrow
> 
> See latest attatched test-program (starts to get bloated but this time more
> C-compatible though:-)

Thanks.  This runs fine with the current HEAD of topic/fifo.

Ken

Sv: Sv: Sv: Sv: Sv: Sv: Named pipes and multiple writers

[sten.kristian.ivarsson]

> On 3/31/2020 5:10 PM, sten.kristian.ivarsson@gmail.com wrote:
> >> On 3/28/2020 10:19 PM, Ken Brown via Cygwin wrote:
> >>> On 3/28/2020 11:43 AM, Ken Brown via Cygwin wrote:
> >>>> On 3/28/2020 8:10 AM, sten.kristian.ivarsson@gmail.com wrote:
> >>>>>> On 3/27/2020 10:53 AM, sten.kristian.ivarsson@gmail.com wrote:
> >>>>>>>> On 3/26/2020 7:19 PM, Ken Brown via Cygwin wrote:
> >>>>>>>>> On 3/26/2020 6:39 PM, Ken Brown via Cygwin wrote:
> >>>>>>>>>> On 3/26/2020 6:01 PM, sten.kristian.ivarsson@gmail.com wrote:
> >>>>>>>>>>> The ENIXIO occurs when parallel child-processes
> >>>>>>>>>>> simultaneously using O_NONBLOCK opening the descriptor.

[snip] 

> Thanks.  This runs fine with the current HEAD of topic/fifo.

We are very grateful for your efforts 

The test-program works fine now with the latest commits in topic/fifo

There are some (possible other) issues that make our application doesn't
work as in Linux and we haven't had the time to dig into what the problems
are yet. A quick guess is that it is related to signals and possible related
to pselect, but we need to dig deeper into the logic to narrow down the
problems. Shall we "close" this issue even if we find out that there's still
problems with "named pipes" later on ?

The application server (the open source project) that we're trying to make
work in Windows can be found here https://bitbucket.org/casualcore/ (if
interested)

Once again, tnx Ken

Kristian

> Ken

Sv: Sv: Sv: Sv: Sv: Sv: Named pipes and multiple writers

[sten.kristian.ivarsson]

> On 3/31/2020 5:10 PM, sten.kristian.ivarsson@gmail.com wrote:
> >> On 3/28/2020 10:19 PM, Ken Brown via Cygwin wrote:
> >>> On 3/28/2020 11:43 AM, Ken Brown via Cygwin wrote:
> >>>> On 3/28/2020 8:10 AM, sten.kristian.ivarsson@gmail.com wrote:
> >>>>>> On 3/27/2020 10:53 AM, sten.kristian.ivarsson@gmail.com wrote:
> >>>>>>>> On 3/26/2020 7:19 PM, Ken Brown via Cygwin wrote:
> >>>>>>>>> On 3/26/2020 6:39 PM, Ken Brown via Cygwin wrote:
> >>>>>>>>>> On 3/26/2020 6:01 PM, sten.kristian.ivarsson@gmail.com wrote:
> >>>>>>>>>>> The ENIXIO occurs when parallel child-processes
> >>>>>>>>>>> simultaneously using O_NONBLOCK opening the descriptor.
> >>>>>>>>>>
> >>>>>>>>>> This is consistent with my guess that the error is generated
> >>>>>>>>>> by fhandler_fifo::wait.  I have a feeling that read_ready
> >>>>>>>>>> should have been created as a manual-reset event, and that
> >>>>>>>>>> more care is needed to make sure it's set when it should be.
> >>>>>>>>>>
> >>>>>>>>>>> I could provide a code-snippet to reproduce it if wanted ?
> >>>>>>>>>>
> >>>>>>>>>> Yes, please!
> >>>>>>>>>
> >>>>>>>>> That might not be necessary.  If you're able to build the git
> >>>>>>>>> repo master branch, please try the attached patch.
> >>>>>>>
> >>>>>>>> Here's a better patch.
> >>>>>>>
> >>>>>>>
> >>>>>>> I finally succeeded to build latest master (make is not my
> >>>>>>> favourite
> >>>>>>> tool) and added the patch, but still no success in my little
> >>>>>>> test-program (see
> >>>>>>> attachment) when creating a write-file-descriptor with
> >>>>>>> O_NONBLOCK
> >>>>>
> >>>>>> Your test program fails for me on Linux too.  Here's the output
> >>>>>> from one
> >>>>> run:
> >>>>>
> >>>>> You're right. That was extremely careless of me to not test this
> >>>>> in Linux first :-)
> >>>>
> >>>> No problem.
> >>>>
> >>>>> I can assure that we have a use case that works on Linux but not
> >>>>> in Cygwin, but it seems like I failed to narrow it down in the
> >>>>> wrong way
> >>>>>
> >>>>> I'll try to rearrange my code (that works in Linux) to mimic our
> >>>>> application but in a simple way (I'll be back)
> >>>>
> >>>> OK, I'll be waiting for you.  BTW, if it's not too hard to write
> >>>> your test case in plain C, or at least less modern C++, that would
> >>>> simplify things for me.  For example, your pipe.cpp failed to
> >>>> compile on one Linux machine I wanted to test it on, presumably
> >>>> because that
> > machine had an older C++ compiler.
> >>>
> >>> Never mind.  I was able to reproduce the problem and find the cause.
> >>> What happens is that when the first subprocess exits,
> >>> fhandler_fifo::close resets read_ready.  That causes the second and
> >>> subsequent subprocesses to think that there's no reader open, so
> >>> their attempts to open a writer with O_NONBLOCK fail with ENXIO.
> >>>
> >>> I should be able to fix this tomorrow.
> >
> >> I've pushed what I think is a fix to the topic/fifo branch.  I tested
> >> it
> > with the attached program, which is a variant of the test case you
> > sent last week.
> >> Please test it in your use case.
> >
> >> Note: If you've previously pulled the topic/fifo branch, then you
> >> will
> > probably get a lot of conflicts when you pull again, because I did a
> > forced push a few days ago.  If that happens, just do
> >
> >>    git reset --hard origin/topic/fifo
> >
> >> It turned out that the fix required some of the ideas that I've been
> > working on in connection with allowing multiple readers.  Even though
> > the code allows a FIFO to be *explicitly* opened for reading only
> > once, there can still be several open file descriptors for readers
> > because of dup and fork.  The existing code on git master doesn't
> > handle those situations properly.
> >
> >> The code on topic/fifo doesn't completely fix that yet, but I think
> >> it
> > should work under the following assumptions:
> >
> >> 1. The FIFO is opened only once for reading.
> >
> >> 2. The file descriptor obtained from this is the only one on which a
> >> read
> > is attempted.
> >
> >> I'm working on removing both of these restrictions.
> >
> >> Ken
> >
> > We finally took the time to make some kind of a simplified "hack" that
> > works on Ubuntu and BSD/OSX but with latest on master newlib-cygwin gave
"ENXIO"
> > now and then but with your previous patch attached, there was no ENXIO
> > but ::read returns EAGIN (until exhausted) (with cygwin) almost every
> > run
> >
> > I will try your newest things tomorrow
> >
> > See latest attatched test-program (starts to get bloated but this time
> > more C-compatible though:-)
> 
> Thanks.  This runs fine with the current HEAD of topic/fifo.

I wrote in a previous mail in this topic that it seemed to work fine for me
as well, but when I bumped up the numbers of writers and/or the number of
messages (e.g. 25/25) it starts to fail again

The initial thought is that we're bumping into some kind of system resource
limit, but I haven't had the time to dig into details (yet) (I'm sorry for
that)

Kristian

> Ken

Re: Sv: Sv: Sv: Sv: Sv: Sv: Named pipes and multiple writers

[Ken Brown]

On 4/1/2020 3:45 AM, sten.kristian.ivarsson@gmail.com wrote:
>> On 3/31/2020 5:10 PM, sten.kristian.ivarsson@gmail.com wrote:
>>>> On 3/28/2020 10:19 PM, Ken Brown via Cygwin wrote:
>>>>> On 3/28/2020 11:43 AM, Ken Brown via Cygwin wrote:
>>>>>> On 3/28/2020 8:10 AM, sten.kristian.ivarsson@gmail.com wrote:
>>>>>>>> On 3/27/2020 10:53 AM, sten.kristian.ivarsson@gmail.com wrote:
>>>>>>>>>> On 3/26/2020 7:19 PM, Ken Brown via Cygwin wrote:
>>>>>>>>>>> On 3/26/2020 6:39 PM, Ken Brown via Cygwin wrote:
>>>>>>>>>>>> On 3/26/2020 6:01 PM, sten.kristian.ivarsson@gmail.com wrote:
>>>>>>>>>>>>> The ENIXIO occurs when parallel child-processes
>>>>>>>>>>>>> simultaneously using O_NONBLOCK opening the descriptor.
> 
> [snip]
> 
>> Thanks.  This runs fine with the current HEAD of topic/fifo.
> 
> We are very grateful for your efforts
> 
> The test-program works fine now with the latest commits in topic/fifo
> 
> There are some (possible other) issues that make our application doesn't
> work as in Linux and we haven't had the time to dig into what the problems
> are yet. A quick guess is that it is related to signals and possible related
> to pselect, but we need to dig deeper into the logic to narrow down the
> problems. Shall we "close" this issue even if we find out that there's still
> problems with "named pipes" later on ?

You can write again any time that you discover further problems.  The Cygwin 
project doesn't have any mechanism for closing issues.

> The application server (the open source project) that we're trying to make
> work in Windows can be found here https://bitbucket.org/casualcore/ (if
> interested)

Thanks, I'll take a look at some point.

> Once again, tnx Ken

Thank you for reporting the problem.  The code probably hasn't been exercised 
very much up to now.

Ken

Re: Sv: Sv: Sv: Sv: Sv: Sv: Named pipes and multiple writers

[Ken Brown]

On 4/1/2020 4:52 AM, sten.kristian.ivarsson@gmail.com wrote:
>> On 3/31/2020 5:10 PM, sten.kristian.ivarsson@gmail.com wrote:
>>>> On 3/28/2020 10:19 PM, Ken Brown via Cygwin wrote:
>>>>> On 3/28/2020 11:43 AM, Ken Brown via Cygwin wrote:
>>>>>> On 3/28/2020 8:10 AM, sten.kristian.ivarsson@gmail.com wrote:
>>>>>>>> On 3/27/2020 10:53 AM, sten.kristian.ivarsson@gmail.com wrote:
>>>>>>>>>> On 3/26/2020 7:19 PM, Ken Brown via Cygwin wrote:
>>>>>>>>>>> On 3/26/2020 6:39 PM, Ken Brown via Cygwin wrote:
>>>>>>>>>>>> On 3/26/2020 6:01 PM, sten.kristian.ivarsson@gmail.com wrote:
>>>>>>>>>>>>> The ENIXIO occurs when parallel child-processes
>>>>>>>>>>>>> simultaneously using O_NONBLOCK opening the descriptor.
>>>>>>>>>>>>
>>>>>>>>>>>> This is consistent with my guess that the error is generated
>>>>>>>>>>>> by fhandler_fifo::wait.  I have a feeling that read_ready
>>>>>>>>>>>> should have been created as a manual-reset event, and that
>>>>>>>>>>>> more care is needed to make sure it's set when it should be.
>>>>>>>>>>>>
>>>>>>>>>>>>> I could provide a code-snippet to reproduce it if wanted ?
>>>>>>>>>>>>
>>>>>>>>>>>> Yes, please!
>>>>>>>>>>>
>>>>>>>>>>> That might not be necessary.  If you're able to build the git
>>>>>>>>>>> repo master branch, please try the attached patch.
>>>>>>>>>
>>>>>>>>>> Here's a better patch.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I finally succeeded to build latest master (make is not my
>>>>>>>>> favourite
>>>>>>>>> tool) and added the patch, but still no success in my little
>>>>>>>>> test-program (see
>>>>>>>>> attachment) when creating a write-file-descriptor with
>>>>>>>>> O_NONBLOCK
>>>>>>>
>>>>>>>> Your test program fails for me on Linux too.  Here's the output
>>>>>>>> from one
>>>>>>> run:
>>>>>>>
>>>>>>> You're right. That was extremely careless of me to not test this
>>>>>>> in Linux first :-)
>>>>>>
>>>>>> No problem.
>>>>>>
>>>>>>> I can assure that we have a use case that works on Linux but not
>>>>>>> in Cygwin, but it seems like I failed to narrow it down in the
>>>>>>> wrong way
>>>>>>>
>>>>>>> I'll try to rearrange my code (that works in Linux) to mimic our
>>>>>>> application but in a simple way (I'll be back)
>>>>>>
>>>>>> OK, I'll be waiting for you.  BTW, if it's not too hard to write
>>>>>> your test case in plain C, or at least less modern C++, that would
>>>>>> simplify things for me.  For example, your pipe.cpp failed to
>>>>>> compile on one Linux machine I wanted to test it on, presumably
>>>>>> because that
>>> machine had an older C++ compiler.
>>>>>
>>>>> Never mind.  I was able to reproduce the problem and find the cause.
>>>>> What happens is that when the first subprocess exits,
>>>>> fhandler_fifo::close resets read_ready.  That causes the second and
>>>>> subsequent subprocesses to think that there's no reader open, so
>>>>> their attempts to open a writer with O_NONBLOCK fail with ENXIO.
>>>>>
>>>>> I should be able to fix this tomorrow.
>>>
>>>> I've pushed what I think is a fix to the topic/fifo branch.  I tested
>>>> it
>>> with the attached program, which is a variant of the test case you
>>> sent last week.
>>>> Please test it in your use case.
>>>
>>>> Note: If you've previously pulled the topic/fifo branch, then you
>>>> will
>>> probably get a lot of conflicts when you pull again, because I did a
>>> forced push a few days ago.  If that happens, just do
>>>
>>>>     git reset --hard origin/topic/fifo
>>>
>>>> It turned out that the fix required some of the ideas that I've been
>>> working on in connection with allowing multiple readers.  Even though
>>> the code allows a FIFO to be *explicitly* opened for reading only
>>> once, there can still be several open file descriptors for readers
>>> because of dup and fork.  The existing code on git master doesn't
>>> handle those situations properly.
>>>
>>>> The code on topic/fifo doesn't completely fix that yet, but I think
>>>> it
>>> should work under the following assumptions:
>>>
>>>> 1. The FIFO is opened only once for reading.
>>>
>>>> 2. The file descriptor obtained from this is the only one on which a
>>>> read
>>> is attempted.
>>>
>>>> I'm working on removing both of these restrictions.
>>>
>>>> Ken
>>>
>>> We finally took the time to make some kind of a simplified "hack" that
>>> works on Ubuntu and BSD/OSX but with latest on master newlib-cygwin gave
> "ENXIO"
>>> now and then but with your previous patch attached, there was no ENXIO
>>> but ::read returns EAGIN (until exhausted) (with cygwin) almost every
>>> run
>>>
>>> I will try your newest things tomorrow
>>>
>>> See latest attatched test-program (starts to get bloated but this time
>>> more C-compatible though:-)
>>
>> Thanks.  This runs fine with the current HEAD of topic/fifo.
> 
> I wrote in a previous mail in this topic that it seemed to work fine for me
> as well, but when I bumped up the numbers of writers and/or the number of
> messages (e.g. 25/25) it starts to fail again
> 
> The initial thought is that we're bumping into some kind of system resource
> limit, but I haven't had the time to dig into details (yet) (I'm sorry for
> that)

Yes, it is a resource issue.  There is a limit on the number of writers that can 
be open at one time, currently 64.  I chose that number arbitrarily, with no 
idea what might actually be needed in practice, and it can easily be changed.

In addition, a writer isn't recognized as closed until a reader tries to read 
and gets an error.  In your example with 25/25, the list of writers quickly gets 
to 64 before the parent ever tries to read.

I'll see if I can find a better way to manage this.

Ken

Sv: Sv: Sv: Sv: Sv: Sv: Sv: Named pipes and multiple writers

[sten.kristian.ivarsson]

> On 4/1/2020 4:52 AM, sten.kristian.ivarsson@gmail.com wrote:
> >> On 3/31/2020 5:10 PM, sten.kristian.ivarsson@gmail.com wrote:
> >>>> On 3/28/2020 10:19 PM, Ken Brown via Cygwin wrote:
> >>>>> On 3/28/2020 11:43 AM, Ken Brown via Cygwin wrote:
> >>>>>> On 3/28/2020 8:10 AM, sten.kristian.ivarsson@gmail.com wrote:
> >>>>>>>> On 3/27/2020 10:53 AM, sten.kristian.ivarsson@gmail.com wrote:
> >>>>>>>>>> On 3/26/2020 7:19 PM, Ken Brown via Cygwin wrote:
> >>>>>>>>>>> On 3/26/2020 6:39 PM, Ken Brown via Cygwin wrote:
> >>>>>>>>>>>> On 3/26/2020 6:01 PM, sten.kristian.ivarsson@gmail.com wrote:
> >>>>>>>>>>>>> The ENIXIO occurs when parallel child-processes
> >>>>>>>>>>>>> simultaneously using O_NONBLOCK opening the descriptor.
> >>>>>>>>>>>>
> >>>>>>>>>>>> This is consistent with my guess that the error is
> >>>>>>>>>>>> generated by fhandler_fifo::wait.  I have a feeling that
> >>>>>>>>>>>> read_ready should have been created as a manual-reset
> >>>>>>>>>>>> event, and that more care is needed to make sure it's set
when it should be.

[snip] 

> >>>>> Never mind.  I was able to reproduce the problem and find the cause.
> >>>>> What happens is that when the first subprocess exits,
> >>>>> fhandler_fifo::close resets read_ready.  That causes the second
> >>>>> and subsequent subprocesses to think that there's no reader open,
> >>>>> so their attempts to open a writer with O_NONBLOCK fail with ENXIO.

[snip] 

> > I wrote in a previous mail in this topic that it seemed to work fine
> > for me as well, but when I bumped up the numbers of writers and/or the
> > number of messages (e.g. 25/25) it starts to fail again

[snip] 

> Yes, it is a resource issue.  There is a limit on the number of writers
that can be open at one
> time, currently 64.  I chose that number arbitrarily, with no idea what
might actually be
> needed in practice, and it can easily be changed.

Does it have to be a limit at all ? We would rather see that the application
decide how much resources it would like to use. In our particular case there
will be a process-manager with an incoming pipe that possible several
thousands of processes will write to

Just for fiddling around (to figure out if this is the limit that make other
things work a bit odd), where's this 64 limit defined now ?

> In addition, a writer isn't recognized as closed until a reader tries to
read and gets an error.
> In your example with 25/25, the list of writers quickly gets to 64 before
the parent ever tries
> to read.

That explains the behaviour, but should there be some error returned from
open/write (maybe it is but I'm missing it) ?

> I'll see if I can find a better way to manage this.
> 
> Ken

Kristian
Kristian 

Re: Sv: Sv: Sv: Sv: Sv: Sv: Sv: Named pipes and multiple writers

[Ken Brown]

On 4/1/2020 1:14 PM, sten.kristian.ivarsson@gmail.com wrote:
>> On 4/1/2020 4:52 AM, sten.kristian.ivarsson@gmail.com wrote:
>>>> On 3/31/2020 5:10 PM, sten.kristian.ivarsson@gmail.com wrote:
>>>>>> On 3/28/2020 10:19 PM, Ken Brown via Cygwin wrote:
>>>>>>> On 3/28/2020 11:43 AM, Ken Brown via Cygwin wrote:
>>>>>>>> On 3/28/2020 8:10 AM, sten.kristian.ivarsson@gmail.com wrote:
>>>>>>>>>> On 3/27/2020 10:53 AM, sten.kristian.ivarsson@gmail.com wrote:
>>>>>>>>>>>> On 3/26/2020 7:19 PM, Ken Brown via Cygwin wrote:
>>>>>>>>>>>>> On 3/26/2020 6:39 PM, Ken Brown via Cygwin wrote:
>>>>>>>>>>>>>> On 3/26/2020 6:01 PM, sten.kristian.ivarsson@gmail.com wrote:
>>>>>>>>>>>>>>> The ENIXIO occurs when parallel child-processes
>>>>>>>>>>>>>>> simultaneously using O_NONBLOCK opening the descriptor.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This is consistent with my guess that the error is
>>>>>>>>>>>>>> generated by fhandler_fifo::wait.  I have a feeling that
>>>>>>>>>>>>>> read_ready should have been created as a manual-reset
>>>>>>>>>>>>>> event, and that more care is needed to make sure it's set
> when it should be.
> 
> [snip]
> 
>>>>>>> Never mind.  I was able to reproduce the problem and find the cause.
>>>>>>> What happens is that when the first subprocess exits,
>>>>>>> fhandler_fifo::close resets read_ready.  That causes the second
>>>>>>> and subsequent subprocesses to think that there's no reader open,
>>>>>>> so their attempts to open a writer with O_NONBLOCK fail with ENXIO.
> 
> [snip]
> 
>>> I wrote in a previous mail in this topic that it seemed to work fine
>>> for me as well, but when I bumped up the numbers of writers and/or the
>>> number of messages (e.g. 25/25) it starts to fail again
> 
> [snip]
> 
>> Yes, it is a resource issue.  There is a limit on the number of writers
> that can be open at one
>> time, currently 64.  I chose that number arbitrarily, with no idea what
> might actually be
>> needed in practice, and it can easily be changed.
> 
> Does it have to be a limit at all ? We would rather see that the application
> decide how much resources it would like to use. In our particular case there
> will be a process-manager with an incoming pipe that possible several
> thousands of processes will write to

I agree.

> Just for fiddling around (to figure out if this is the limit that make other
> things work a bit odd), where's this 64 limit defined now ?

It's MAX_CLIENTS, defined in fhandler.h.  But there seem to be other resource 
issues also; simply increasing MAX_CLIENTS doesn't solve the problem.  I think 
there are also problems with the number of threads, for example.  Each time your 
program forks, the subprocess inherits the rfd file descriptor and its 
"fifo_reader_thread" starts up.  This is unnecessary for your application, so I 
tried disabling it (in fhandler_fifo::fixup_after_fork), just as an experiment.

But then I ran into some deadlocks, suggesting that one of the locks I'm using 
isn't robust enough.  So I've got a lot of things to work on.

>> In addition, a writer isn't recognized as closed until a reader tries to
> read and gets an error.
>> In your example with 25/25, the list of writers quickly gets to 64 before
> the parent ever tries
>> to read.
> 
> That explains the behaviour, but should there be some error returned from
> open/write (maybe it is but I'm missing it) ?

The error is discovered in add_client_handler, called from thread_func.  I think 
you'll only see it if you run the program under strace.  I'll see if I can find 
a way to report it.  Currently, there's a retry loop in fhandler_fifo::open when 
a writer tries to open, and I think I need to limit the number of retries and 
then error out.

Ken

Re: Sv: Sv: Sv: Sv: Sv: Sv: Sv: Named pipes and multiple writers

[Ken Brown]

On 4/1/2020 2:34 PM, Ken Brown via Cygwin wrote:
> On 4/1/2020 1:14 PM, sten.kristian.ivarsson@gmail.com wrote:
>>> On 4/1/2020 4:52 AM, sten.kristian.ivarsson@gmail.com wrote:
>>>>> On 3/31/2020 5:10 PM, sten.kristian.ivarsson@gmail.com wrote:
>>>>>>> On 3/28/2020 10:19 PM, Ken Brown via Cygwin wrote:
>>>>>>>> On 3/28/2020 11:43 AM, Ken Brown via Cygwin wrote:
>>>>>>>>> On 3/28/2020 8:10 AM, sten.kristian.ivarsson@gmail.com wrote:
>>>>>>>>>>> On 3/27/2020 10:53 AM, sten.kristian.ivarsson@gmail.com wrote:
>>>>>>>>>>>>> On 3/26/2020 7:19 PM, Ken Brown via Cygwin wrote:
>>>>>>>>>>>>>> On 3/26/2020 6:39 PM, Ken Brown via Cygwin wrote:
>>>>>>>>>>>>>>> On 3/26/2020 6:01 PM, sten.kristian.ivarsson@gmail.com wrote:
>>>>>>>>>>>>>>>> The ENIXIO occurs when parallel child-processes
>>>>>>>>>>>>>>>> simultaneously using O_NONBLOCK opening the descriptor.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This is consistent with my guess that the error is
>>>>>>>>>>>>>>> generated by fhandler_fifo::wait.  I have a feeling that
>>>>>>>>>>>>>>> read_ready should have been created as a manual-reset
>>>>>>>>>>>>>>> event, and that more care is needed to make sure it's set
>> when it should be.
>>
>> [snip]
>>
>>>>>>>> Never mind.  I was able to reproduce the problem and find the cause.
>>>>>>>> What happens is that when the first subprocess exits,
>>>>>>>> fhandler_fifo::close resets read_ready.  That causes the second
>>>>>>>> and subsequent subprocesses to think that there's no reader open,
>>>>>>>> so their attempts to open a writer with O_NONBLOCK fail with ENXIO.
>>
>> [snip]
>>
>>>> I wrote in a previous mail in this topic that it seemed to work fine
>>>> for me as well, but when I bumped up the numbers of writers and/or the
>>>> number of messages (e.g. 25/25) it starts to fail again
>>
>> [snip]
>>
>>> Yes, it is a resource issue.  There is a limit on the number of writers
>> that can be open at one
>>> time, currently 64.  I chose that number arbitrarily, with no idea what
>> might actually be
>>> needed in practice, and it can easily be changed.
>>
>> Does it have to be a limit at all ? We would rather see that the application
>> decide how much resources it would like to use. In our particular case there
>> will be a process-manager with an incoming pipe that possible several
>> thousands of processes will write to
> 
> I agree.
> 
>> Just for fiddling around (to figure out if this is the limit that make other
>> things work a bit odd), where's this 64 limit defined now ?
> 
> It's MAX_CLIENTS, defined in fhandler.h.  But there seem to be other resource 
> issues also; simply increasing MAX_CLIENTS doesn't solve the problem.  I think 
> there are also problems with the number of threads, for example.  Each time your 
> program forks, the subprocess inherits the rfd file descriptor and its 
> "fifo_reader_thread" starts up.  This is unnecessary for your application, so I 
> tried disabling it (in fhandler_fifo::fixup_after_fork), just as an experiment.
> 
> But then I ran into some deadlocks, suggesting that one of the locks I'm using 
> isn't robust enough.  So I've got a lot of things to work on.
> 
>>> In addition, a writer isn't recognized as closed until a reader tries to
>> read and gets an error.
>>> In your example with 25/25, the list of writers quickly gets to 64 before
>> the parent ever tries
>>> to read.
>>
>> That explains the behaviour, but should there be some error returned from
>> open/write (maybe it is but I'm missing it) ?
> 
> The error is discovered in add_client_handler, called from thread_func.  I think 
> you'll only see it if you run the program under strace.  I'll see if I can find 
> a way to report it.  Currently, there's a retry loop in fhandler_fifo::open when 
> a writer tries to open, and I think I need to limit the number of retries and 
> then error out.

I pushed a few improvements and bug fixes, and your 25/25 example now runs 
without a problem.  I increased MAX_CLIENTS to 1024 just for the sake of this 
example, but I'll work on letting the number of writers increase dynamically as 
needed.

Ken

Sv: Sv: Sv: Sv: Sv: Sv: Sv: Sv: Named pipes and multiple writers

[sten.kristian.ivarsson]

> On 4/1/2020 2:34 PM, Ken Brown via Cygwin wrote:
> > On 4/1/2020 1:14 PM, sten.kristian.ivarsson@gmail.com wrote:
> >>> On 4/1/2020 4:52 AM, sten.kristian.ivarsson@gmail.com wrote:
> >>>>> On 3/31/2020 5:10 PM, sten.kristian.ivarsson@gmail.com wrote:
> >>>>>>> On 3/28/2020 10:19 PM, Ken Brown via Cygwin wrote:
> >>>>>>>> On 3/28/2020 11:43 AM, Ken Brown via Cygwin wrote:
> >>>>>>>>> On 3/28/2020 8:10 AM, sten.kristian.ivarsson@gmail.com wrote:
> >>>>>>>>>>> On 3/27/2020 10:53 AM, sten.kristian.ivarsson@gmail.com wrote:
> >>>>>>>>>>>>> On 3/26/2020 7:19 PM, Ken Brown via Cygwin wrote:
> >>>>>>>>>>>>>> On 3/26/2020 6:39 PM, Ken Brown via Cygwin wrote:
> >>>>>>>>>>>>>>> On 3/26/2020 6:01 PM, sten.kristian.ivarsson@gmail.com
wrote:
> >>>>>>>>>>>>>>>> The ENIXIO occurs when parallel child-processes
> >>>>>>>>>>>>>>>> simultaneously using O_NONBLOCK opening the descriptor.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> This is consistent with my guess that the error is
> >>>>>>>>>>>>>>> generated by fhandler_fifo::wait.  I have a feeling that
> >>>>>>>>>>>>>>> read_ready should have been created as a manual-reset
> >>>>>>>>>>>>>>> event, and that more care is needed to make sure it's
> >>>>>>>>>>>>>>> set
> >> when it should be.
> >>
> >> [snip]
> >>
> >>>>>>>> Never mind.  I was able to reproduce the problem and find the
cause.
> >>>>>>>> What happens is that when the first subprocess exits,
> >>>>>>>> fhandler_fifo::close resets read_ready.  That causes the second
> >>>>>>>> and subsequent subprocesses to think that there's no reader
> >>>>>>>> open, so their attempts to open a writer with O_NONBLOCK fail
with ENXIO.
> >>
> >> [snip]
> >>
> >>>> I wrote in a previous mail in this topic that it seemed to work
> >>>> fine for me as well, but when I bumped up the numbers of writers
> >>>> and/or the number of messages (e.g. 25/25) it starts to fail again
> >>
> >> [snip]
> >>
> >>> Yes, it is a resource issue.  There is a limit on the number of
> >>> writers
> >> that can be open at one
> >>> time, currently 64.  I chose that number arbitrarily, with no idea
> >>> what
> >> might actually be
> >>> needed in practice, and it can easily be changed.
> >>
> >> Does it have to be a limit at all ? We would rather see that the
> >> application decide how much resources it would like to use. In our
> >> particular case there will be a process-manager with an incoming pipe
> >> that possible several thousands of processes will write to
> >
> > I agree.
> >
> >> Just for fiddling around (to figure out if this is the limit that
> >> make other things work a bit odd), where's this 64 limit defined now ?
> >
> > It's MAX_CLIENTS, defined in fhandler.h.  But there seem to be other
> > resource issues also; simply increasing MAX_CLIENTS doesn't solve the
> > problem.  I think there are also problems with the number of threads,
> > for example.  Each time your program forks, the subprocess inherits
> > the rfd file descriptor and its "fifo_reader_thread" starts up.  This
> > is unnecessary for your application, so I tried disabling it (in
> fhandler_fifo::fixup_after_fork), just as an experiment.
> >
> > But then I ran into some deadlocks, suggesting that one of the locks
> > I'm using isn't robust enough.  So I've got a lot of things to work on.
> >
> >>> In addition, a writer isn't recognized as closed until a reader
> >>> tries to
> >> read and gets an error.
> >>> In your example with 25/25, the list of writers quickly gets to 64
> >>> before
> >> the parent ever tries
> >>> to read.
> >>
> >> That explains the behaviour, but should there be some error returned
> >> from open/write (maybe it is but I'm missing it) ?
> >
> > The error is discovered in add_client_handler, called from
> > thread_func.  I think you'll only see it if you run the program under
> > strace.  I'll see if I can find a way to report it.  Currently,
> > there's a retry loop in fhandler_fifo::open when a writer tries to
> > open, and I think I need to limit the number of retries and then error
out.
> 
> I pushed a few improvements and bug fixes, and your 25/25 example now runs
without a
> problem.  I increased MAX_CLIENTS to 1024 just for the sake of this
example, but I'll
> work on letting the number of writers increase dynamically as needed.

I pulled it and tried it out and yes, the sample test program with 25/25
worked well and a whole bunch of our unit-tests passed with ok result now

We still do have some issues, but I cannot yet tell if they are related to
named pipes or not

It is great that you're looking into a totally dynamic solution

Kristian

> Ken

Re: Sv: Sv: Sv: Sv: Sv: Sv: Sv: Sv: Named pipes and multiple wri

[Gregery Barton]

How do I unsubscribed from cygwin mailing lists? I tried simple unsubscribe but it's not so simple. I don't have a password and it won't send me one.

Sent from Yahoo7 Mail on Android 
 
  On Thu, 2 Apr 2020 at 7:06 pm, Kristian Ivarsson via Cygwin<cygwin@cygwin.com> wrote:   > On 4/1/2020 2:34 PM, Ken Brown via Cygwin wrote:
> > On 4/1/2020 1:14 PM, sten.kristian.ivarsson@gmail.com wrote:
> >>> On 4/1/2020 4:52 AM, sten.kristian.ivarsson@gmail.com wrote:
> >>>>> On 3/31/2020 5:10 PM, sten.kristian.ivarsson@gmail.com wrote:
> >>>>>>> On 3/28/2020 10:19 PM, Ken Brown via Cygwin wrote:
> >>>>>>>> On 3/28/2020 11:43 AM, Ken Brown via Cygwin wrote:
> >>>>>>>>> On 3/28/2020 8:10 AM, sten.kristian.ivarsson@gmail.com wrote:
> >>>>>>>>>>> On 3/27/2020 10:53 AM, sten.kristian.ivarsson@gmail.com wrote:
> >>>>>>>>>>>>> On 3/26/2020 7:19 PM, Ken Brown via Cygwin wrote:
> >>>>>>>>>>>>>> On 3/26/2020 6:39 PM, Ken Brown via Cygwin wrote:
> >>>>>>>>>>>>>>> On 3/26/2020 6:01 PM, sten.kristian.ivarsson@gmail.com
wrote:
> >>>>>>>>>>>>>>>> The ENIXIO occurs when parallel child-processes
> >>>>>>>>>>>>>>>> simultaneously using O_NONBLOCK opening the descriptor.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> This is consistent with my guess that the error is
> >>>>>>>>>>>>>>> generated by fhandler_fifo::wait.  I have a feeling that
> >>>>>>>>>>>>>>> read_ready should have been created as a manual-reset
> >>>>>>>>>>>>>>> event, and that more care is needed to make sure it's
> >>>>>>>>>>>>>>> set
> >> when it should be.
> >>
> >> [snip]
> >>
> >>>>>>>> Never mind.  I was able to reproduce the problem and find the
cause.
> >>>>>>>> What happens is that when the first subprocess exits,
> >>>>>>>> fhandler_fifo::close resets read_ready.  That causes the second
> >>>>>>>> and subsequent subprocesses to think that there's no reader
> >>>>>>>> open, so their attempts to open a writer with O_NONBLOCK fail
with ENXIO.
> >>
> >> [snip]
> >>
> >>>> I wrote in a previous mail in this topic that it seemed to work
> >>>> fine for me as well, but when I bumped up the numbers of writers
> >>>> and/or the number of messages (e.g. 25/25) it starts to fail again
> >>
> >> [snip]
> >>
> >>> Yes, it is a resource issue.  There is a limit on the number of
> >>> writers
> >> that can be open at one
> >>> time, currently 64.  I chose that number arbitrarily, with no idea
> >>> what
> >> might actually be
> >>> needed in practice, and it can easily be changed.
> >>
> >> Does it have to be a limit at all ? We would rather see that the
> >> application decide how much resources it would like to use. In our
> >> particular case there will be a process-manager with an incoming pipe
> >> that possible several thousands of processes will write to
> >
> > I agree.
> >
> >> Just for fiddling around (to figure out if this is the limit that
> >> make other things work a bit odd), where's this 64 limit defined now ?
> >
> > It's MAX_CLIENTS, defined in fhandler.h.  But there seem to be other
> > resource issues also; simply increasing MAX_CLIENTS doesn't solve the
> > problem.  I think there are also problems with the number of threads,
> > for example.  Each time your program forks, the subprocess inherits
> > the rfd file descriptor and its "fifo_reader_thread" starts up.  This
> > is unnecessary for your application, so I tried disabling it (in
> fhandler_fifo::fixup_after_fork), just as an experiment.
> >
> > But then I ran into some deadlocks, suggesting that one of the locks
> > I'm using isn't robust enough.  So I've got a lot of things to work on.
> >
> >>> In addition, a writer isn't recognized as closed until a reader
> >>> tries to
> >> read and gets an error.
> >>> In your example with 25/25, the list of writers quickly gets to 64
> >>> before
> >> the parent ever tries
> >>> to read.
> >>
> >> That explains the behaviour, but should there be some error returned
> >> from open/write (maybe it is but I'm missing it) ?
> >
> > The error is discovered in add_client_handler, called from
> > thread_func.  I think you'll only see it if you run the program under
> > strace.  I'll see if I can find a way to report it.  Currently,
> > there's a retry loop in fhandler_fifo::open when a writer tries to
> > open, and I think I need to limit the number of retries and then error
out.
> 
> I pushed a few improvements and bug fixes, and your 25/25 example now runs
without a
> problem.  I increased MAX_CLIENTS to 1024 just for the sake of this
example, but I'll
> work on letting the number of writers increase dynamically as needed.

I pulled it and tried it out and yes, the sample test program with 25/25
worked well and a whole bunch of our unit-tests passed with ok result now

We still do have some issues, but I cannot yet tell if they are related to
named pipes or not

It is great that you're looking into a totally dynamic solution

Kristian

> Ken

--
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:    https://cygwin.com/ml/#unsubscribe-simple  

Re: Sv: Sv: Sv: Sv: Sv: Sv: Sv: Sv: Named pipes and multiple writers

[Ken Brown]

On 4/2/2020 4:05 AM, sten.kristian.ivarsson@gmail.com wrote:
> It is great that you're looking into a totally dynamic solution

I've just pushed a first attempt at this.  I tested it by running your test 
program with parameters 50/50.  It looks like there are about 2000 writers open 
at once during this run.

Ken