From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 6057 invoked by alias); 6 Sep 2011 10:12:35 -0000 Received: (qmail 6048 invoked by uid 22791); 6 Sep 2011 10:12:33 -0000 X-SWARE-Spam-Status: No, hits=-6.6 required=5.0 tests=AWL,BAYES_00,RCVD_IN_DNSWL_HI,RP_MATCHES_RCVD,SPF_HELO_PASS,TW_RK X-Spam-Check-By: sourceware.org Received: from mx1.redhat.com (HELO mx1.redhat.com) (209.132.183.28) by sourceware.org (qpsmtpd/0.43rc1) with ESMTP; Tue, 06 Sep 2011 10:12:16 +0000 Received: from int-mx12.intmail.prod.int.phx2.redhat.com (int-mx12.intmail.prod.int.phx2.redhat.com [10.5.11.25]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id p86ACFZG003160 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Tue, 6 Sep 2011 06:12:16 -0400 Received: from springer.wildebeest.org (ovpn-113-38.phx2.redhat.com [10.3.113.38]) by int-mx12.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id p86ACEuV009444 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 6 Sep 2011 06:12:15 -0400 Received: by springer.wildebeest.org (Postfix, from userid 500) id 4352D409C3; Tue, 6 Sep 2011 12:12:12 +0200 (CEST) Subject: RE: Making the transport layer more robust From: Mark Wielaard To: "Turgis, Frederic" Cc: "systemtap@sourceware.org" In-Reply-To: <13872098A06B02418CF379A158C0F1460163182604@dnce02.ent.ti.com> References: <1311065908.9144.27.camel@springer.wildebeest.org> <20110812174324.GA1394@hermans.wildebeest.org> <4E4965B3.6080700@redhat.com> <1313500965.3393.5.camel@springer.wildebeest.org> <13872098A06B02418CF379A158C0F1460162DC0A08@dnce02.ent.ti.com> <1315222009.3431.22.camel@springer.wildebeest.org> <13872098A06B02418CF379A158C0F1460163182604@dnce02.ent.ti.com> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Date: Tue, 06 Sep 2011 10:12:00 -0000 Message-ID: <1315303931.3895.30.camel@springer.wildebeest.org> Mime-Version: 1.0 Mailing-List: contact systemtap-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Post: List-Help: , Sender: systemtap-owner@sourceware.org X-SW-Source: 2011-q3/txt/msg00272.txt.bz2 Hi Turgis, On Mon, 2011-09-05 at 16:04 +0200, Turgis, Frederic wrote: > >The kernel side "polling" is not just for exit, it is for any > >cmd message that is generated from a possible "unsafe" > >context >=20 > I have then probably not understood code correctly (code was before > latest changes, now this polling is mandatory as I mentioned later in > mail). "unsafe" context is associated to unannounced message, no ? Yes, it depends on the probe point the message is generated from. Since in general we cannot know whether a probe is "safe" or not, we always just queue the message and continue. The kernel timer then periodically checks the queue (and the exit flag) and announces it. I added a particularly nasty example of a probe point where immediately announcing the message to user space would be really bad: testsuite/systemtap.base/wakeup.exp > Well, even for announced messages, I have the impression that reading > message relies only on user side polling because user side is not > waiting for a wake-up of _stp_ctl_ready_q. Here is my understanding > but I didn't take time to perform some traces: >=20 > - for annouced messages or on kernel polling (_stp_ctl_work_callback), > I understand that we trigger a reading through > wake_up_interruptible(&_stp_ctl_wq); Correct. > - on user side, main reading loop does: > flags |=3D O_NONBLOCK; > fcntl(control_channel, F_SETFL, flags); > nb =3D read(control_channel, &recvbuf, sizeof(recvbuf)); > So I expect a non blocking read (however, there may be another place wher= e we read cmd message) Indeed. That is the first pass, if there is anything to read, we read it, but then (if select is supported) we call select() on the control_channel. This means we will wait there in user space until the kernel explicitly wakes us up to tell there are messages waiting, but we don't need to poll ourselves in user space. Only when select returns will we try to read a message from the kernel again. > This ends in "_stp_ctl_read_cmd" in kernel doing: > while (list_empty(&_stp_ctl_ready_q)) { > spin_unlock_irqrestore(&_stp_ctl_ready_lock, flags); > if (file->f_flags & O_NONBLOCK) -> non blocking read, we = rely on polling to recheck _stp_ctl_ready_q > return -EAGAIN; > if (wait_event_interruptible(_stp_ctl_wq, !list_empty(&_s= tp_ctl_ready_q))) -> code not reached so kernel polling (or even message an= noucement) useless ? > return -ERESTARTSYS; Yeah, you are right, that second bit is a little strange, we kind of depend on user space always calling us with O_NONBLOCK. But if it doesn't, then at least we do something somewhat sensible. > >I am very interested in any results you get from the new code. >=20 > Never tested bulk mode. We quite like filling up buffer and doing a > big dump but doing very small regular dumps could also work. I think you should also try bulk mode (stap -b) if possible. It won't give you immediate results, but because of the extra buffering it should definitely be less overhead. > Our modifications are just ugly hacks to understand the internals. > They make sense for some, but for some other parts, we probably have > different requirements between a server and an embedded platform. > Capability to tune a timer would be OK (or maybe bulk-mode would be > good) >=20 > Here are the v1.3 experiments we performed few months ago (latest > months have been too busy with customer to share before :-( ). Goal > was to remove "stapio:5192" process and "systemtap/1:5189" workqueue > from waking up too often, here every 200ms and 7.8125ms (HZ=3D128) > respectively. Attached picture shows what each core runs in time > (blue=3Dcore 0, red=3Dcore 1, the lower a process, the more it runs, > except swapper=3DIdle tasks always at bottom) Looks like the attachement somehow didn't reach the mailinglist. odd. > Control channel userpace polling (well, I consider control everything tha= t is not trace/data output from script) > - usleep (250*1000); /* sleep 250ms between polls */ > + usleep (2000*1000); /* sleep 250ms between polls */ So with current git trunk, you get pselect support, so don't need to tweak this anymore. > Control channel kernel polling (you might find it a bit extreme ;-) ) > - if (likely(_stp_ctl_attached)) > - queue_delayed_work(_stp_wq, &_stp_work, STP_WORK_TIMER); > + //if (likely(_stp_ctl_attached)) > + // queue_delayed_work(_stp_wq, &_stp_work, STP_WORK_TIMER); yeah, that is somewhat extreme, if possible try just -DSTP_WORK_TIMER=3D... to get a good balance. I am interested in what value works well for you. > Data channel userspace timeout of select() > - struct timespec tim =3D {.tv_sec=3D0, .tv_nsec=3D200000000}, *tim= eout =3D &tim; > + struct timespec tim =3D {.tv_sec=3D5, .tv_nsec=3D0}, *timeout =3D= &tim; With bulk mode this shouldn't be necessary. But it might be nice to make this tunable too with some -D option. > Data channel kernel polling > -#define STP_RELAY_TIMER_INTERVAL ((HZ + 99) / 100) > +#define STP_RELAY_TIMER_INTERVAL HZ /* ((HZ + 99) / 1= 00) */ -> wake-up every s Hopefully you can use a -DSTP_RELAY_TIMER_INTERVAL for this too on the command line instead. I am really hoping we make it so that you don't have to hack the sources anymore just for experimenting with the timers/poll intervals. Thanks for your experiments and report. Cheers, Mark