[ECOS] Serial overruns

[ECOS] Serial overruns

[Matt Holgate]

Hi,

I have a very simple application with two threads - one that reads from a
serial port into a circular buffer (the producer), and the other that
writes the data back out to the same serial port from the buffer (the
consumer). The effect should be that any data sent over the serial line is
echoed back. Mutexes and condition variables are used to synchronise the
two threads and protect the shared buffer.

The program runs fine if send a small amount of data down the serial line.
However, if I send more than 16 characters (the size of the FIFO in the
UART), then I get buffer overruns and only receive the first 16
characters.

I've enabled hardware flow control, although I'm pretty sure the problem
is that the serial DSR is not responding in time to keep up with the data
rate (only 38400 baud), rather than the eCos buffer overflowing (it is set
to the default size of 128 bytes).

I've no idea what the problem is, so any suggestions would be much
appreciated!

thanks,
Matt

Re: [ECOS] Serial overruns

[Robin Farine]

Two things to try:

Adjust the RX FIFO threshold.

To reduce delays, you could get rid of the producer thread by filling the ring
from the ISR and replacing the mutex/condition_var by a counting semaphore that
gets signalled from the DSR.

Robin

Matt Holgate <matt@dcs.gla.ac.uk> writes:

> Hi,
> 
> I have a very simple application with two threads - one that reads from a
> serial port into a circular buffer (the producer), and the other that
> writes the data back out to the same serial port from the buffer (the
> consumer). The effect should be that any data sent over the serial line is
> echoed back. Mutexes and condition variables are used to synchronise the
> two threads and protect the shared buffer.
> 
> The program runs fine if send a small amount of data down the serial line.
> However, if I send more than 16 characters (the size of the FIFO in the
> UART), then I get buffer overruns and only receive the first 16
> characters.
> 
> I've enabled hardware flow control, although I'm pretty sure the problem
> is that the serial DSR is not responding in time to keep up with the data
> rate (only 38400 baud), rather than the eCos buffer overflowing (it is set
> to the default size of 128 bytes).
> 
> I've no idea what the problem is, so any suggestions would be much
> appreciated!
> 
> thanks,
> Matt

RE: [ECOS] Serial overruns

[Robert Cragie]

I have had problems with serial overruns on the ARM Integrator platform
running at 57600 bps. Enabling flow control made no difference. Also the Rx
FIFO threshold is fixed at 8 in the ARM Prime Cell UARTs.

I added an extra interim buffer of 256 bytes between the ISR and the DSR -
this improved the situation but did not eliminate the overruns.

The problem only occurred when running out of Flash memory; I ended up
putting the problem down to the 'interesting' bus FIFO implementation which
buffers up accesses from the CPU module. The solution ended up being
bootstrapping the code to RAM first.

In your case, I would suggest perhaps trying the interim buffer approach.
What platform are you using?

Regards,

Robert Cragie
Design Engineer
Jennic Ltd.
Furnival Street
Sheffield
S1 4QT
United Kingdom
Tel: +44 (0) 114 281 4512
Fax: +44 (0) 114 281 2951
mailto:rcc@jennic.com
http://www.jennic.com


> -----Original Message-----
> From: ecos-discuss-owner@sources.redhat.com
> [ mailto:ecos-discuss-owner@sources.redhat.com]On Behalf Of Matt Holgate
> Sent: 17 September 2001 18:33
> To: ecos-discuss@sources.redhat.com
> Subject: [ECOS] Serial overruns
>
>
> Hi,
>
> I have a very simple application with two threads - one that reads from a
> serial port into a circular buffer (the producer), and the other that
> writes the data back out to the same serial port from the buffer (the
> consumer). The effect should be that any data sent over the serial line is
> echoed back. Mutexes and condition variables are used to synchronise the
> two threads and protect the shared buffer.
>
> The program runs fine if send a small amount of data down the serial line.
> However, if I send more than 16 characters (the size of the FIFO in the
> UART), then I get buffer overruns and only receive the first 16
> characters.
>
> I've enabled hardware flow control, although I'm pretty sure the problem
> is that the serial DSR is not responding in time to keep up with the data
> rate (only 38400 baud), rather than the eCos buffer overflowing (it is set
> to the default size of 128 bytes).
>
> I've no idea what the problem is, so any suggestions would be much
> appreciated!
>
> thanks,
> Matt
>
>
>
>
>
>
>

Re: [ECOS] Serial overruns

[Matt Holgate]

"Robert Cragie" <rcc@jennic.com> writes:

> In your case, I would suggest perhaps trying the interim buffer approach.
> What platform are you using?

I'm using a standard i386 PC, booting with redboot over the other
serial line.

I tried using a different RX threshold, but this made no difference
(it was already set at 1 anyway, so I wouldn't have expected any
improvement). I also tried dropping the speed to 9600 baud, but still
got overruns, although not as many.

I'll give the interim buffer approach a go and let you know how I get
on.

Thanks for all the help,

Matt

RE: [ECOS] Serial overruns

[David Airlie]

>
> The problem only occurred when running out of Flash memory; I ended up
> putting the problem down to the 'interesting' bus FIFO implementation which
> buffers up accesses from the CPU module. The solution ended up being
> bootstrapping the code to RAM first.

I think putting it down to the simpler FLASH has more wait states than RAM
so is slower so the ISR takes longer to run so it drops bytes? that's what
I put similiar problems on my own internal ARM board down to :-)

I made the ROMRAM patches that I contributed back for that reason
(Jonathan?? :-)

Dave.

>
> In your case, I would suggest perhaps trying the interim buffer approach.
> What platform are you using?
>
> Regards,
>
> Robert Cragie
> Design Engineer
> Jennic Ltd.
> Furnival Street
> Sheffield
> S1 4QT
> United Kingdom
> Tel: +44 (0) 114 281 4512
> Fax: +44 (0) 114 281 2951
> mailto:rcc@jennic.com
> http://www.jennic.com
>
>
> > -----Original Message-----
> > From: ecos-discuss-owner@sources.redhat.com
> > [ mailto:ecos-discuss-owner@sources.redhat.com]On Behalf Of Matt Holgate
> > Sent: 17 September 2001 18:33
> > To: ecos-discuss@sources.redhat.com
> > Subject: [ECOS] Serial overruns
> >
> >
> > Hi,
> >
> > I have a very simple application with two threads - one that reads from a
> > serial port into a circular buffer (the producer), and the other that
> > writes the data back out to the same serial port from the buffer (the
> > consumer). The effect should be that any data sent over the serial line is
> > echoed back. Mutexes and condition variables are used to synchronise the
> > two threads and protect the shared buffer.
> >
> > The program runs fine if send a small amount of data down the serial line.
> > However, if I send more than 16 characters (the size of the FIFO in the
> > UART), then I get buffer overruns and only receive the first 16
> > characters.
> >
> > I've enabled hardware flow control, although I'm pretty sure the problem
> > is that the serial DSR is not responding in time to keep up with the data
> > rate (only 38400 baud), rather than the eCos buffer overflowing (it is set
> > to the default size of 128 bytes).
> >
> > I've no idea what the problem is, so any suggestions would be much
> > appreciated!
> >
> > thanks,
> > Matt
> >
> >
> >
> >
> >
> >
> >
>

-- 
      David Airlie, Software Engineer, Parthus Technologies plc.,
       Mary Rosse Centre, National Tech Park, Limerick, Ireland.
   t: +353-61-508116 / f: +353-61-508101 / David.Airlie@parthus.com

RE: [ECOS] Serial overruns

[David Airlie]

>
> I made the ROMRAM patches that I contributed back for that reason
> (Jonathan?? :-)

*hide*.. just noticed that stuff is in there :-)...

apologies..

Dave.

>
> Dave.
>
> >
> > In your case, I would suggest perhaps trying the interim buffer approach.
> > What platform are you using?
> >
> > Regards,
> >
> > Robert Cragie
> > Design Engineer
> > Jennic Ltd.
> > Furnival Street
> > Sheffield
> > S1 4QT
> > United Kingdom
> > Tel: +44 (0) 114 281 4512
> > Fax: +44 (0) 114 281 2951
> > mailto:rcc@jennic.com
> > http://www.jennic.com
> >
> >
> > > -----Original Message-----
> > > From: ecos-discuss-owner@sources.redhat.com
> > > [ mailto:ecos-discuss-owner@sources.redhat.com]On Behalf Of Matt Holgate
> > > Sent: 17 September 2001 18:33
> > > To: ecos-discuss@sources.redhat.com
> > > Subject: [ECOS] Serial overruns
> > >
> > >
> > > Hi,
> > >
> > > I have a very simple application with two threads - one that reads from a
> > > serial port into a circular buffer (the producer), and the other that
> > > writes the data back out to the same serial port from the buffer (the
> > > consumer). The effect should be that any data sent over the serial line is
> > > echoed back. Mutexes and condition variables are used to synchronise the
> > > two threads and protect the shared buffer.
> > >
> > > The program runs fine if send a small amount of data down the serial line.
> > > However, if I send more than 16 characters (the size of the FIFO in the
> > > UART), then I get buffer overruns and only receive the first 16
> > > characters.
> > >
> > > I've enabled hardware flow control, although I'm pretty sure the problem
> > > is that the serial DSR is not responding in time to keep up with the data
> > > rate (only 38400 baud), rather than the eCos buffer overflowing (it is set
> > > to the default size of 128 bytes).
> > >
> > > I've no idea what the problem is, so any suggestions would be much
> > > appreciated!
> > >
> > > thanks,
> > > Matt
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> >
>
>

-- 
      David Airlie, Software Engineer, Parthus Technologies plc.,
       Mary Rosse Centre, National Tech Park, Limerick, Ireland.
   t: +353-61-508116 / f: +353-61-508101 / David.Airlie@parthus.com

Re: [ECOS] Serial overruns

[Jonathan Larmour]

David Airlie wrote:
> 
> >
> > I made the ROMRAM patches that I contributed back for that reason
> > (Jonathan?? :-)
> 
> *hide*.. just noticed that stuff is in there :-)...

Only some of it it seems. Right - I promise if you send me an updated patch
I'll apply it in some form this week (or soon after if you don't reply this
week :-)).

Jifl
-- 
Red Hat, Rustat House, Clifton Road, Cambridge, UK. Tel: +44 (1223) 271062
Maybe this world is another planet's Hell -Aldous Huxley || Opinions==mine

Re: [ECOS] Serial overruns

[Jonathan Larmour]

Matt Holgate wrote:
> 
> I've enabled hardware flow control, although I'm pretty sure the problem
> is that the serial DSR is not responding in time to keep up with the data
> rate (only 38400 baud), rather than the eCos buffer overflowing (it is set
> to the default size of 128 bytes).

Still, hardware flow control should prevent this. It looks like it's gotten
broken since the PC target uses the generic 16550 serial driver (assuming
you are using anonymous CVS, not 1.3.1), and that supports it.

This used to work. So if anyone has time to look at that, that would be
good.

Jifl
-- 
Red Hat, Rustat House, Clifton Road, Cambridge, UK. Tel: +44 (1223) 271062
Maybe this world is another planet's Hell -Aldous Huxley || Opinions==mine

Re: [ECOS] Serial overruns

[Matt Holgate]

Jonathan Larmour <jlarmour@redhat.com> writes:

> Still, hardware flow control should prevent this. It looks like it's gotten
> broken since the PC target uses the generic 16550 serial driver (assuming
> you are using anonymous CVS, not 1.3.1), and that supports it.

What, even though it is the FIFO in the chip that is overflowing? I
thought the hardware flow control was activated by software when the
buffer in the serial driver reached the high watermark? If this is the
case, then hardware flow control won't help a bit...or can the UART
itself active flow control?

Incidentally, I tried an intermediate buffer in the ISR, and it didn't
make any difference, so I'm guessing the ISR is just not getting run
in time.

> This used to work. So if anyone has time to look at that, that would be
> good.

I'll have a look, see what I can come up with. It could well be
something else I've screwed up :), so I'll test it with a much
simpler application.

I'm using anonymous CVS, although not the very most recent version so
I'll do an update first.

Thanks,

Matt

Re: [ECOS] Serial overruns

[Jonathan Larmour]

Matt Holgate wrote:
> 
> Jonathan Larmour <jlarmour@redhat.com> writes:
> 
> > Still, hardware flow control should prevent this. It looks like it's gotten
> > broken since the PC target uses the generic 16550 serial driver (assuming
> > you are using anonymous CVS, not 1.3.1), and that supports it.
> 
> What, even though it is the FIFO in the chip that is overflowing? I
> thought the hardware flow control was activated by software when the
> buffer in the serial driver reached the high watermark? If this is the
> case, then hardware flow control won't help a bit...or can the UART
> itself active flow control?

I also meant that the processor would have to be really slow (or
interrupts/DSRs disabled for a long time) for it to miss the opportunity to
run the DSR. If you are (as I recall) on a PC, then assuming it is recent
it should have more than enough oomph.

You could try looking at lowering the RX threshold, but it's looking like
there's something else going on.

A thought: do you have the network package in your configuration? If so,
try removing it ("ecosconfig remove net").

> I'm using anonymous CVS, although not the very most recent version so
> I'll do an update first.

I don't believe anything has changed in this area for a while.

Jifl
-- 
Red Hat, Rustat House, Clifton Road, Cambridge, UK. Tel: +44 (1223) 271062
Maybe this world is another planet's Hell -Aldous Huxley || Opinions==mine

Re: [ECOS] Serial overruns

[Jesper Skov]

>>>>> "Matt" == Matt Holgate <matt@dcs.gla.ac.uk> writes:

Matt> Jonathan Larmour <jlarmour@redhat.com> writes:
>> Still, hardware flow control should prevent this. It looks like
>> it's gotten broken since the PC target uses the generic 16550
>> serial driver (assuming you are using anonymous CVS, not 1.3.1),
>> and that supports it.

Matt> What, even though it is the FIFO in the chip that is
Matt> overflowing? I thought the hardware flow control was activated
Matt> by software when the buffer in the serial driver reached the
Matt> high watermark? If this is the case, then hardware flow control
Matt> won't help a bit...or can the UART itself active flow control?

Indeed - that's what HW flow control is about.

Jesper

Re: [ECOS] Serial overruns

[Jonathan Larmour]

Jesper Skov wrote:
> 
> >>>>> "Matt" == Matt Holgate <matt@dcs.gla.ac.uk> writes:
> 
> Matt> Jonathan Larmour <jlarmour@redhat.com> writes:
> >> Still, hardware flow control should prevent this. It looks like
> >> it's gotten broken since the PC target uses the generic 16550
> >> serial driver (assuming you are using anonymous CVS, not 1.3.1),
> >> and that supports it.
> 
> Matt> What, even though it is the FIFO in the chip that is
> Matt> overflowing? I thought the hardware flow control was activated
> Matt> by software when the buffer in the serial driver reached the
> Matt> high watermark? If this is the case, then hardware flow control
> Matt> won't help a bit...or can the UART itself active flow control?
> 
> Indeed - that's what HW flow control is about.

I don't believe so actually in general:
http://www.linuxdoc.org/HOWTO/Text-Terminal-HOWTO-10.html#ss10.8
and
http://www.uclinux.org/pub/uCsimm/archive/0524.html

I do think you came across some hardware that _did_ really do it in
hardware though which may be what you were thinking of.

Jifl
-- 
Red Hat, Rustat House, Clifton Road, Cambridge, UK. Tel: +44 (1223) 271062
Maybe this world is another planet's Hell -Aldous Huxley || Opinions==mine

Re: [ECOS] Serial overruns

[Jesper Skov]

>>>>> "Jonathan" == Jonathan Larmour <jlarmour@redhat.com> writes:

Jonathan> Jesper Skov wrote:
>>  >>>>> "Matt" == Matt Holgate <matt@dcs.gla.ac.uk> writes:
>> 
Matt> Jonathan Larmour <jlarmour@redhat.com> writes:
>> >> Still, hardware flow control should prevent this. It looks like
>> >> it's gotten broken since the PC target uses the generic 16550 >>
>> serial driver (assuming you are using anonymous CVS, not 1.3.1), >>
>> and that supports it.
>> 
Matt> What, even though it is the FIFO in the chip that is
Matt> overflowing? I thought the hardware flow control was activated
Matt> by software when the buffer in the serial driver reached the
Matt> high watermark? If this is the case, then hardware flow control
Matt> won't help a bit...or can the UART itself active flow control?
>>  Indeed - that's what HW flow control is about.

Jonathan> I don't believe so actually in general:
Jonathan> http://www.linuxdoc.org/HOWTO/Text-Terminal-HOWTO-10.html#ss10.8
Jonathan> and http://www.uclinux.org/pub/uCsimm/archive/0524.html

I see.

Jonathan> I do think you came across some hardware that _did_ really
Jonathan> do it in hardware though which may be what you were thinking
Jonathan> of.

Probably. For reference, it's the SH serial controllers that do it all
automagically.

Jesper

RE: [ECOS] Serial overruns

[Robert Cragie]

> Jonathan Larmour <jlarmour@redhat.com> writes:
>
> > Still, hardware flow control should prevent this. It looks like
> it's gotten
> > broken since the PC target uses the generic 16550 serial driver
> (assuming
> > you are using anonymous CVS, not 1.3.1), and that supports it.
>
> What, even though it is the FIFO in the chip that is overflowing? I
> thought the hardware flow control was activated by software when the
> buffer in the serial driver reached the high watermark? If this is the
> case, then hardware flow control won't help a bit...or can the UART
> itself active flow control?

Some UARTs can handle h/w flow control automatically in the chip - the bog
standard 16550 can't. So I agree that it won't make any difference.

> Incidentally, I tried an intermediate buffer in the ISR, and it didn't
> make any difference, so I'm guessing the ISR is just not getting run
> in time.

I had a look at my initial solution and realised it was flawed simply
because the ISR was disabling all interrupts on the serial port, i.e.:

<snip>
    cyg_drv_interrupt_mask(apcm7tdmi_chan->int_num);
    cyg_drv_interrupt_acknowledge(apcm7tdmi_chan->int_num);
    return CYG_ISR_CALL_DSR;  // Cause DSR to be run
}

at the end of the function.

(Incidentally, is it supposed to return CYG_ISR_HANDLED | CYG_ISR_CALL_DSR
or just CYG_ISR_CALL_DSR at the end? The use seems inconsistent between
platforms)

This means that the ISR will not run again until re-enabled right at the end
of the DSR, so the interim buffer only helps by sucking out (in my case) 8
characters from the Rx FIFO, which gives it twice as long to overflow -
hence the small improvement I saw. Having the serial port interrupt disabled
from the ISR to the end of the DSR would therefore I guess probably account
for the overflow on the receive side I am seeing, especially when running
from relatively slow memory.

I have subsequently modified my serial port drivers to only disable the
interrupts which are handled in the DSR, i.e. leave Rx interrupts enabled,
and to use the interim buffer. This seems to have fixed the problem - I can
now run out of flash with no overrun errors.

I am in the process of applying the equivalent changes to ser_16x5x.c, but I
have no way of testing it. It's not quite as straightforward as the ARM
Prime Cell UART to implement. If you'd like me to send you the completed
file, let me know.

>
> > This used to work. So if anyone has time to look at that, that would be
> > good.
>
> I'll have a look, see what I can come up with. It could well be
> something else I've screwed up :), so I'll test it with a much
> simpler application.

Any other information you find out would be appreciated.

Robert Cragie
Design Engineer
Jennic Ltd.
Furnival Street
Sheffield
S1 4QT
United Kingdom
Tel: +44 (0) 114 281 4512
Fax: +44 (0) 114 281 2951
mailto:rcc@jennic.com
http://www.jennic.com

Re: [ECOS] Serial overruns

[Jonathan Larmour]

Robert Cragie wrote:
> 
> (Incidentally, is it supposed to return CYG_ISR_HANDLED | CYG_ISR_CALL_DSR
> or just CYG_ISR_CALL_DSR at the end? The use seems inconsistent between
> platforms)

You set CYG_ISR_HANDLED to indicate that if we have chained interrupts, no
more chained ISRs should be called - this one has handled it. On many
platforms this is irrelevant because we know we are (or should be) the only
ISR on this vector, hence the inconsistent use. Although, generally you
would set it if setting CYG_ISR_CALL_DSR (I can't think of any situation
you wouldn't in fact).

When doing interrupt chaining the ISR should check whether the device
associated with the driver really is the one that caused the interrupt. We
should do this in the generic drivers really, conditionally on chained
interrupts.

> I am in the process of applying the equivalent changes to ser_16x5x.c, but I
> have no way of testing it. It's not quite as straightforward as the ARM
> Prime Cell UART to implement. If you'd like me to send you the completed
> file, let me know.

If you are considering sending this back to Red Hat, please be sure to make
it configurable - not every platform needs the extra complexity/buffer
space required.

Jifl
-- 
Red Hat, Rustat House, Clifton Road, Cambridge, UK. Tel: +44 (1223) 271062
Maybe this world is another planet's Hell -Aldous Huxley || Opinions==mine

Re: [ECOS] Serial overruns

[Matt Holgate]

"Robert Cragie" <rcc@jennic.com> writes:

> I have subsequently modified my serial port drivers to only disable the
> interrupts which are handled in the DSR, i.e. leave Rx interrupts enabled,
> and to use the interim buffer. This seems to have fixed the problem - I can
> now run out of flash with no overrun errors.

That seems like a good idea. My solution was a very quick hack just to
test things out -- it just handled RX interrupts in the ISR, and set a
flag when TX interrupts occurred which were later handled in the DSR.
I ignored the status interrupts, just so I could see if the
intermediate buffer helped at all.

I didn't mask any interrupts at the end of the ISR though, and I still
got the overrun problems, which makes me wonder if something else
strange is going on...haven't had much chance to look at it today, but
hope to investigate further tomorrow.

> I am in the process of applying the equivalent changes to ser_16x5x.c, but I
> have no way of testing it. It's not quite as straightforward as the ARM
> Prime Cell UART to implement. If you'd like me to send you the completed
> file, let me know.

That would be good, thanks.

Thanks for all your help with this,

Matt