DMA capable Ethernet driver... (zero-copy)?

DMA capable Ethernet driver... (zero-copy)?

[Lars Povlsen]

Hello!

On working on a.k.a. Ethernet for eCos I was initially triggered by
seeing a stack-allocated Ethernet header (redboot/stand-alone) to
investigate other Ethernet drivers... I was sad to see only copying
send() implementations until I saw the following comment in the eCos
docs:

"Note: In future, this function may be extended so that the data need
not be copied by having the function return a "disposition" code (done,
send pending, etc). At this point, you should move the data to some
"safe" location before returning."

Is this really the state of affairs? Copy no matter that the hardware
may have all kinds of fancy scatter/gather frame DMA?

I guess I may DMA the frame directly out - spinning on completion. Ouch!

Suggestions welcome...

Sincerely,

---Lars

Re: DMA capable Ethernet driver... (zero-copy)?

[Nick Garnett]

"Lars Povlsen" <lpovlsen@vitesse.com> writes:

> Hello!
> 
> On working on a.k.a. Ethernet for eCos I was initially triggered by
> seeing a stack-allocated Ethernet header (redboot/stand-alone) to
> investigate other Ethernet drivers... I was sad to see only copying
> send() implementations until I saw the following comment in the eCos
> docs:
> 
> "Note: In future, this function may be extended so that the data need
> not be copied by having the function return a "disposition" code (done,
> send pending, etc). At this point, you should move the data to some
> "safe" location before returning."
> 
> Is this really the state of affairs? Copy no matter that the hardware
> may have all kinds of fancy scatter/gather frame DMA?


Direct DMA from the buffers should work since they should not be
released until the driver calls the tx_done() function. A key is
passed back and forth so that the calls can be properly matched. As an
example the Intel 82544 driver does direct DMA transmission.

I have to admit that I'm not entirely sure what that note means, since
the functionality it describes is already mostly present in the
send()/tx_done() interface. Perhaps Gary, who probably wrote it
several years ago, could explain what he meant.

> 
> I guess I may DMA the frame directly out - spinning on completion. Ouch!

No need to spin or even wait in the send() function. You just return
once the transmission has been started. Eventually an interrupt will
indicate completion, and cause the deliver function to be called, in
which you call tx_done().


-- 
Nick Garnett                                 eCos Kernel Architect
http://www.ecoscentric.com            The eCos and RedBoot experts

Re: DMA capable Ethernet driver... (zero-copy)?

[Lars Povlsen]

Nick Garnett wrote:
>
> "Lars Povlsen" <lpovlsen@vitesse.com> writes:
>
> > Hello!
> >
> > On working on a.k.a. Ethernet for eCos I was initially triggered by
> > seeing a stack-allocated Ethernet header (redboot/stand-alone) to
> > investigate other Ethernet drivers... I was sad to see only copying
> > send() implementations until I saw the following comment in the eCos
> > docs:
> >
> > "Note: In future, this function may be extended so that the data need
> > not be copied by having the function return a "disposition" code (done,
> > send pending, etc). At this point, you should move the data to some
> > "safe" location before returning."
> >
> > Is this really the state of affairs? Copy no matter that the hardware
> > may have all kinds of fancy scatter/gather frame DMA?
>
>
> Direct DMA from the buffers should work since they should not be
> released until the driver calls the tx_done() function. A key is
> passed back and forth so that the calls can be properly matched. As an
> example the Intel 82544 driver does direct DMA transmission.
>
Thanks for the pointers - I'll have a look.
>
>
> I have to admit that I'm not entirely sure what that note means, since
> the functionality it describes is already mostly present in the
> send()/tx_done() interface. Perhaps Gary, who probably wrote it
> several years ago, could explain what he meant.
>
> >
> > I guess I may DMA the frame directly out - spinning on completion. Ouch!
>
> No need to spin or even wait in the send() function. You just return
> once the transmission has been started. Eventually an interrupt will
> indicate completion, and cause the deliver function to be called, in
> which you call tx_done().
>
See redboot/current/src/net/enet.c:

void
__enet_send(pktbuf_t *pkt, enet_addr_t *dest, int eth_type)
{
=>  eth_header_t eth_hdr;

    // Set up ethernet header
    memcpy(&eth_hdr.destination, dest, sizeof(enet_addr_t));
    memcpy(&eth_hdr.source, __local_enet_addr, sizeof(enet_addr_t));
    eth_hdr.type = htons(eth_type);

    eth_drv_write((char *)&eth_hdr, (char *)pkt->buf, pkt->pkt_bytes);
#ifdef ENET_STATS
    ++num_transmitted;
#endif
}

Unless the fragments (notably the first - the ethernet header) are 
squirelled away, it will be thrashed randomly by subsequent stack use.

So is this a bug?

Off course, other network stacks may have better characteristics, but at 
least redboot will exhibit random data thrashing if I use straight DMA. 
The occurrence will be dependent on how fast the DMA is started (other 
system/network) activity - which may be non-existing in the redboot - 
which could explain why this apparently is not a problem...?

---Lars

> --
> Nick Garnett                                 eCos Kernel Architect
> http://www.ecoscentric.com            The eCos and RedBoot experts
>

Re: DMA capable Ethernet driver... (zero-copy)?

[Nick Garnett]

Lars Povlsen <lpovlsen@vitesse.com> writes:

> Unless the fragments (notably the first - the ethernet header) are
> squirelled away, it will be thrashed randomly by subsequent stack use.
> 
> So is this a bug?
> 
> Off course, other network stacks may have better characteristics, but
> at least redboot will exhibit random data thrashing if I use straight
> DMA. The occurrence will be dependent on how fast the DMA is started
> (other system/network) activity - which may be non-existing in the
> redboot -
> which could explain why this apparently is not a problem...?

RedBoot doesn't use interrupts, so eth_drv_write() is synchronous. The
implementation of eth_drv_write() used by the RedBoot stack polls
until the packet has been sent. The send() routine is called, then
the driver's poll() routine is called repeatedly until the tx_done()
routine is called, then eth_drv_write() returns.

The ethernet drivers are only asynchronous when interrupts are
available, and only the BSD and LwIP stacks are able to take advantage
of that since they run with the kernel enabled.


-- 
Nick Garnett                                 eCos Kernel Architect
http://www.ecoscentric.com            The eCos and RedBoot experts

Re: DMA capable Ethernet driver... (zero-copy)?

[Lars Povlsen]

Nick Garnett wrote:
> Lars Povlsen <lpovlsen@vitesse.com> writes:
>
>   
>> Unless the fragments (notably the first - the ethernet header) are
>> squirelled away, it will be thrashed randomly by subsequent stack use.
>>
>> So is this a bug?
>>
>> Off course, other network stacks may have better characteristics, but
>> at least redboot will exhibit random data thrashing if I use straight
>> DMA. The occurrence will be dependent on how fast the DMA is started
>> (other system/network) activity - which may be non-existing in the
>> redboot -
>> which could explain why this apparently is not a problem...?
>>     
>
> RedBoot doesn't use interrupts, so eth_drv_write() is synchronous. The
> implementation of eth_drv_write() used by the RedBoot stack polls
> until the packet has been sent. The send() routine is called, then
> the driver's poll() routine is called repeatedly until the tx_done()
> routine is called, then eth_drv_write() returns.
>
> The ethernet drivers are only asynchronous when interrupts are
> available, and only the BSD and LwIP stacks are able to take advantage
> of that since they run with the kernel enabled.
>
>   
Ahhh, I did'nt realize it was eth_drv_write() itself poll()'ing - so no 
stack unwind until done. That explains everything.

So I will pretend like I never read the "At this point, you should move 
the data to some “safe” location before returning" comment... False 
alarm, it seems!

Thanks again,

---Lars

Re: DMA capable Ethernet driver... (zero-copy)?

[Gary Thomas]

Nick Garnett wrote:
> "Lars Povlsen" <lpovlsen@vitesse.com> writes:
> 
>> Hello!
>>
>> On working on a.k.a. Ethernet for eCos I was initially triggered by
>> seeing a stack-allocated Ethernet header (redboot/stand-alone) to
>> investigate other Ethernet drivers... I was sad to see only copying
>> send() implementations until I saw the following comment in the eCos
>> docs:
>>
>> "Note: In future, this function may be extended so that the data need
>> not be copied by having the function return a "disposition" code (done,
>> send pending, etc). At this point, you should move the data to some
>> "safe" location before returning."
>>
>> Is this really the state of affairs? Copy no matter that the hardware
>> may have all kinds of fancy scatter/gather frame DMA?
> 
> 
> Direct DMA from the buffers should work since they should not be
> released until the driver calls the tx_done() function. A key is
> passed back and forth so that the calls can be properly matched. As an
> example the Intel 82544 driver does direct DMA transmission.
> 
> I have to admit that I'm not entirely sure what that note means, since
> the functionality it describes is already mostly present in the
> send()/tx_done() interface. Perhaps Gary, who probably wrote it
> several years ago, could explain what he meant.

The comment was mostly an indication of what the hardware driver
*could* do.  As Nick said, the framework is in place to let the
driver do whatever it wants with the data.  If you want to use
zero-copy and your hardware can handle it, the 'key' with callback
mechanism allows for the data buffers to be held (not reused) by
the stacks until the transmission is complete.  Achieving zero
copy on input is a bit trickier and the framework doesn't quite
cut it - there would need to be a way to create incoming buffers
before hand that the driver could be instructed to use and the
receive callback would then just indicate that the data had arrived.
At this time though, the middle layer (eth_drv.c and friends) does
not support this.


-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------