Questions on network and block device related IRQ handling

Questions on network and block device related IRQ handling

[Shane Wayne]

Hi all:
I've recently doing some research on network and block device related
IRQ handling, so I use some database and network cache application as
the testbed.
I probed the following probes below:
netdev receive
netdev transmit
netdev hard_transmit
ioblock request
ioblock end

irq_handler.entry
irq_handler,exit
softirq.entry
softirq.exit

After the profiling, I notice that, when netdev receive is triggered.
It always appears within a softirq, and the softirq has a irq_handler
in front of it. It mean the sequence is like:

irq_handler.entry
irq_handler.exit
softirq.entry
    one or several netdev receive
softirq.exit

And I found that, the irq could happened in any process, even in my
application's context, but in most condition, it was found in swapper
process.

And when it comes to netdev transmit, it seems that this action do not
relate with irq? The sequence just happens in the application context.

netdev transmit begin
        netdev hard_transmit begin
        netdev hard_transmit end
netdev transmit end

And it doesn't use irq?

So here is my question:
1, it seems that irq_handler from network device just be responsible
to notify the package arrival, and the actual handling was done by the
following soft irq?

2, the irq handling seems could happen in any process, and do not stop
current context, but when the server is not heavily loaded, the irq
prefer to happen in swapper process?

3, Or just because the swapper take more cpu time when the server was
not heavily loaded?

4, does the netdev send or ioblock request really do not need irq's assistant?

5, if this is true, how could we determine when the netdev transmit
finish it's job? is the time of netdev hard_transmit end correct?

THX~

-- 
------------------------------------------------------------
ZHANG Xiao

The Institute of Computer Software Engineering and Theory in Xi'an
Jiaotong University
Master-and-Doctoral Program Student

ACM Student Member
CCF Student Member
IEEE Computer Society Sister Society Associate Member

Xi'an Jiaotong University
Bachelor of Engineering in Computer Science and Technology

The Chinese University of Hong Kong
International Asia Study Program in CS/CE/EE/IE

Re: Questions on network and block device related IRQ handling

[William Cohen]

On 12/09/2015 10:13 PM, Shane Wayne wrote:
> Hi all:
> I've recently doing some research on network and block device related
> IRQ handling, so I use some database and network cache application as
> the testbed.
> I probed the following probes below:
> netdev receive
> netdev transmit
> netdev hard_transmit
> ioblock request
> ioblock end
> 
> irq_handler.entry
> irq_handler,exit
> softirq.entry
> softirq.exit
> 
> After the profiling, I notice that, when netdev receive is triggered.
> It always appears within a softirq, and the softirq has a irq_handler
> in front of it. It mean the sequence is like:
> 
> irq_handler.entry
> irq_handler.exit
> softirq.entry
>     one or several netdev receive
> softirq.exit
> 
> And I found that, the irq could happened in any process, even in my
> application's context, but in most condition, it was found in swapper
> process.
> 
> And when it comes to netdev transmit, it seems that this action do not
> relate with irq? The sequence just happens in the application context.
> 
> netdev transmit begin
>         netdev hard_transmit begin
>         netdev hard_transmit end
> netdev transmit end
> 
> And it doesn't use irq?
> 
> So here is my question:
> 1, it seems that irq_handler from network device just be responsible
> to notify the package arrival, and the actual handling was done by the
> following soft irq?

Yes, that is correct.  The irq handlers try to minimize the time they run.  The irq task mainly schedule some task to do the majority of the work in the softirq.

http://www.makelinux.net/ldd3/chp-10-sect-4
> 
> 2, the irq handling seems could happen in any process, and do not stop
> current context, but when the server is not heavily loaded, the irq
> prefer to happen in swapper process?

irqs can happen any time they are not masked.  There are sections of kernel code that disable them to make sure that certain operation are atomic.

> 
> 3, Or just because the swapper take more cpu time when the server was
> not heavily loaded?
> 
> 4, does the netdev send or ioblock request really do not need irq's assistant?

The netdev and ioblock works needs to be scheduled.  The irq are indicating that the device needs feeding/attentions.  The alternative would be to do polling which would be probably be less efficient.  However, the NAPI netork code does going in to a polling mode to reduce the number of interrupts
(https://en.wikipedia.org/wiki/New_API).

> 
> 5, if this is true, how could we determine when the netdev transmit
> finish it's job? is the time of netdev hard_transmit end correct?

Actually it can be hard to determine precisely when the transmit has finished sending the packet.  The packet is given to the device, but it could take some time for the device to complete sending it on the wire.  One could look and when the packet buffer is freed for an upper limit on the time it took to send.


-Will

Re: Questions on network and block device related IRQ handling

[Shane Wayne]

Thanks William, your interpretation really helps me a lot, after
viewing all the refs you mentioned apart from this I also have done
some search on NAPI, this term makes me confused:
1, in the ref:https://web.archive.org/web/20111030030517/http://www.linuxfoundation.org/collaborate/workgroups/networking/networkoverview
it says that: "This interrupt handler will be called when a frame is received"
In this quote, does the term "interrupt handler" means the irq_handler.entry?

2, If that is true, does that mean each frame receive would trigger an
individual irq_handler(It seems this is non-NAPI condition)?

3, Besides NAPI, I also found Rnx and Tnx mode of network card, does
that mean the same thing?

4,  in the ref:https://web.archive.org/web/20111030030517/http://www.linuxfoundation.org/collaborate/workgroups/networking/networkoverview
it also says that:"The same interrupt handler will be called when
transmission of a frame is finished and under other conditions.
(depends on the NIC; sometimes, the interrupt handler will be called
when there is some error)."
Does that mean netdev.hard_transmit would trigger the same irq_handler
used in netdev.receive? But I can't profile any irq even soft irq, Is
that a "NIC depended" problem?

I am using virtio network device.

2015-12-11 1:35 GMT+08:00 William Cohen <wcohen@redhat.com>:
> On 12/09/2015 10:13 PM, Shane Wayne wrote:
>> Hi all:
>> I've recently doing some research on network and block device related
>> IRQ handling, so I use some database and network cache application as
>> the testbed.
>> I probed the following probes below:
>> netdev receive
>> netdev transmit
>> netdev hard_transmit
>> ioblock request
>> ioblock end
>>
>> irq_handler.entry
>> irq_handler,exit
>> softirq.entry
>> softirq.exit
>>
>> After the profiling, I notice that, when netdev receive is triggered.
>> It always appears within a softirq, and the softirq has a irq_handler
>> in front of it. It mean the sequence is like:
>>
>> irq_handler.entry
>> irq_handler.exit
>> softirq.entry
>>     one or several netdev receive
>> softirq.exit
>>
>> And I found that, the irq could happened in any process, even in my
>> application's context, but in most condition, it was found in swapper
>> process.
>>
>> And when it comes to netdev transmit, it seems that this action do not
>> relate with irq? The sequence just happens in the application context.
>>
>> netdev transmit begin
>>         netdev hard_transmit begin
>>         netdev hard_transmit end
>> netdev transmit end
>>
>> And it doesn't use irq?
>>
>> So here is my question:
>> 1, it seems that irq_handler from network device just be responsible
>> to notify the package arrival, and the actual handling was done by the
>> following soft irq?
>
> Yes, that is correct.  The irq handlers try to minimize the time they run.  The irq task mainly schedule some task to do the majority of the work in the softirq.
>
> http://www.makelinux.net/ldd3/chp-10-sect-4
>>
>> 2, the irq handling seems could happen in any process, and do not stop
>> current context, but when the server is not heavily loaded, the irq
>> prefer to happen in swapper process?
>
> irqs can happen any time they are not masked.  There are sections of kernel code that disable them to make sure that certain operation are atomic.
>
>>
>> 3, Or just because the swapper take more cpu time when the server was
>> not heavily loaded?
>>
>> 4, does the netdev send or ioblock request really do not need irq's assistant?
>
> The netdev and ioblock works needs to be scheduled.  The irq are indicating that the device needs feeding/attentions.  The alternative would be to do polling which would be probably be less efficient.  However, the NAPI netork code does going in to a polling mode to reduce the number of interrupts
> (https://en.wikipedia.org/wiki/New_API).
>
>>
>> 5, if this is true, how could we determine when the netdev transmit
>> finish it's job? is the time of netdev hard_transmit end correct?
>
> Actually it can be hard to determine precisely when the transmit has finished sending the packet.  The packet is given to the device, but it could take some time for the device to complete sending it on the wire.  One could look and when the packet buffer is freed for an upper limit on the time it took to send.
>
>
> -Will



-- 
------------------------------------------------------------
ZHANG Xiao

The Institute of Computer Software Engineering and Theory in Xi'an
Jiaotong University
Master-and-Doctoral Program Student

ACM Student Member
CCF Student Member
IEEE Computer Society Sister Society Associate Member

Xi'an Jiaotong University
Bachelor of Engineering in Computer Science and Technology

The Chinese University of Hong Kong
International Asia Study Program in CS/CE/EE/IE

Re: Questions on network and block device related IRQ handling

[William Cohen]

On 12/11/2015 12:11 AM, Shane Wayne wrote:
> Thanks William, your interpretation really helps me a lot, after
> viewing all the refs you mentioned apart from this I also have done
> some search on NAPI, this term makes me confused:
> 1, in the ref:https://web.archive.org/web/20111030030517/http://www.linuxfoundation.org/collaborate/workgroups/networking/networkoverview
> it says that: "This interrupt handler will be called when a frame is received"
> In this quote, does the term "interrupt handler" means the irq_handler.entry?
> 
> 2, If that is true, does that mean each frame receive would trigger an
> individual irq_handler(It seems this is non-NAPI condition)?

Hi,

NAPI doesn't eliminate all interrupts. There needs to be an initial interrupt for starting fetching the packet.  It reduces the number of interrupts by polling the device to see if there is additional work that it needs to do on the device while the device is already being serviced. Skip the overhead of the interrupt handler when all it is going to do is schedule what is already running.

> 
> 3, Besides NAPI, I also found Rnx and Tnx mode of network card, does
> that mean the same thing?
> 
> 4,  in the ref:https://web.archive.org/web/20111030030517/http://www.linuxfoundation.org/collaborate/workgroups/networking/networkoverview
> it also says that:"The same interrupt handler will be called when
> transmission of a frame is finished and under other conditions.
> (depends on the NIC; sometimes, the interrupt handler will be called
> when there is some error)."
> Does that mean netdev.hard_transmit would trigger the same irq_handler
> used in netdev.receive? But I can't profile any irq even soft irq, Is
> that a "NIC depended" problem?
> 
> I am using virtio network device.

In most cases there is just one interrupt for a particular device.  The irq routine would need to fetch some status bits to figure out what precisely needs to be done for the device (for example schedule a packet send or a packet receive).


-Will
> 
> 2015-12-11 1:35 GMT+08:00 William Cohen <wcohen@redhat.com>:
>> On 12/09/2015 10:13 PM, Shane Wayne wrote:
>>> Hi all:
>>> I've recently doing some research on network and block device related
>>> IRQ handling, so I use some database and network cache application as
>>> the testbed.
>>> I probed the following probes below:
>>> netdev receive
>>> netdev transmit
>>> netdev hard_transmit
>>> ioblock request
>>> ioblock end
>>>
>>> irq_handler.entry
>>> irq_handler,exit
>>> softirq.entry
>>> softirq.exit
>>>
>>> After the profiling, I notice that, when netdev receive is triggered.
>>> It always appears within a softirq, and the softirq has a irq_handler
>>> in front of it. It mean the sequence is like:
>>>
>>> irq_handler.entry
>>> irq_handler.exit
>>> softirq.entry
>>>     one or several netdev receive
>>> softirq.exit
>>>
>>> And I found that, the irq could happened in any process, even in my
>>> application's context, but in most condition, it was found in swapper
>>> process.
>>>
>>> And when it comes to netdev transmit, it seems that this action do not
>>> relate with irq? The sequence just happens in the application context.
>>>
>>> netdev transmit begin
>>>         netdev hard_transmit begin
>>>         netdev hard_transmit end
>>> netdev transmit end
>>>
>>> And it doesn't use irq?
>>>
>>> So here is my question:
>>> 1, it seems that irq_handler from network device just be responsible
>>> to notify the package arrival, and the actual handling was done by the
>>> following soft irq?
>>
>> Yes, that is correct.  The irq handlers try to minimize the time they run.  The irq task mainly schedule some task to do the majority of the work in the softirq.
>>
>> http://www.makelinux.net/ldd3/chp-10-sect-4
>>>
>>> 2, the irq handling seems could happen in any process, and do not stop
>>> current context, but when the server is not heavily loaded, the irq
>>> prefer to happen in swapper process?
>>
>> irqs can happen any time they are not masked.  There are sections of kernel code that disable them to make sure that certain operation are atomic.
>>
>>>
>>> 3, Or just because the swapper take more cpu time when the server was
>>> not heavily loaded?
>>>
>>> 4, does the netdev send or ioblock request really do not need irq's assistant?
>>
>> The netdev and ioblock works needs to be scheduled.  The irq are indicating that the device needs feeding/attentions.  The alternative would be to do polling which would be probably be less efficient.  However, the NAPI netork code does going in to a polling mode to reduce the number of interrupts
>> (https://en.wikipedia.org/wiki/New_API).
>>
>>>
>>> 5, if this is true, how could we determine when the netdev transmit
>>> finish it's job? is the time of netdev hard_transmit end correct?
>>
>> Actually it can be hard to determine precisely when the transmit has finished sending the packet.  The packet is given to the device, but it could take some time for the device to complete sending it on the wire.  One could look and when the packet buffer is freed for an upper limit on the time it took to send.
>>
>>
>> -Will
> 
> 
>