* Questions on network and block device related IRQ handling @ 2015-12-10 3:14 Shane Wayne 2015-12-10 17:35 ` William Cohen 0 siblings, 1 reply; 4+ messages in thread From: Shane Wayne @ 2015-12-10 3:14 UTC (permalink / raw) To: systemtap Hi all: I've recently doing some research on network and block device related IRQ handling, so I use some database and network cache application as the testbed. I probed the following probes below: netdev receive netdev transmit netdev hard_transmit ioblock request ioblock end irq_handler.entry irq_handler,exit softirq.entry softirq.exit After the profiling, I notice that, when netdev receive is triggered. It always appears within a softirq, and the softirq has a irq_handler in front of it. It mean the sequence is like: irq_handler.entry irq_handler.exit softirq.entry one or several netdev receive softirq.exit And I found that, the irq could happened in any process, even in my application's context, but in most condition, it was found in swapper process. And when it comes to netdev transmit, it seems that this action do not relate with irq? The sequence just happens in the application context. netdev transmit begin netdev hard_transmit begin netdev hard_transmit end netdev transmit end And it doesn't use irq? So here is my question: 1, it seems that irq_handler from network device just be responsible to notify the package arrival, and the actual handling was done by the following soft irq? 2, the irq handling seems could happen in any process, and do not stop current context, but when the server is not heavily loaded, the irq prefer to happen in swapper process? 3, Or just because the swapper take more cpu time when the server was not heavily loaded? 4, does the netdev send or ioblock request really do not need irq's assistant? 5, if this is true, how could we determine when the netdev transmit finish it's job? is the time of netdev hard_transmit end correct? THX~ -- ------------------------------------------------------------ ZHANG Xiao The Institute of Computer Software Engineering and Theory in Xi'an Jiaotong University Master-and-Doctoral Program Student ACM Student Member CCF Student Member IEEE Computer Society Sister Society Associate Member Xi'an Jiaotong University Bachelor of Engineering in Computer Science and Technology The Chinese University of Hong Kong International Asia Study Program in CS/CE/EE/IE ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Questions on network and block device related IRQ handling 2015-12-10 3:14 Questions on network and block device related IRQ handling Shane Wayne @ 2015-12-10 17:35 ` William Cohen 2015-12-11 5:11 ` Shane Wayne 0 siblings, 1 reply; 4+ messages in thread From: William Cohen @ 2015-12-10 17:35 UTC (permalink / raw) To: Shane Wayne, systemtap On 12/09/2015 10:13 PM, Shane Wayne wrote: > Hi all: > I've recently doing some research on network and block device related > IRQ handling, so I use some database and network cache application as > the testbed. > I probed the following probes below: > netdev receive > netdev transmit > netdev hard_transmit > ioblock request > ioblock end > > irq_handler.entry > irq_handler,exit > softirq.entry > softirq.exit > > After the profiling, I notice that, when netdev receive is triggered. > It always appears within a softirq, and the softirq has a irq_handler > in front of it. It mean the sequence is like: > > irq_handler.entry > irq_handler.exit > softirq.entry > one or several netdev receive > softirq.exit > > And I found that, the irq could happened in any process, even in my > application's context, but in most condition, it was found in swapper > process. > > And when it comes to netdev transmit, it seems that this action do not > relate with irq? The sequence just happens in the application context. > > netdev transmit begin > netdev hard_transmit begin > netdev hard_transmit end > netdev transmit end > > And it doesn't use irq? > > So here is my question: > 1, it seems that irq_handler from network device just be responsible > to notify the package arrival, and the actual handling was done by the > following soft irq? Yes, that is correct. The irq handlers try to minimize the time they run. The irq task mainly schedule some task to do the majority of the work in the softirq. http://www.makelinux.net/ldd3/chp-10-sect-4 > > 2, the irq handling seems could happen in any process, and do not stop > current context, but when the server is not heavily loaded, the irq > prefer to happen in swapper process? irqs can happen any time they are not masked. There are sections of kernel code that disable them to make sure that certain operation are atomic. > > 3, Or just because the swapper take more cpu time when the server was > not heavily loaded? > > 4, does the netdev send or ioblock request really do not need irq's assistant? The netdev and ioblock works needs to be scheduled. The irq are indicating that the device needs feeding/attentions. The alternative would be to do polling which would be probably be less efficient. However, the NAPI netork code does going in to a polling mode to reduce the number of interrupts (https://en.wikipedia.org/wiki/New_API). > > 5, if this is true, how could we determine when the netdev transmit > finish it's job? is the time of netdev hard_transmit end correct? Actually it can be hard to determine precisely when the transmit has finished sending the packet. The packet is given to the device, but it could take some time for the device to complete sending it on the wire. One could look and when the packet buffer is freed for an upper limit on the time it took to send. -Will ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Questions on network and block device related IRQ handling 2015-12-10 17:35 ` William Cohen @ 2015-12-11 5:11 ` Shane Wayne 2015-12-11 16:34 ` William Cohen 0 siblings, 1 reply; 4+ messages in thread From: Shane Wayne @ 2015-12-11 5:11 UTC (permalink / raw) To: William Cohen; +Cc: systemtap Thanks William, your interpretation really helps me a lot, after viewing all the refs you mentioned apart from this I also have done some search on NAPI, this term makes me confused: 1, in the ref:https://web.archive.org/web/20111030030517/http://www.linuxfoundation.org/collaborate/workgroups/networking/networkoverview it says that: "This interrupt handler will be called when a frame is received" In this quote, does the term "interrupt handler" means the irq_handler.entry? 2, If that is true, does that mean each frame receive would trigger an individual irq_handler(It seems this is non-NAPI condition)? 3, Besides NAPI, I also found Rnx and Tnx mode of network card, does that mean the same thing? 4, in the ref:https://web.archive.org/web/20111030030517/http://www.linuxfoundation.org/collaborate/workgroups/networking/networkoverview it also says that:"The same interrupt handler will be called when transmission of a frame is finished and under other conditions. (depends on the NIC; sometimes, the interrupt handler will be called when there is some error)." Does that mean netdev.hard_transmit would trigger the same irq_handler used in netdev.receive? But I can't profile any irq even soft irq, Is that a "NIC depended" problem? I am using virtio network device. 2015-12-11 1:35 GMT+08:00 William Cohen <wcohen@redhat.com>: > On 12/09/2015 10:13 PM, Shane Wayne wrote: >> Hi all: >> I've recently doing some research on network and block device related >> IRQ handling, so I use some database and network cache application as >> the testbed. >> I probed the following probes below: >> netdev receive >> netdev transmit >> netdev hard_transmit >> ioblock request >> ioblock end >> >> irq_handler.entry >> irq_handler,exit >> softirq.entry >> softirq.exit >> >> After the profiling, I notice that, when netdev receive is triggered. >> It always appears within a softirq, and the softirq has a irq_handler >> in front of it. It mean the sequence is like: >> >> irq_handler.entry >> irq_handler.exit >> softirq.entry >> one or several netdev receive >> softirq.exit >> >> And I found that, the irq could happened in any process, even in my >> application's context, but in most condition, it was found in swapper >> process. >> >> And when it comes to netdev transmit, it seems that this action do not >> relate with irq? The sequence just happens in the application context. >> >> netdev transmit begin >> netdev hard_transmit begin >> netdev hard_transmit end >> netdev transmit end >> >> And it doesn't use irq? >> >> So here is my question: >> 1, it seems that irq_handler from network device just be responsible >> to notify the package arrival, and the actual handling was done by the >> following soft irq? > > Yes, that is correct. The irq handlers try to minimize the time they run. The irq task mainly schedule some task to do the majority of the work in the softirq. > > http://www.makelinux.net/ldd3/chp-10-sect-4 >> >> 2, the irq handling seems could happen in any process, and do not stop >> current context, but when the server is not heavily loaded, the irq >> prefer to happen in swapper process? > > irqs can happen any time they are not masked. There are sections of kernel code that disable them to make sure that certain operation are atomic. > >> >> 3, Or just because the swapper take more cpu time when the server was >> not heavily loaded? >> >> 4, does the netdev send or ioblock request really do not need irq's assistant? > > The netdev and ioblock works needs to be scheduled. The irq are indicating that the device needs feeding/attentions. The alternative would be to do polling which would be probably be less efficient. However, the NAPI netork code does going in to a polling mode to reduce the number of interrupts > (https://en.wikipedia.org/wiki/New_API). > >> >> 5, if this is true, how could we determine when the netdev transmit >> finish it's job? is the time of netdev hard_transmit end correct? > > Actually it can be hard to determine precisely when the transmit has finished sending the packet. The packet is given to the device, but it could take some time for the device to complete sending it on the wire. One could look and when the packet buffer is freed for an upper limit on the time it took to send. > > > -Will -- ------------------------------------------------------------ ZHANG Xiao The Institute of Computer Software Engineering and Theory in Xi'an Jiaotong University Master-and-Doctoral Program Student ACM Student Member CCF Student Member IEEE Computer Society Sister Society Associate Member Xi'an Jiaotong University Bachelor of Engineering in Computer Science and Technology The Chinese University of Hong Kong International Asia Study Program in CS/CE/EE/IE ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Questions on network and block device related IRQ handling 2015-12-11 5:11 ` Shane Wayne @ 2015-12-11 16:34 ` William Cohen 0 siblings, 0 replies; 4+ messages in thread From: William Cohen @ 2015-12-11 16:34 UTC (permalink / raw) To: Shane Wayne; +Cc: systemtap On 12/11/2015 12:11 AM, Shane Wayne wrote: > Thanks William, your interpretation really helps me a lot, after > viewing all the refs you mentioned apart from this I also have done > some search on NAPI, this term makes me confused: > 1, in the ref:https://web.archive.org/web/20111030030517/http://www.linuxfoundation.org/collaborate/workgroups/networking/networkoverview > it says that: "This interrupt handler will be called when a frame is received" > In this quote, does the term "interrupt handler" means the irq_handler.entry? > > 2, If that is true, does that mean each frame receive would trigger an > individual irq_handler(It seems this is non-NAPI condition)? Hi, NAPI doesn't eliminate all interrupts. There needs to be an initial interrupt for starting fetching the packet. It reduces the number of interrupts by polling the device to see if there is additional work that it needs to do on the device while the device is already being serviced. Skip the overhead of the interrupt handler when all it is going to do is schedule what is already running. > > 3, Besides NAPI, I also found Rnx and Tnx mode of network card, does > that mean the same thing? > > 4, in the ref:https://web.archive.org/web/20111030030517/http://www.linuxfoundation.org/collaborate/workgroups/networking/networkoverview > it also says that:"The same interrupt handler will be called when > transmission of a frame is finished and under other conditions. > (depends on the NIC; sometimes, the interrupt handler will be called > when there is some error)." > Does that mean netdev.hard_transmit would trigger the same irq_handler > used in netdev.receive? But I can't profile any irq even soft irq, Is > that a "NIC depended" problem? > > I am using virtio network device. In most cases there is just one interrupt for a particular device. The irq routine would need to fetch some status bits to figure out what precisely needs to be done for the device (for example schedule a packet send or a packet receive). -Will > > 2015-12-11 1:35 GMT+08:00 William Cohen <wcohen@redhat.com>: >> On 12/09/2015 10:13 PM, Shane Wayne wrote: >>> Hi all: >>> I've recently doing some research on network and block device related >>> IRQ handling, so I use some database and network cache application as >>> the testbed. >>> I probed the following probes below: >>> netdev receive >>> netdev transmit >>> netdev hard_transmit >>> ioblock request >>> ioblock end >>> >>> irq_handler.entry >>> irq_handler,exit >>> softirq.entry >>> softirq.exit >>> >>> After the profiling, I notice that, when netdev receive is triggered. >>> It always appears within a softirq, and the softirq has a irq_handler >>> in front of it. It mean the sequence is like: >>> >>> irq_handler.entry >>> irq_handler.exit >>> softirq.entry >>> one or several netdev receive >>> softirq.exit >>> >>> And I found that, the irq could happened in any process, even in my >>> application's context, but in most condition, it was found in swapper >>> process. >>> >>> And when it comes to netdev transmit, it seems that this action do not >>> relate with irq? The sequence just happens in the application context. >>> >>> netdev transmit begin >>> netdev hard_transmit begin >>> netdev hard_transmit end >>> netdev transmit end >>> >>> And it doesn't use irq? >>> >>> So here is my question: >>> 1, it seems that irq_handler from network device just be responsible >>> to notify the package arrival, and the actual handling was done by the >>> following soft irq? >> >> Yes, that is correct. The irq handlers try to minimize the time they run. The irq task mainly schedule some task to do the majority of the work in the softirq. >> >> http://www.makelinux.net/ldd3/chp-10-sect-4 >>> >>> 2, the irq handling seems could happen in any process, and do not stop >>> current context, but when the server is not heavily loaded, the irq >>> prefer to happen in swapper process? >> >> irqs can happen any time they are not masked. There are sections of kernel code that disable them to make sure that certain operation are atomic. >> >>> >>> 3, Or just because the swapper take more cpu time when the server was >>> not heavily loaded? >>> >>> 4, does the netdev send or ioblock request really do not need irq's assistant? >> >> The netdev and ioblock works needs to be scheduled. The irq are indicating that the device needs feeding/attentions. The alternative would be to do polling which would be probably be less efficient. However, the NAPI netork code does going in to a polling mode to reduce the number of interrupts >> (https://en.wikipedia.org/wiki/New_API). >> >>> >>> 5, if this is true, how could we determine when the netdev transmit >>> finish it's job? is the time of netdev hard_transmit end correct? >> >> Actually it can be hard to determine precisely when the transmit has finished sending the packet. The packet is given to the device, but it could take some time for the device to complete sending it on the wire. One could look and when the packet buffer is freed for an upper limit on the time it took to send. >> >> >> -Will > > > ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2015-12-11 16:34 UTC | newest] Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2015-12-10 3:14 Questions on network and block device related IRQ handling Shane Wayne 2015-12-10 17:35 ` William Cohen 2015-12-11 5:11 ` Shane Wayne 2015-12-11 16:34 ` William Cohen
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).