[ECOS] nested interrupts

[ECOS] nested interrupts

[Xavier Wang]

I got confused about nested interrupts. In
http://sources.redhat.com/ecos/docs-latest/porting/hal-interrupts.html
it seems that interrupts are disabled in ISRs, but enabled in DSRs.
But Hugo's description in the same page said that higher priority
interrupts are enabled before calling ISR. Which is true for nested
interrupts? If it's the former, should I rewrite the 'hal_cpu_int_enable'
macro used in hal_interrupt_stack_call_pending_DSRs (in vector.S)
to enable only higher priority interrupts? If it's the latter, what should
I do to support nested interrupts? (I'm using TX39/JMR3904 platform,
eCos version 1.3.1)

Besides, by looking into the source code, it seems that if there is any
pending DSRs, no thread will be run. So can a high priority thread be
blocked by a low priority interrupt (DSR)? Will it introduce some
problems for a real-time application?

Thanks.

Re: [ECOS] nested interrupts

[Jesper Skov]

>>>>> "Xavier" == Xavier Wang <xavierwang@ms19.url.com.tw> writes:

Xavier> I got confused about nested interrupts. In
Xavier> http://sources.redhat.com/ecos/docs-latest/porting/hal-interrupts.html
Xavier> it seems that interrupts are disabled in ISRs, but enabled in
Xavier> DSRs.

Xavier> But Hugo's description in the same page said that
Xavier> higher priority interrupts are enabled before calling
Xavier> ISR. Which is true for nested interrupts?

Both are true, but depending on configuration. There is an option that
allows nested interrupts - when disabled, the former is valid, when
enabled, the latter is valid.

Xavier> If it's the former, should I rewrite the 'hal_cpu_int_enable'
Xavier> macro used in hal_interrupt_stack_call_pending_DSRs (in
Xavier> vector.S) to enable only higher priority interrupts? If it's

No. All interrupts are allowed when executing DSRs. DSRs are only
executin when no interrupts are pending.

Xavier> the latter, what should I do to support nested interrupts?
Xavier> (I'm using TX39/JMR3904 platform, eCos version 1.3.1)

You need code that enabled higher level interrupts depending on the
state of CYGSEM_HAL_COMMON_INTERRUPTS_ALLOW_NESTING.

The code needs to go either somewhere in vectors.S around the place
where the hal_cpu_except_enable macro is called. I don't know if a
variant/platform specific version of that macro would be a suitable
solution. None of the other MIPS platforms allow nested interrupts.

Xavier> Besides, by looking into the source code, it seems that if
Xavier> there is any pending DSRs, no thread will be run. So can a
Xavier> high priority thread be blocked by a low priority interrupt
Xavier> (DSR)?

Yes.

Xavier> Will it introduce some problems for a real-time application?

I guess that depends on how you program the DSRs/threads. But I don't
have anything intelligent to say to that, so I won't.

Jesper

Re: [ECOS] nested interrupts

[Xavier Wang]

Thanks, Jesper.


> >>>>> "Xavier" == Xavier Wang <xavierwang@ms19.url.com.tw> writes:
> 
> Xavier> I got confused about nested interrupts. In
> Xavier> http://sources.redhat.com/ecos/docs-latest/porting/hal-interrupts.html
> Xavier> it seems that interrupts are disabled in ISRs, but enabled in
> Xavier> DSRs.
> 
> Xavier> But Hugo's description in the same page said that
> Xavier> higher priority interrupts are enabled before calling
> Xavier> ISR. Which is true for nested interrupts?
> 
> Both are true, but depending on configuration. There is an option that
> allows nested interrupts - when disabled, the former is valid, when
> enabled, the latter is valid.
> 
> Xavier> If it's the former, should I rewrite the 'hal_cpu_int_enable'
> Xavier> macro used in hal_interrupt_stack_call_pending_DSRs (in
> Xavier> vector.S) to enable only higher priority interrupts? If it's
> 
> No. All interrupts are allowed when executing DSRs. DSRs are only
> executin when no interrupts are pending.

If there are some pending DSRs, it seems that these DSRs are executed 
in reverse order of interrupts/ISRs rather than in priority order. Does it
cause more unpredictability for a real-time system? Can DSRs be
prioritized? Is there a way to work around this?

Thanks.

Re: [ECOS] nested interrupts

[Xavier Wang]

Thanks, Jesper.

One more question. By looking into the source code, I still can't
figure out that why DSRs can call most kernel functions but
ISRs can't. Is it the stack problem or scheduler locking problem
or something else?

> Xavier> I got confused about nested interrupts. In
> Xavier> http://sources.redhat.com/ecos/docs-latest/porting/hal-interrupts.html
> Xavier> it seems that interrupts are disabled in ISRs, but enabled in
> Xavier> DSRs.
> 
> Xavier> But Hugo's description in the same page said that
> Xavier> higher priority interrupts are enabled before calling
> Xavier> ISR. Which is true for nested interrupts?
> 
> Both are true, but depending on configuration. There is an option that
> allows nested interrupts - when disabled, the former is valid, when
> enabled, the latter is valid.
> 
> Xavier> If it's the former, should I rewrite the 'hal_cpu_int_enable'
> Xavier> macro used in hal_interrupt_stack_call_pending_DSRs (in
> Xavier> vector.S) to enable only higher priority interrupts? If it's
> 
> No. All interrupts are allowed when executing DSRs. DSRs are only
> executin when no interrupts are pending.
> 
> Xavier> the latter, what should I do to support nested interrupts?
> Xavier> (I'm using TX39/JMR3904 platform, eCos version 1.3.1)
> 
> You need code that enabled higher level interrupts depending on the
> state of CYGSEM_HAL_COMMON_INTERRUPTS_ALLOW_NESTING.
> 
> The code needs to go either somewhere in vectors.S around the place
> where the hal_cpu_except_enable macro is called. I don't know if a
> variant/platform specific version of that macro would be a suitable
> solution. None of the other MIPS platforms allow nested interrupts.
> 
> Xavier> Besides, by looking into the source code, it seems that if
> Xavier> there is any pending DSRs, no thread will be run. So can a
> Xavier> high priority thread be blocked by a low priority interrupt
> Xavier> (DSR)?
> 
> Yes.
> 
> Xavier> Will it introduce some problems for a real-time application?
> 
> I guess that depends on how you program the DSRs/threads. But I don't
> have anything intelligent to say to that, so I won't.
> 
> Jesper

Re: [ECOS] nested interrupts

[Nick Garnett]

"Xavier Wang" <xavierwang@ms19.url.com.tw> writes:

> If there are some pending DSRs, it seems that these DSRs are executed 
> in reverse order of interrupts/ISRs rather than in priority order. Does it
> cause more unpredictability for a real-time system? Can DSRs be
> prioritized? Is there a way to work around this?
> 

DSRs are unprioritized. The priority supplied when creating an
interruopt is intended to prioritize the ISR in the hardware if that
is possible, or select its place in the chain when chained interrupts
are enabled.

There would be no advantage in prioritizing DSRs. All pending DSRs are
always run whenever any DSR can be run, and all DSRs are higher
priority than any threads and lower priority than any interrupts. The
order they get run in at this point is irrelevant, since the system
will not proceed until all of them have finished. Prioritized
execution should happen in threads, that's what they are there for.

-- 
Nick Garnett, eCos Kernel Architect
Red Hat, Cambridge, UK

Re: [ECOS] nested interrupts

[Nick Garnett]

"Xavier Wang" <xavierwang@ms19.url.com.tw> writes:

> Thanks, Jesper.
> 
> One more question. By looking into the source code, I still can't
> figure out that why DSRs can call most kernel functions but
> ISRs can't. Is it the stack problem or scheduler locking problem
> or something else?

The kernel does not block interrupts during critical sections, so an
ISR can occur at any point, even while the scheduler is manipulating
the run queues. If the ISR then tries to manipulate the same data
structures, by calling a kernel function, then they will be
corrupted. Hence the use of DSRs to defer these calls until the data
structures are in a consistent state.


-- 
Nick Garnett, eCos Kernel Architect
Red Hat, Cambridge, UK

Re: [ECOS] nested interrupts

[Sergei Slobodov]

> 
> Xavier> Besides, by looking into the source code, it seems that if
> Xavier> there is any pending DSRs, no thread will be run. So can a
> Xavier> high priority thread be blocked by a low priority interrupt
> Xavier> (DSR)?
> 
> Yes.
> 
> Xavier> Will it introduce some problems for a real-time application?


> 
> Interrupts are always higher priority than tasks (threads), and DSRs 
> are all at the same priority, higher than any thread and lower than 
> any interrupt. When designing a real-time system, it is a good idea to 
> include the total time of all possible interrupts (and their DSRs) 
> when calculating thread response time

> 
> As I was explained here, a main purpose of a DSR is to avoid 
> scheduler's ready list manipulation from an interrupt. DSR mechanism 
> serializes access to the scheduler (to suspend or resume a thread, or 
> to send a message), thus completely avoiding disabling interrupts when 
> posting. Quite neat. There is a price to pay, however, since DSR 
> mechanism nearly doubles the time between posting interrupt and thread 
> resumption. Bearing this in mind, the only thing DSR absolutely has to 
> do is to notify the scheduler. All the rest ought to be done either in 
> an ISR or in a thread.
> 
Regards,
Sergei Slobodov

Re: [ECOS] nested interrupts

[Xavier Wang]

> 
> As I was explained here, a main purpose of a DSR is to avoid 
> scheduler's ready list manipulation from an interrupt. DSR mechanism 
> serializes access to the scheduler (to suspend or resume a thread, or 
> to send a message), thus completely avoiding disabling interrupts when 
> posting. Quite neat. There is a price to pay, however, since DSR 
> mechanism nearly doubles the time between posting interrupt and thread 
> resumption. Bearing this in mind, the only thing DSR absolutely has to 
> do is to notify the scheduler. All the rest ought to be done either in 
> an ISR or in a thread.
> 
> Regards,
> Sergei Slobodov

Thanks for your explanation. But as I knew, ISRs should be as short as
possible to reduce the time of masking interrupts and other non-critical
stuff can be processed in DSRs. So I don't understand what the following
means:

    "Bearing this in mind, the only thing DSR absolutely has to do is to
    notify the scheduler. All the rest ought to be done either in an ISR or
    in a thread."

Re: [ECOS] nested interrupts

[Xavier Wang]

> > 
> > One more question. By looking into the source code, I still can't
> > figure out that why DSRs can call most kernel functions but
> > ISRs can't. Is it the stack problem or scheduler locking problem
> > or something else?
> 
> The kernel does not block interrupts during critical sections, so an
> ISR can occur at any point, even while the scheduler is manipulating
> the run queues. If the ISR then tries to manipulate the same data
> structures, by calling a kernel function, then they will be
> corrupted. Hence the use of DSRs to defer these calls until the data
> structures are in a consistent state.
> 
> 
> -- 
> Nick Garnett, eCos Kernel Architect
> Red Hat, Cambridge, UK

Thanks for your explanation. But I still got confused.
As described by Gary Thomas in
http://sources.redhat.com/ecos/docs-latest/porting/hal-interrupts.html

> The way it is supposed to work is like this (pseudo-code to abstract
> machine details):
>
> VSR:
>   mask interrupts (if necessary)
>   save interrupt context on thread stack 
>   lock scheduler
>   switch to interrupt stack (if defined)
>   call ISR
>   switch back to thread stack
>   call interrupt_end
>   restore interrupt context from thread stack
>   exit VSR
> 
> Note: interrupt_end called with interrupts disabled, scheduler locked
> 
> interrupt_end:
>   possibly schedule a DSR, based on paramters from VSR
>   drop scheduler lock  
>   if scheduling enabled, run pending DSRs (actually just part of scheduling)
>   note: interrupts are still disabled when entered from VSR
> 
> hal_interrupt_stack_call_pending_DSRs:
>   [disable interrupts (may be moot)]
>   switch to interrupt stack
>   enable interrupts
>   run DSRs
>   disable interrupts
>   switch back to thread stack
>   restore interrupts

Which step above does indeed make the kernel state safe to be called by DSRs?

For example, an interrupt occurred while a thread is running in scheduler's critical
section. It's not safe to manipulate scheduler's data in ISR obviously, but it seems
also not safe to access scheduler's data in DSR since the interrupted thread still
doesn't finish its job to make the data consistent. Am I wrong here?

Thanks.

Re: [ECOS] nested interrupts

[Jonathan Larmour]

Xavier Wang wrote:
[snip] 
> Which step above does indeed make the kernel state safe to be called by DSRs?
> 
> For example, an interrupt occurred while a thread is running in scheduler's critical
> section. It's not safe to manipulate scheduler's data in ISR obviously, but it seems
> also not safe to access scheduler's data in DSR since the interrupted thread still
> doesn't finish its job to make the data consistent. Am I wrong here?

DSRs are not called if the scheduler was already locked *before* the
interrupt. So the running thread just gets restored in the return from the
interrupt. It is the process of unlocking the scheduler, and specifically
the scheduler lock transitioning from the 1 to 0 state that causes DSRs to
run.

The sequence of events you cited is a simplification. For detail at this
level (after all, why do you need to know how - just know it is safe), Use
The Source Luke. DSRs are called from Cyg_Scheduler::unlock_inner() in
kernel/VERSION/src/sched/sched.cxx.

Jifl
-- 
Red Hat, Rustat House, Clifton Road, Cambridge, UK. Tel: +44 (1223) 271062
Un cheval, pas du glue. Pas du cheval, beaucoup du glue. || Opinions==mine

Re: [ECOS] nested interrupts

[Jonathan Larmour]

Xavier Wang wrote:
> But as I knew, ISRs should be as short as
> possible to reduce the time of masking interrupts and other non-critical
> stuff can be processed in DSRs. So I don't understand what the following
> means:
> 
>     "Bearing this in mind, the only thing DSR absolutely has to do is to
>     notify the scheduler. All the rest ought to be done either in an ISR or
>     in a thread."

The DSR should just wake or signal a thread to let it know that the
interrupt happened, that's all. You can't do that from an ISR. All other
processing should take place in a thread - both ISRs and DSRs should be
short and simple like that, otherwise you will lose determinism.

Jifl
-- 
Red Hat, Rustat House, Clifton Road, Cambridge, UK. Tel: +44 (1223) 271062
Un cheval, pas du glue. Pas du cheval, beaucoup du glue. || Opinions==mine

Re: [ECOS] Nested Interrupts.

[Jonathan Larmour]

Sandeep Rikhi wrote:
> 
>  In case an interrupt has occured and we are in the ISR part of it.
> Suppose at the same time a high priority interrupt ( say a
> reset) occurs... what would be the situation then? Do we leave the current
> ISR and switch over to new interrupt ( high priority ). If it is so, what
> was the effect of disabling the interrupts?

Interrupts are completely disabled while ISRs are running. ISRs are meant
to be very short with any substantial work deferred to a DSR, or better
still by the DSR then waking a thread. Typically, if more work is being
done in a DSR, then the ISR will mask the interrupt. This then allows other
interrupts to be processed.

With many systems, masking the interrupt will actually mask that interrupt
*and* all lower priority ones, so that you only receive higher priority
ones (until the interrupt is unmasked in the DSR). It depends on the
interrupt controller being used, although you could also write
HAL_INTERRUPT_MASK() for your particular platform to do this "manually" -
this isn't done by default, as this requires tracking what interrupts are
available in the system and their priority.

Jifl
-- 
Red Hat, 35 Cambridge Place, Cambridge, UK. CB2 1NS  Tel: +44 (1223) 728762
"Plan to be spontaneous tomorrow."  ||  These opinions are all my own fault

[ECOS] Nested Interrupts.

[Sandeep Rikhi]

Sir,
 In case an interrupt has occured and we are in the ISR part of it.
Suppose at the same time a high priority interrupt ( say a
reset) occurs... what would be the situation then? Do we leave the current
ISR and switch over to new interrupt ( high priority ). If it is so, what
was the effect of disabling the interrupts?
*********************************************************
        Sandeep Rikhi
        GET-GCPE Group.
        Switching Division B1,
        C-DOT, B'lore
        Ph: (080) 2263399 ext:205/217,2250009,2282119   
*********************************************************