From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 5369 invoked by alias); 4 Mar 2014 16:17:04 -0000 Mailing-List: contact ecos-discuss-help@ecos.sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: ecos-discuss-owner@ecos.sourceware.org Received: (qmail 5355 invoked by uid 89); 4 Mar 2014 16:17:03 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-2.6 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_LOW autolearn=ham version=3.3.2 X-HELO: mail-ee0-f42.google.com Received: from mail-ee0-f42.google.com (HELO mail-ee0-f42.google.com) (74.125.83.42) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with (AES128-SHA encrypted) ESMTPS; Tue, 04 Mar 2014 16:17:00 +0000 Received: by mail-ee0-f42.google.com with SMTP id d17so4940872eek.1 for ; Tue, 04 Mar 2014 08:16:57 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-type :content-transfer-encoding; bh=gMUfOVFqw5R6ddSpOVryNoxe6Cm8HZpPbxg/7dlIrDY=; b=I7mxHEE2k7+t02dVVqngLHGpOXYXljDMmA2tCoj7ySED0MJGJp8Q99lexx34/iI9Y3 loDhHDIAua1TYz832Eml70HOTQlEoA/11tG4bduRiDGhJRHLhUrxXBjtbaJ5lWVQYr32 GkijIaN70Qr6rPFaPirNYVThle6cSTxP/pgi2h19wkzYZpsRC6SgYHnLkTZVu0vuHSw9 p8A58ZLHrG+iwX87+26eJNzTNJ2WJqNF/9a2ivc1HhmWABxeM/AM1o286zVIMK/FT3bL qqpdO+8KzUMfyRyb/4rHpk6Z8Wkhqz0anG4TmVQz+APpIm7tNZY8yNduipNVr+KyTSWT TjSA== X-Gm-Message-State: ALoCoQloszmidZGRwrvwuJjZ5Oe5oTD6XT9UhrlwDUQMDqN0HYXiQ1K85XI9sKzUedx5fP5zqGM+ X-Received: by 10.14.7.65 with SMTP id 41mr334521eeo.100.1393949816909; Tue, 04 Mar 2014 08:16:56 -0800 (PST) Received: from [137.133.125.254] ([77.40.227.130]) by mx.google.com with ESMTPSA id 48sm64622795eee.2.2014.03.04.08.16.55 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 04 Mar 2014 08:16:56 -0800 (PST) Message-ID: <5315FC76.8030002@stmi.com> Date: Tue, 04 Mar 2014 16:17:00 -0000 From: christophe User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.3.0 MIME-Version: 1.0 To: Michael Jones CC: =?ISO-8859-1?Q?Lambrecht_J=FCrgen?= , ecos discuss References: <496B24D9-62B6-48F2-BD53-1F6B9ABE2083@linear.com> <5315F31D.9060007@stmi.com> <55AB601E-FC02-4510-B3A7-C1970FA2E187@linear.com> In-Reply-To: <55AB601E-FC02-4510-B3A7-C1970FA2E187@linear.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-IsSubscribed: yes Subject: Re: [ECOS] Scheduler startup question X-SW-Source: 2014-03/txt/msg00003.txt.bz2 Michael, I am not sure what you mean by adding code in interrupt_end to take the lock. The locking mechanism is present for SMP target, no change required: externC void interrupt_end( cyg_uint32 isr_ret, Cyg_Interrupt *intr, HAL_SavedRegisters *regs ) { // CYG_REPORT_FUNCTION(); #ifdef CYGPKG_KERNEL_SMP_SUPPORT Cyg_Scheduler::lock(); #endif The macro for incrementing the lock in SMP looks at the current owner of the lock and spin when required. I found the kernel instrumentation option very useful for debugging deadlocks. I was using CodeConfidence plugin in Eclipse to analyze the trace which makes it pretty efficient debugging. Christophe On 3/4/2014 4:58 PM, Michael Jones wrote: > Christophe, > > When I first got SMP to work I added some code in interrupt_end to take the lock, but I moved it back to Vectors.S because I was trying to reduce changes to the kernel. Functionally, the only difference is getting the lock before the ISR is executed or not. > > My bigger concern is how the lock is taken. When I increase the lock count, the core doing so (core 0) may not be the holder of the lock, which leads to assertions. And if it spins while taking the lock, it deadlocks. I have not traced down the deadlock, but I think the problem is in the scheduler, where some secondary CPU is waiting. > > My current solution is to use a trylock in Vectors.S and living with the fact that when it fails, it will take another real time clock interrupt to try again. So interrupt_end is not guaranteed to called on each interrupt. This keeps things simple. All interrupts go to core 0 except inter cpu interrupts. Some latency is added because taking the lock is not guaranteed. > > Other ways to handle this is to send interrupts to all cores, use inter core interrupts, etc, in an effort to guarantee a lock is incremented by the core that holds the lock. > > I was not able to figure our how i386 handled this. Does anyone know how the i386 SMP incremented the lock if the core that got the interrupt did not hold the lock? > > Mike > > > On Mar 4, 2014, at 8:37 AM, christophe wrote: > >> Hi Michael, >> >> I might remember wrong but I think in case of SMP target, the lock is not taken in Vector.S but directly after entering interrupt_end. Of course this is spinlock based so it might delay posting/scheduling of the DSR. >> >> Christophe >> >> On 3/2/2014 9:19 PM, Michael Jones wrote: >>> Jurgen, >>> >>> I think I fully understand how the scheduler locking works during interrupt now. Vectors.S takes the lock, and interrupt_end clears it. However, the normal technique of incrementing the lock count does not work with SMP. The problem is that another CPU may have the lock. Incrementing anyway leads to assertions. Attempting to take the lock with the spinlock can lead to deadlocks or an unresponsive network application. >>> >>> So I changed things so that in Vectors.S, during an interrupt, an attempt at locking is made. This means trying to take a spinlock that might fail. If the lock is taken, interrupt_end is called. If the lock fails, interrupt_end is not called. >>> >>> This means that a DSR may not be posted on that interrupt. This can cause some latency based on the real time clock interrupt rate, or time until a thread switch. However, it is stable and assertion free. Also, a HAL could implement a timeout on the try spinlock which might reduce latency. >>> >>> To support the try and testing if the lock was taken, I had to add some functions to the kernel. The following wiki page has been updated to reflect the kernel changes. >>> >>> https://sourceforge.net/p/ecosfreescale/wiki/SMP%20Kernel/ >>> >>> Anyone with SMP knowledge might want to take a look. There may be better solutions to some of these problems. But at least for now, the IMX6 SMP HAL seems stable and I can run IO intensive Lua scripts over telnet reliably, even when the client aborts. >>> >>> The client abort means telnet has to kill a thread. This was quite a challenge. Telnet is creating a separate heap for Lua so it can kill the thread and reclaim memory. The remaining problem is closing file handles. I still get some assertions when a handle is sometimes killed by a thread that does not own it. I don't think that can be solved without adding some new functions dedicated to clean up of file handles by an outside thread. >>> >>> Mike >>> >>> >>> >>> On Feb 26, 2014, at 11:40 PM, Lambrecht Jürgen wrote: >>> >>>> As far as I know the scheduler is started after cyg_user_start(), used by your application to initialize everything. Do you use cyg_user_start? >>>> >>>> >>>> Verzonden vanaf Samsung Mobile >>>> >>>> >>>> >>>> -------- Oorspronkelijk bericht -------- >>>> Van: Michael Jones >>>> Datum: >>>> Aan: ecos discuss >>>> Onderwerp: [ECOS] Scheduler startup question >>>> >>>> >>>> I have a question about proper scheduler locking startup behavior. >>>> >>>> The context is I am cleaning up my iMX6 HAL and attempting to make things work without a couple of kernel hacks I added to make it work. >>>> >>>> The question has to do with sched_lock. By default this has a value of 1, so during startup the scheduler is locked. >>>> >>>> When there is an interrupt, sched_lock is incremented in Vectors.S, and decremented in interrupt_end. >>>> >>>> However, I am getting an assert in sync.h which is part of the BSD stack. The assert is because it expects the lock to be zero. >>>> >>>> The question is, during the startup process, how does the lock get set to zero after initialization? Is it supposed to stay 1 while hardware is initialized and through all the constructors, etc? Is it cleared by the scheduler somehow? Is the HAL supposed to zero it at some point during startup? >>>> >>>> My HAL is part of the ARM hal, so if this is device specific, it is the ARM HAL I am working with. >>>> >>>> Mike >>>> -- >>>> Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos >>>> and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss >>>> >>>> >>>> -- >>>> Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos >>>> and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss >>>> >> >> -- >> Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos >> and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss >> -- Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss