Apparent kernel bug with GDB on ppc405

Apparent kernel bug with GDB on ppc405

[Matt Mackall]

I'm trying to debug a trivial statically-linked hello world program on
a Xilinx PPC 405 and I'm seeing the following behavior:

With direct gdb on target, I can set a breakpoint at main, run, and
the breakpoint is triggered.

With gdbserver and gdb with "target remote localhost:1234", the above
still works.

With gdb on target redirected to a PC and tunneled back
to the target, everything still works.

With gdb on a PC, execution continues past the breakpoint. Comparing
the gdb protocol streams here and and on the previous (working) case
are identical up to the point of hitting the breakpoint (which never
happens in the latter case).

Raising the load on the PC to 4 and running gdb under nice -n 19 makes
things work again. So this begins to look like a kernel cache or
timing bug rather than a problem with the PC tool. It appears that the
breakpoint written to the executable at continue time is not visible
to the CPU at execute time.

My first suspicion was a dcache/icache coherency issue in
copy_to_user_page, so I added flush_dcache_icache_page(page) here to
no avail. On closer inspection, it looks like both icache and dcache
are already being flushed by flush_icache_user_range().

Adding printk(".") (or any printk) in this function here fixes things
(serial console at 115k), while printk("") and udelay(100) do not.
Which still suggests an icache bug..?

Any suggestions?

-- 
 "Love the dolphins," she advised him. "Write by W.A.S.T.E.." 

Re: Apparent kernel bug with GDB on ppc405

[Grant Likely]

On 10/24/07, Matt Mackall <mpm@selenic.com> wrote:
> I'm trying to debug a trivial statically-linked hello world program on
> a Xilinx PPC 405 and I'm seeing the following behavior:
>
<snip>
>
> Any suggestions?

http://thread.gmane.org/gmane.linux.ports.ppc.embedded/11202

I was fighting with a similar problem almost 2 years ago.  Looks like
it might be related.  At some point the problem seemed to go away and
I determined what the root cause was.  :-(

I haven't been using gdb lately, so I don't know if it's the same
problem.  Nobody I had talked to had seen the issue on other 405
platforms.  It could very well be something virtex-specific.

That's not much help, but maybe it will give you some clues.

Cheers,
g.

-- 
Grant Likely, B.Sc., P.Eng.
Secret Lab Technologies Ltd.
grant.likely@secretlab.ca
(403) 399-0195

Re: Apparent kernel bug with GDB on ppc405

[Matt Mackall]

On Wed, Oct 24, 2007 at 02:28:14PM -0600, Grant Likely wrote:
> On 10/24/07, Matt Mackall <mpm@selenic.com> wrote:
> > I'm trying to debug a trivial statically-linked hello world program on
> > a Xilinx PPC 405 and I'm seeing the following behavior:
> >
> <snip>
> >
> > Any suggestions?
> 
> http://thread.gmane.org/gmane.linux.ports.ppc.embedded/11202
> 
> I was fighting with a similar problem almost 2 years ago.  Looks like
> it might be related.  At some point the problem seemed to go away and
> I determined what the root cause was.  :-(
> 
> I haven't been using gdb lately, so I don't know if it's the same
> problem.  Nobody I had talked to had seen the issue on other 405
> platforms.  It could very well be something virtex-specific.

Could be the same problem, but I'm seeing only your symptom 3 so far.

I've tried throwing some larger hammers at the problem. Flushing all
of the dcache and icache (flush_dcache_all and
flush_instruction_cache) isn't helping. But printk(".") does!

-- 
Mathematics is the supreme nostalgia of our time.

Re: Apparent kernel bug with GDB on ppc405

[Grant Likely]

On 10/24/07, Matt Mackall <mpm@selenic.com> wrote:
> On Wed, Oct 24, 2007 at 02:28:14PM -0600, Grant Likely wrote:
> > On 10/24/07, Matt Mackall <mpm@selenic.com> wrote:
> > > I'm trying to debug a trivial statically-linked hello world program on
> > > a Xilinx PPC 405 and I'm seeing the following behavior:
> > >
> > <snip>
> > >
> > > Any suggestions?
> >
> > http://thread.gmane.org/gmane.linux.ports.ppc.embedded/11202
> >
> > I was fighting with a similar problem almost 2 years ago.  Looks like
> > it might be related.  At some point the problem seemed to go away and
> > I determined what the root cause was.  :-(
> >
> > I haven't been using gdb lately, so I don't know if it's the same
> > problem.  Nobody I had talked to had seen the issue on other 405
> > platforms.  It could very well be something virtex-specific.
>
> Could be the same problem, but I'm seeing only your symptom 3 so far.
>
> I've tried throwing some larger hammers at the problem. Flushing all
> of the dcache and icache (flush_dcache_all and
> flush_instruction_cache) isn't helping. But printk(".") does!

It's really true; printk *is* the most valuable tool kernel hackers
have for debugging.

g.

-- 
Grant Likely, B.Sc., P.Eng.
Secret Lab Technologies Ltd.
grant.likely@secretlab.ca
(403) 399-0195

Re: Apparent kernel bug with GDB on ppc405

[Matt Mackall]

On Wed, Oct 24, 2007 at 03:42:16PM -0500, Matt Mackall wrote:
> On Wed, Oct 24, 2007 at 02:28:14PM -0600, Grant Likely wrote:
> > On 10/24/07, Matt Mackall <mpm@selenic.com> wrote:
> > > I'm trying to debug a trivial statically-linked hello world program on
> > > a Xilinx PPC 405 and I'm seeing the following behavior:
> > >
> > <snip>
> > >
> > > Any suggestions?
> > 
> > http://thread.gmane.org/gmane.linux.ports.ppc.embedded/11202
> > 
> > I was fighting with a similar problem almost 2 years ago.  Looks like
> > it might be related.  At some point the problem seemed to go away and
> > I determined what the root cause was.  :-(
> > 
> > I haven't been using gdb lately, so I don't know if it's the same
> > problem.  Nobody I had talked to had seen the issue on other 405
> > platforms.  It could very well be something virtex-specific.
> 
> Could be the same problem, but I'm seeing only your symptom 3 so far.
> 
> I've tried throwing some larger hammers at the problem. Flushing all
> of the dcache and icache (flush_dcache_all and
> flush_instruction_cache) isn't helping. But printk(".") does!

Well there was one remaining cache - the TLB. This patch seems to make
things work, but don't ask me why:

--- include/asm-ppc/cacheflush.h        (revision 10439)
+++ include/asm-ppc/cacheflush.h        (working copy)
@@ -11,6 +11,7 @@
 #define _PPC_CACHEFLUSH_H

 #include <linux/mm.h>
+#include <asm/tlbflush.h>

 /*
  * No cache flushing is required when address mappings are
@@ -35,10 +36,23 @@
 extern void flush_icache_user_range(struct vm_area_struct *vma,
                struct page *page, unsigned long addr, int len);

 #define copy_to_user_page(vma, page, vaddr, dst, src, len) \
 do { memcpy(dst, src, len); \
      flush_icache_user_range(vma, page, vaddr, len); \
+     _tlbia(); \
 } while (0)

-- 
Mathematics is the supreme nostalgia of our time.

Re: Apparent kernel bug with GDB on ppc405

[Grant Likely]

On 10/24/07, Matt Mackall <mpm@selenic.com> wrote:
> On Wed, Oct 24, 2007 at 03:42:16PM -0500, Matt Mackall wrote:
> > On Wed, Oct 24, 2007 at 02:28:14PM -0600, Grant Likely wrote:
> > > On 10/24/07, Matt Mackall <mpm@selenic.com> wrote:
> > > > I'm trying to debug a trivial statically-linked hello world program on
> > > > a Xilinx PPC 405 and I'm seeing the following behavior:
> > > >
> > > <snip>
> > > >
> > > > Any suggestions?
> > >
> > > http://thread.gmane.org/gmane.linux.ports.ppc.embedded/11202
> > >
> > > I was fighting with a similar problem almost 2 years ago.  Looks like
> > > it might be related.  At some point the problem seemed to go away and
> > > I determined what the root cause was.  :-(
> > >
> > > I haven't been using gdb lately, so I don't know if it's the same
> > > problem.  Nobody I had talked to had seen the issue on other 405
> > > platforms.  It could very well be something virtex-specific.
> >
> > Could be the same problem, but I'm seeing only your symptom 3 so far.
> >
> > I've tried throwing some larger hammers at the problem. Flushing all
> > of the dcache and icache (flush_dcache_all and
> > flush_instruction_cache) isn't helping. But printk(".") does!
>
> Well there was one remaining cache - the TLB. This patch seems to make
> things work, but don't ask me why:
>
> --- include/asm-ppc/cacheflush.h        (revision 10439)
> +++ include/asm-ppc/cacheflush.h        (working copy)
> @@ -11,6 +11,7 @@
>  #define _PPC_CACHEFLUSH_H
>
>  #include <linux/mm.h>
> +#include <asm/tlbflush.h>
>
>  /*
>   * No cache flushing is required when address mappings are
> @@ -35,10 +36,23 @@
>  extern void flush_icache_user_range(struct vm_area_struct *vma,
>                 struct page *page, unsigned long addr, int len);
>
>  #define copy_to_user_page(vma, page, vaddr, dst, src, len) \
>  do { memcpy(dst, src, len); \
>       flush_icache_user_range(vma, page, vaddr, len); \
> +     _tlbia(); \
>  } while (0)

Hmmm; thinking out loud here...

- so tlbia invalidates all TLB entries
- When gdb inserts a breakpoint the .text pages are marked as read
only, so the kernel does a copy on write so that gdb can modify the
instruction.  The kernel also updates the page tables so that the test
process now uses the new page.
- This means that there are now 2 pages for that one section of
executable code; the original and the one with the breakpoint.
- However, the program is still in memory, and there is probably
already a TLB entry pointing to the original page for that range of
addresses.

Could it be that the kernel page tables are getting updated to the new
page; but active set of TLB entries is not getting updated?

If so, then printk(".") probably solves the problem simply because it
touches enough pages in its execution path that the old TLB entry gets
overwritten?  There are only 64 TLB entries afterall.

Thoughts?

g.

-- 
Grant Likely, B.Sc., P.Eng.
Secret Lab Technologies Ltd.
grant.likely@secretlab.ca
(403) 399-0195

Re: Apparent kernel bug with GDB on ppc405

[Matt Mackall]

On Wed, Oct 24, 2007 at 04:27:52PM -0600, Grant Likely wrote:
> On 10/24/07, Matt Mackall <mpm@selenic.com> wrote:
> > On Wed, Oct 24, 2007 at 03:42:16PM -0500, Matt Mackall wrote:
> > > On Wed, Oct 24, 2007 at 02:28:14PM -0600, Grant Likely wrote:
> > > > On 10/24/07, Matt Mackall <mpm@selenic.com> wrote:
> > > > > I'm trying to debug a trivial statically-linked hello world program on
> > > > > a Xilinx PPC 405 and I'm seeing the following behavior:
> > > > >
> > > > <snip>
> > > > >
> > > > > Any suggestions?
> > > >
> > > > http://thread.gmane.org/gmane.linux.ports.ppc.embedded/11202
> > > >
> > > > I was fighting with a similar problem almost 2 years ago.  Looks like
> > > > it might be related.  At some point the problem seemed to go away and
> > > > I determined what the root cause was.  :-(
> > > >
> > > > I haven't been using gdb lately, so I don't know if it's the same
> > > > problem.  Nobody I had talked to had seen the issue on other 405
> > > > platforms.  It could very well be something virtex-specific.
> > >
> > > Could be the same problem, but I'm seeing only your symptom 3 so far.
> > >
> > > I've tried throwing some larger hammers at the problem. Flushing all
> > > of the dcache and icache (flush_dcache_all and
> > > flush_instruction_cache) isn't helping. But printk(".") does!
> >
> > Well there was one remaining cache - the TLB. This patch seems to make
> > things work, but don't ask me why:
> >
> > --- include/asm-ppc/cacheflush.h        (revision 10439)
> > +++ include/asm-ppc/cacheflush.h        (working copy)
> > @@ -11,6 +11,7 @@
> >  #define _PPC_CACHEFLUSH_H
> >
> >  #include <linux/mm.h>
> > +#include <asm/tlbflush.h>
> >
> >  /*
> >   * No cache flushing is required when address mappings are
> > @@ -35,10 +36,23 @@
> >  extern void flush_icache_user_range(struct vm_area_struct *vma,
> >                 struct page *page, unsigned long addr, int len);
> >
> >  #define copy_to_user_page(vma, page, vaddr, dst, src, len) \
> >  do { memcpy(dst, src, len); \
> >       flush_icache_user_range(vma, page, vaddr, len); \
> > +     _tlbia(); \
> >  } while (0)
> 
> Hmmm; thinking out loud here...
> 
> - so tlbia invalidates all TLB entries
> - When gdb inserts a breakpoint the .text pages are marked as read
> only, so the kernel does a copy on write so that gdb can modify the
> instruction.  The kernel also updates the page tables so that the test
> process now uses the new page.
> - This means that there are now 2 pages for that one section of
> executable code; the original and the one with the breakpoint.
> - However, the program is still in memory, and there is probably
> already a TLB entry pointing to the original page for that range of
> addresses.
> 
> Could it be that the kernel page tables are getting updated to the new
> page; but active set of TLB entries is not getting updated?
> 
> If so, then printk(".") probably solves the problem simply because it
> touches enough pages in its execution path that the old TLB entry gets
> overwritten?  There are only 64 TLB entries afterall.
> 
> Thoughts?

Not completely implausible, but a) why isn't this seen on basically
every machine with software TLB? b) why does -local- GDB, which is
presumably doing much less work than gdbserver + network stack, not fail?

-- 
Mathematics is the supreme nostalgia of our time.

Re: Apparent kernel bug with GDB on ppc405

[Grant Likely]

On 10/24/07, Matt Mackall <mpm@selenic.com> wrote:
> On Wed, Oct 24, 2007 at 04:27:52PM -0600, Grant Likely wrote:
> > On 10/24/07, Matt Mackall <mpm@selenic.com> wrote:
> > > On Wed, Oct 24, 2007 at 03:42:16PM -0500, Matt Mackall wrote:
> > > > On Wed, Oct 24, 2007 at 02:28:14PM -0600, Grant Likely wrote:
> > > > > On 10/24/07, Matt Mackall <mpm@selenic.com> wrote:
> > > > > > I'm trying to debug a trivial statically-linked hello world program on
> > > > > > a Xilinx PPC 405 and I'm seeing the following behavior:
> > > > > >
> > > > > <snip>
> > > > > >
> > > > > > Any suggestions?
> > > > >
> > > > > http://thread.gmane.org/gmane.linux.ports.ppc.embedded/11202
> > > > >
> > > > > I was fighting with a similar problem almost 2 years ago.  Looks like
> > > > > it might be related.  At some point the problem seemed to go away and
> > > > > I determined what the root cause was.  :-(
> > > > >
> > > > > I haven't been using gdb lately, so I don't know if it's the same
> > > > > problem.  Nobody I had talked to had seen the issue on other 405
> > > > > platforms.  It could very well be something virtex-specific.
> > > >
> > > > Could be the same problem, but I'm seeing only your symptom 3 so far.
> > > >
> > > > I've tried throwing some larger hammers at the problem. Flushing all
> > > > of the dcache and icache (flush_dcache_all and
> > > > flush_instruction_cache) isn't helping. But printk(".") does!
> > >
> > > Well there was one remaining cache - the TLB. This patch seems to make
> > > things work, but don't ask me why:
> > >
> > > --- include/asm-ppc/cacheflush.h        (revision 10439)
> > > +++ include/asm-ppc/cacheflush.h        (working copy)
> > > @@ -11,6 +11,7 @@
> > >  #define _PPC_CACHEFLUSH_H
> > >
> > >  #include <linux/mm.h>
> > > +#include <asm/tlbflush.h>
> > >
> > >  /*
> > >   * No cache flushing is required when address mappings are
> > > @@ -35,10 +36,23 @@
> > >  extern void flush_icache_user_range(struct vm_area_struct *vma,
> > >                 struct page *page, unsigned long addr, int len);
> > >
> > >  #define copy_to_user_page(vma, page, vaddr, dst, src, len) \
> > >  do { memcpy(dst, src, len); \
> > >       flush_icache_user_range(vma, page, vaddr, len); \
> > > +     _tlbia(); \
> > >  } while (0)
> >
> > Hmmm; thinking out loud here...
> >
> > - so tlbia invalidates all TLB entries
> > - When gdb inserts a breakpoint the .text pages are marked as read
> > only, so the kernel does a copy on write so that gdb can modify the
> > instruction.  The kernel also updates the page tables so that the test
> > process now uses the new page.
> > - This means that there are now 2 pages for that one section of
> > executable code; the original and the one with the breakpoint.
> > - However, the program is still in memory, and there is probably
> > already a TLB entry pointing to the original page for that range of
> > addresses.
> >
> > Could it be that the kernel page tables are getting updated to the new
> > page; but active set of TLB entries is not getting updated?
> >
> > If so, then printk(".") probably solves the problem simply because it
> > touches enough pages in its execution path that the old TLB entry gets
> > overwritten?  There are only 64 TLB entries afterall.
> >
> > Thoughts?
>
> Not completely implausible, but a) why isn't this seen on basically
> every machine with software TLB? b) why does -local- GDB, which is
> presumably doing much less work than gdbserver + network stack, not fail?

a) I don't know.... very odd.

b) gdb is big.  It probably touches far more pages (via library calls)
than gdbserver.  The network stack is also big, but it's probably more
localized too.

Niceing down the host also makes sense because if the PC is being slow
then the target may go off and run other things while between setting
the breakpoint and getting the 'go' command.

Can you grab a snapshot of the TLB before and after setting the breakpoint?

g.

-- 
Grant Likely, B.Sc., P.Eng.
Secret Lab Technologies Ltd.
grant.likely@secretlab.ca
(403) 399-0195

Re: Apparent kernel bug with GDB on ppc405

[Grant Likely]

On 10/24/07, Grant Likely <grant.likely@secretlab.ca> wrote:
> > Not completely implausible, but a) why isn't this seen on basically
> > every machine with software TLB? b) why does -local- GDB, which is
> > presumably doing much less work than gdbserver + network stack, not fail?
>
> a) I don't know.... very odd.
>
> b) gdb is big.  It probably touches far more pages (via library calls)
> than gdbserver.  The network stack is also big, but it's probably more
> localized too.
>
> Niceing down the host also makes sense because if the PC is being slow
> then the target may go off and run other things while between setting
> the breakpoint and getting the 'go' command.
>
> Can you grab a snapshot of the TLB before and after setting the breakpoint?

Or; probably more relevant, before and after the page copy?

g.

-- 
Grant Likely, B.Sc., P.Eng.
Secret Lab Technologies Ltd.
grant.likely@secretlab.ca
(403) 399-0195

Re: Apparent kernel bug with GDB on ppc405

[Benjamin Herrenschmidt]

On Wed, 2007-10-24 at 16:40 -0600, Grant Likely wrote:
> On 10/24/07, Grant Likely <grant.likely@secretlab.ca> wrote:
> > > Not completely implausible, but a) why isn't this seen on basically
> > > every machine with software TLB? b) why does -local- GDB, which is
> > > presumably doing much less work than gdbserver + network stack, not fail?
> >
> > a) I don't know.... very odd.
> >
> > b) gdb is big.  It probably touches far more pages (via library calls)
> > than gdbserver.  The network stack is also big, but it's probably more
> > localized too.
> >
> > Niceing down the host also makes sense because if the PC is being slow
> > then the target may go off and run other things while between setting
> > the breakpoint and getting the 'go' command.
> >
> > Can you grab a snapshot of the TLB before and after setting the breakpoint?
> 
> Or; probably more relevant, before and after the page copy?

COW does ptep_clear_flush before setting the new TLB which does a
flush_tlb_page() so that should work.

Ben.

Re: Apparent kernel bug with GDB on ppc405

[Daniel Jacobowitz]

On Wed, Oct 24, 2007 at 05:32:50PM -0500, Matt Mackall wrote:
> Not completely implausible, but a) why isn't this seen on basically
> every machine with software TLB? b) why does -local- GDB, which is
> presumably doing much less work than gdbserver + network stack, not fail?

You said it yourself.  Local gdb does more work -> blows through more
TLB entries.

I can't answer you about the other half, but I'm pretty sure TLB
invalidation is already supposed to be happening... somewhere.

-- 
Daniel Jacobowitz
CodeSourcery

Re: Apparent kernel bug with GDB on ppc405

[Benjamin Herrenschmidt]

On Wed, 2007-10-24 at 18:41 -0400, Daniel Jacobowitz wrote:
> On Wed, Oct 24, 2007 at 05:32:50PM -0500, Matt Mackall wrote:
> > Not completely implausible, but a) why isn't this seen on basically
> > every machine with software TLB? b) why does -local- GDB, which is
> > presumably doing much less work than gdbserver + network stack, not fail?
> 
> You said it yourself.  Local gdb does more work -> blows through more
> TLB entries.
> 
> I can't answer you about the other half, but I'm pretty sure TLB
> invalidation is already supposed to be happening... somewhere.

Yes. do_wp_page() -> ptep_clear_flush() -> flush_tlb_page()

Ben.

Re: Apparent kernel bug with GDB on ppc405

[Josh Boyer]

On Fri, 26 Oct 2007 11:51:22 +1000
Benjamin Herrenschmidt <benh@kernel.crashing.org> wrote:

> 
> On Wed, 2007-10-24 at 18:41 -0400, Daniel Jacobowitz wrote:
> > On Wed, Oct 24, 2007 at 05:32:50PM -0500, Matt Mackall wrote:
> > > Not completely implausible, but a) why isn't this seen on basically
> > > every machine with software TLB? b) why does -local- GDB, which is
> > > presumably doing much less work than gdbserver + network stack, not fail?
> > 
> > You said it yourself.  Local gdb does more work -> blows through more
> > TLB entries.
> > 
> > I can't answer you about the other half, but I'm pretty sure TLB
> > invalidation is already supposed to be happening... somewhere.
> 
> Yes. do_wp_page() -> ptep_clear_flush() -> flush_tlb_page()

Aren't there cases in do_wp_page that don't call ptep_clear_flush?
Seems anonymous pages, and possibly shared writeable pages skip that
step if reuse is true.

josh

Re: Apparent kernel bug with GDB on ppc405

[Benjamin Herrenschmidt]

On Fri, 2007-10-26 at 15:41 -0500, Josh Boyer wrote:
> On Fri, 26 Oct 2007 11:51:22 +1000
> Benjamin Herrenschmidt <benh@kernel.crashing.org> wrote:
> 
> > 
> > On Wed, 2007-10-24 at 18:41 -0400, Daniel Jacobowitz wrote:
> > > On Wed, Oct 24, 2007 at 05:32:50PM -0500, Matt Mackall wrote:
> > > > Not completely implausible, but a) why isn't this seen on basically
> > > > every machine with software TLB? b) why does -local- GDB, which is
> > > > presumably doing much less work than gdbserver + network stack, not fail?
> > > 
> > > You said it yourself.  Local gdb does more work -> blows through more
> > > TLB entries.
> > > 
> > > I can't answer you about the other half, but I'm pretty sure TLB
> > > invalidation is already supposed to be happening... somewhere.
> > 
> > Yes. do_wp_page() -> ptep_clear_flush() -> flush_tlb_page()
> 
> Aren't there cases in do_wp_page that don't call ptep_clear_flush?
> Seems anonymous pages, and possibly shared writeable pages skip that
> step if reuse is true.

Nah, if that was broken, everybody would be in bad shape. I think I know
what's up, see my other email.

Cheers,
Ben.

Re: Apparent kernel bug with GDB on ppc405

[Josh Boyer]

On Sat, 2007-10-27 at 11:36 +1000, Benjamin Herrenschmidt wrote:
> On Fri, 2007-10-26 at 15:41 -0500, Josh Boyer wrote:
> > On Fri, 26 Oct 2007 11:51:22 +1000
> > Benjamin Herrenschmidt <benh@kernel.crashing.org> wrote:
> > 
> > > 
> > > On Wed, 2007-10-24 at 18:41 -0400, Daniel Jacobowitz wrote:
> > > > On Wed, Oct 24, 2007 at 05:32:50PM -0500, Matt Mackall wrote:
> > > > > Not completely implausible, but a) why isn't this seen on basically
> > > > > every machine with software TLB? b) why does -local- GDB, which is
> > > > > presumably doing much less work than gdbserver + network stack, not fail?
> > > > 
> > > > You said it yourself.  Local gdb does more work -> blows through more
> > > > TLB entries.
> > > > 
> > > > I can't answer you about the other half, but I'm pretty sure TLB
> > > > invalidation is already supposed to be happening... somewhere.
> > > 
> > > Yes. do_wp_page() -> ptep_clear_flush() -> flush_tlb_page()
> > 
> > Aren't there cases in do_wp_page that don't call ptep_clear_flush?
> > Seems anonymous pages, and possibly shared writeable pages skip that
> > step if reuse is true.
> 
> Nah, if that was broken, everybody would be in bad shape. I think I know
> what's up, see my other email.

Yeah, I figured that out after I sent this anyway.  In those cases it
should likely get flushed from ptep_set_access_flags.

josh

Re: Apparent kernel bug with GDB on ppc405

[David Daney]

Matt Mackall wrote:
> I'm trying to debug a trivial statically-linked hello world program on
> a Xilinx PPC 405 and I'm seeing the following behavior:
> 
> With direct gdb on target, I can set a breakpoint at main, run, and
> the breakpoint is triggered.
> 
> With gdbserver and gdb with "target remote localhost:1234", the above
> still works.
> 
> With gdb on target redirected to a PC and tunneled back
> to the target, everything still works.
> 
> With gdb on a PC, execution continues past the breakpoint. Comparing
> the gdb protocol streams here and and on the previous (working) case
> are identical up to the point of hitting the breakpoint (which never
> happens in the latter case).
> 
> Raising the load on the PC to 4 and running gdb under nice -n 19 makes
> things work again. So this begins to look like a kernel cache or
> timing bug rather than a problem with the PC tool. It appears that the
> breakpoint written to the executable at continue time is not visible
> to the CPU at execute time.
> 
> My first suspicion was a dcache/icache coherency issue in
> copy_to_user_page, so I added flush_dcache_icache_page(page) here to
> no avail. On closer inspection, it looks like both icache and dcache
> are already being flushed by flush_icache_user_range().
> 

First of all I have never used a similar configuration so this may be 
totally off base.  But...

If the icache is virtually indexed, then I think there are only two ways 
to invalidate it.  The first is from the context of the debugged process 
where the page is mapped at the location the target program will see it. 
   If you try to invalidate from the context of the debugger, the page 
will most likely not be mapped at the virtual address of the target 
program so you might have to invalidate the *entire* icache.

David Daney

Re: Apparent kernel bug with GDB on ppc405

[Benjamin Herrenschmidt]

On Wed, 2007-10-24 at 13:34 -0700, David Daney wrote:
> First of all I have never used a similar configuration so this may be 
> totally off base.  But...
> 
> If the icache is virtually indexed, then I think there are only two
> ways 
> to invalidate it.  The first is from the context of the debugged
> process 
> where the page is mapped at the location the target program will see
> it. 
>    If you try to invalidate from the context of the debugger, the
> page 
> will most likely not be mapped at the virtual address of the target 
> program so you might have to invalidate the *entire* icache.

Yup, sounds likely. We don't handle that virtual icache well at all in
linux. Things tend to work by mere luck. I have various ideas to fix
that but haven't actually implemented them.

Ben.

Re: Apparent kernel bug with GDB on ppc405

[Benjamin Herrenschmidt]

On Wed, 2007-10-24 at 14:46 -0500, Matt Mackall wrote:
> 
> My first suspicion was a dcache/icache coherency issue in
> copy_to_user_page, so I added flush_dcache_icache_page(page) here to
> no avail. On closer inspection, it looks like both icache and dcache
> are already being flushed by flush_icache_user_range().
> 
> Adding printk(".") (or any printk) in this function here fixes things
> (serial console at 115k), while printk("") and udelay(100) do not.
> Which still suggests an icache bug..?
> 
> Any suggestions?

I think you're hitting a known bug of 44x support. Those CPUs have a
crazy virtually tagged icache and the kernel doesn't deal with it at all
(pretty much...). We just are lucky things generally work :-)

That means among other things that flush_icache_* will not work because
they kmap pages and use that mapping. The only way to flush icache user
pages with 44x is to actually flush with the user virtual address
(meaning you have to be in the current context, and you probably need to
have a TLB entry there... yuck)... or just blow the whole icache away.

Try sticking an iccci in there and let me know if that helps.

Ben

Re: Apparent kernel bug with GDB on ppc405

[Grant Likely]

On 10/25/07, Benjamin Herrenschmidt <benh@kernel.crashing.org> wrote:
>
> On Wed, 2007-10-24 at 14:46 -0500, Matt Mackall wrote:
> >
> > My first suspicion was a dcache/icache coherency issue in
> > copy_to_user_page, so I added flush_dcache_icache_page(page) here to
> > no avail. On closer inspection, it looks like both icache and dcache
> > are already being flushed by flush_icache_user_range().
> >
> > Adding printk(".") (or any printk) in this function here fixes things
> > (serial console at 115k), while printk("") and udelay(100) do not.
> > Which still suggests an icache bug..?
> >
> > Any suggestions?
>
> I think you're hitting a known bug of 44x support. Those CPUs have a
> crazy virtually tagged icache and the kernel doesn't deal with it at all
> (pretty much...). We just are lucky things generally work :-)
>
> That means among other things that flush_icache_* will not work because
> they kmap pages and use that mapping. The only way to flush icache user
> pages with 44x is to actually flush with the user virtual address
> (meaning you have to be in the current context, and you probably need to
> have a TLB entry there... yuck)... or just blow the whole icache away.

This is actually 405.  Does that have the same issue?

Cheers,
g.

-- 
Grant Likely, B.Sc., P.Eng.
Secret Lab Technologies Ltd.
grant.likely@secretlab.ca
(403) 399-0195

Re: Apparent kernel bug with GDB on ppc405

[Benjamin Herrenschmidt]

> This is actually 405.  Does that have the same issue?

hrm... I don't remember :-) There -is- something fishy about the icache
on 405 but I don't remember for sure. Try sticking an iccci in there and
tell us if that helps.

Ben.

Re: Apparent kernel bug with GDB on ppc405

[Matt Mackall]

On Fri, Oct 26, 2007 at 01:23:25PM +1000, Benjamin Herrenschmidt wrote:
> 
> > This is actually 405.  Does that have the same issue?
> 
> hrm... I don't remember :-) There -is- something fishy about the icache
> on 405 but I don't remember for sure. Try sticking an iccci in there and
> tell us if that helps.

I did. I stuck an iccci -and- a dccci loop in. I threw in the tlbia
when I ran out of other ideas. And the tlbia + existing
flush_icache_range appears to be sufficient (tested on a non-trivial
app).

According to my docs, the 405fx's icache is virtually
indexed/physically mapped, while the dcache is phys/phys.

-- 
Mathematics is the supreme nostalgia of our time.

Re: Apparent kernel bug with GDB on ppc405

[Benjamin Herrenschmidt]

On Fri, 2007-10-26 at 09:41 -0500, Matt Mackall wrote:
> On Fri, Oct 26, 2007 at 01:23:25PM +1000, Benjamin Herrenschmidt
> wrote:
> > 
> > > This is actually 405.  Does that have the same issue?
> > 
> > hrm... I don't remember :-) There -is- something fishy about the
> icache
> > on 405 but I don't remember for sure. Try sticking an iccci in there
> and
> > tell us if that helps.
> 
> I did. I stuck an iccci -and- a dccci loop in. I threw in the tlbia
> when I ran out of other ideas. And the tlbia + existing
> flush_icache_range appears to be sufficient (tested on a non-trivial
> app).
> 
> According to my docs, the 405fx's icache is virtually
> indexed/physically mapped, while the dcache is phys/phys.

I think there is a bug in the 40x/44x support at this stage, it only
doing invalidations for the current PID. That means that a process
trying to invalidate TLB entries of another address space will fail to
do so ... oops.

That can be fixed by adding a pid argument to to _tlbie, and using it in
the assembly. In the 40x case, probably by switching SPRN_PID to that
and then back, in the 44x case, by using that value instead of SPRN_PID
for the MMUCR register.

It's part of a patche I have around taking dirt that rework signficantly
the 44x (though not the 40x yet) TLB handling, cleaning things and
fixing bugs (such as this one, I remember it now). I need to spend a bit
of time to clean it up, remove some experimental stuff from it, and
submit it.

I'll try to do that early next week, and maybe come up with a fix for
that specific bug today or tomorrow.

Ben.

Re: [PATCH/RFC] powerpc: Pass PID argument to _tlbie (WAS: Apparent kernel bug with GDB on ppc405)

[Matt Mackall]

[adding back gdb list]

On Tue, Oct 30, 2007 at 07:35:06AM +1100, Benjamin Herrenschmidt wrote:
> 
> > Did a boot test on my ebony board with this patch included.  It seems
> > to be happy about things so far.  If Matt gets around to trying this
> > out and it works, we should probably look at getting this into 2.6.24.
> > 
> > Oh, and I'd need a signed-off-by for it Ben :)
> 
> Sure, I'll send you a cleaned up version (the version I posted for test
> breaks 64 bits builds :-)
> 
> But I'm also waiting for Matt to test first.

Ok, just backported the 405 bits of this to my client's sad sad MV
2.6.10 kernel and it appears to work fine. Only tricky bit was
context.id -> context. So:

Tested-by: Matt Mackall <mpm@selenic.com>

And here's my backported patch, just for reference:

Index: k/arch/ppc/kernel/misc.S
===================================================================
--- k.orig/arch/ppc/kernel/misc.S	2007-10-29 15:33:03.000000000 -0500
+++ k/arch/ppc/kernel/misc.S	2007-10-29 15:34:29.000000000 -0500
@@ -496,7 +496,13 @@ _GLOBAL(_tlbia)
  */
 _GLOBAL(_tlbie)
 #if defined(CONFIG_40x)
+	mfmsr	r5
+	mfspr	r6,SPRN_PID
+	wrteei	0
+	mtspr	SPRN_PID,r4
 	tlbsx.	r3, 0, r3
+	mtspr	SPRN_PID,r6
+	wrtee	r5
 	bne	10f
 	sync
 	/* There are only 64 TLB entries, so r3 < 64, which means bit 25 is clear.
Index: k/arch/ppc/mm/fault.c
===================================================================
--- k.orig/arch/ppc/mm/fault.c	2007-10-29 15:30:04.000000000 -0500
+++ k/arch/ppc/mm/fault.c	2007-10-29 16:11:10.000000000 -0500
@@ -234,7 +234,7 @@ good_area:
 				set_bit(PG_arch_1, &page->flags);
 			}
 			pte_update(ptep, 0, _PAGE_HWEXEC);
-			_tlbie(address);
+			_tlbie(address, mm->context);
 			pte_unmap(ptep);
 			up_read(&mm->mmap_sem);
 			ltt_ev_trap_exit();
Index: k/arch/ppc/mm/mmu_decl.h
===================================================================
--- k.orig/arch/ppc/mm/mmu_decl.h	2007-10-29 15:34:48.000000000 -0500
+++ k/arch/ppc/mm/mmu_decl.h	2007-10-29 16:11:25.000000000 -0500
@@ -54,7 +54,7 @@ extern unsigned int num_tlbcam_entries;
 #define mmu_mapin_ram()		(0UL)
 
 #elif defined(CONFIG_4xx)
-#define flush_HPTE(X, va, pg)	_tlbie(va)
+#define flush_HPTE(pid, va, pg)	_tlbie(va, pid)
 extern void MMU_init_hw(void);
 extern unsigned long mmu_mapin_ram(void);
 
Index: k/include/asm-ppc/tlbflush.h
===================================================================
--- k.orig/include/asm-ppc/tlbflush.h	2007-10-29 15:31:29.000000000 -0500
+++ k/include/asm-ppc/tlbflush.h	2007-10-29 16:11:32.000000000 -0500
@@ -13,7 +13,7 @@
 #include <linux/config.h>
 #include <linux/mm.h>
 
-extern void _tlbie(unsigned long address);
+extern void _tlbie(unsigned long address, unsigned int pid);
 extern void _tlbia(void);
 
 #if defined(CONFIG_4xx)
@@ -28,10 +28,10 @@ static inline void flush_tlb_mm(struct m
 	{ __tlbia(); }
 static inline void flush_tlb_page(struct vm_area_struct *vma,
 				unsigned long vmaddr)
-	{ _tlbie(vmaddr); }
+	{ _tlbie(vmaddr, vma->vm_mm->context); }
 static inline void flush_tlb_page_nohash(struct vm_area_struct *vma,
 					 unsigned long vmaddr)
-	{ _tlbie(vmaddr); }
+	{ _tlbie(vmaddr, vma->vm_mm->context); }
 static inline void flush_tlb_range(struct vm_area_struct *vma,
 				unsigned long start, unsigned long end)
 	{ __tlbia(); }


-- 
Mathematics is the supreme nostalgia of our time.