[Bug translator/7042] New: Make upstream kernel developers happy

[Bug translator/7042] New: Make upstream kernel developers happy

[mhiramat at redhat dot com]

They requires more explicit commitments to upstream kernel.
- refine their requirements
   - picking up previous mails.
   - ask them what we can do for them.
- better integration with upstream kernel
   - merging minimum runtime.
   - merging minimum systemtap(or systemtap like scriptable tracer).
- easy to use of systemtap on upstream kernel
   - no-dwarf(debuginfo) mode by default.
   - porting tapsets to upstream kernel.

This entry is a meta entry, please list up your ideas.

-- 
           Summary: Make upstream kernel developers happy
           Product: systemtap
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P1
         Component: translator
        AssignedTo: systemtap at sources dot redhat dot com
        ReportedBy: mhiramat at redhat dot com


http://sourceware.org/bugzilla/show_bug.cgi?id=7042

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/7042] Make upstream kernel developers happy

[mhiramat at redhat dot com]

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
  BugsThisDependsOn|                            |7043


http://sourceware.org/bugzilla/show_bug.cgi?id=7042

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/7042] Make upstream kernel developers happy

[mhiramat at redhat dot com]

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
  BugsThisDependsOn|                            |7052


http://sourceware.org/bugzilla/show_bug.cgi?id=7042

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/7042] Make upstream kernel developers happy

[mhiramat at redhat dot com]

------- Additional Comments From mhiramat at redhat dot com  2008-12-04 18:21 -------
Here is a video of James presentation in a conference in Japan, 

http://www.linuxfoundation.org/events/node/174

From 00:54:00-, he answered a question about off-kernel project like systemtap.
We might have to respond his message and change our way.


-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=7042

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/7042] Make upstream kernel developers happy

[fche at redhat dot com]

------- Additional Comments From fche at redhat dot com  2008-12-04 22:47 -------
> http://www.linuxfoundation.org/events/node/174
> From 00:54:00-, he answered a question about off-kernel project like systemtap.
> We might have to respond his message

http://sources.redhat.com/systemtap/wiki/Myths

> and change our way

OK, but what changes would that particular video inspire?


-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=7042

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/7042] Make upstream kernel developers happy

[mhiramat at redhat dot com]

------- Additional Comments From mhiramat at redhat dot com  2008-12-05 04:21 -------
(In reply to comment #2)
> OK, but what changes would that particular video inspire?

From his presentation, I think he want us to follow "upstream first" policy.
As he said in the video that out-of-kernel project like closed source driver costs
high to follow the upstream kernel.

I definitely agree that it's hard to merge whole of the systemtap into kernel tree.

however, I think most of our runtime code(and parts of code generated by translator)
can be merged into kernel. (ex. safety memory access(which has already been
merged!),
trace framework including probing and buffering)

And if we merge them, we can concentrate on developing translator, and will not take
care about upstream kernel changes any more.


-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=7042

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/7042] Make upstream kernel developers happy

[fche at redhat dot com]

------- Additional Comments From fche at redhat dot com  2008-12-05 17:14 -------
> > OK, but what changes would that particular video inspire?
> From his presentation, I think he want us to follow "upstream first" policy.
He made many claims in that talk, but that one did not seem to be one of them.

I'm not saying that attempting to put some parts of the runtime into
the kernel is a bad thing - just that he was off that topic.


-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=7042

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/7042] Make upstream kernel developers happy

[mhiramat at redhat dot com]

------- Additional Comments From mhiramat at redhat dot com  2008-12-07 00:12 -------
BTW, here is another video of the presentation by Ted Ts'o.

http://www.linuxfoundation.org/events/node/175

From 00:39:40-, I think he explained what was the problem of Systemtap project
more clearly.
- There is no *senior* kernel developers in the project.
 - To develop useful tapset, they(systemtap) need a help from kernel subsystem
maintainers.
 - RAS tools is usually for Enterprise users and not for kernel community, it's
not useful.
- Now kernel community developing their own tracer(ftrace).



-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=7042

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/7042] Make upstream kernel developers happy

[mhiramat at redhat dot com]

------- Additional Comments From mhiramat at redhat dot com  2008-12-07 00:18 -------
(In reply to comment #2)
> http://sources.redhat.com/systemtap/wiki/Myths

By the way, I think that many people claimed that the systemtap always required
kernel-debuginfo which was too big and was hard to find an appropriate package.


-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=7042

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

Re: [Bug translator/7042] Make upstream kernel developers happy

[Frank Ch. Eigler]

"mhiramat at redhat dot com" <sourceware-bugzilla@sourceware.org> writes:

> By the way, I think that many people claimed that the systemtap
> always required kernel-debuginfo which was too big and was hard to
> find an appropriate package.

I wouldn't call this a myth.  It is big and on some distributions hard
to find.  However, the latter should not impact upstream kernel
developers who build their own kernels.  And the former is not up to
systemtap per se to fix.

- FChE

[Bug translator/7042] Make upstream kernel developers happy

[mhiramat at redhat dot com]

------- Additional Comments From mhiramat at redhat dot com  2009-01-12 19:53 -------
Here is the action ideas which I sent to systemtap-ml on
(http://sources.redhat.com/ml/systemtap/2008-q4/msg00639.html)

- Communication issue
 - Notifying releasing on LKML frequently.
 - Preparing more documents for kernel developers.
 - Feeding back kernel developers' opinions.

- Solving debuginfo issue
 - in the short term, notifying how to minimize debuginfo size.
 - in the mid term, making dummy (function-entry) debuginfo.
 - in the long term, Roland's dwarf compressor can solve this issue.

- Upstream first issue
 - Utrace/Uprobe
  - Develop code on LKML and embrace feedbacks.
  - Involving main kernel maintainer to speed up the code merging.
 - Systemtap itself
  - Merging a part of runtime and basic tapset to kernel tree, which
   will provide us a permanent API for systemtap generated code.
  (BTW, I'd like to suggest modifying ftrace plug-able and systemtap
   generating ftrace-plugin tracers. This will be more acceptable for
   upstream because we don't need to add so many code).


-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=7042

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/7042] Make upstream kernel developers happy

[ananth at in dot ibm dot com]

------- Additional Comments From ananth at in dot ibm dot com  2009-01-13 04:34 -------
Would what you illustrate in comment #7 entail a new git branch for upstream
merge, especially for the runtime and buffer portions?

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=7042

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/7042] Make upstream kernel developers happy

[mhiramat at redhat dot com]

------- Additional Comments From mhiramat at redhat dot com  2009-01-13 07:47 -------
(In reply to comment #8)
> Would what you illustrate in comment #7 entail a new git branch for upstream
> merge, especially for the runtime and buffer portions?

Yes, because we need to clean up the runtime code for merging, and share it.
After merging, we may still have a thin layer of runtime code.


-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=7042

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/7042] Make upstream kernel developers happy

[mhiramat at redhat dot com]

------- Additional Comments From mhiramat at redhat dot com  2009-01-22 17:12 -------
(In reply to comment #7)
>  - Systemtap itself
>   - Merging a part of runtime and basic tapset to kernel tree, which
>    will provide us a permanent API for systemtap generated code.
>   (BTW, I'd like to suggest modifying ftrace plug-able and systemtap
>    generating ftrace-plugin tracers. This will be more acceptable for
>    upstream because we don't need to add so many code).

What I suggested here is changing runtime code (and translator too).
It will lose some kind of backward compatibility, but since we can't bump ftrace
out from kernel any more, there is no way to merge runtime directly to kernel.
Instead of that, we have to use and improve ftrace interface for systemtap.

So, Here are what we have to do, I think
- Assign someone to runtime merging task, who may talk with kernel developers.
- Modify ftrace to support kernel modules. this might have to involve Steven
Rostedt.
- Modify staprun/stapio to support ftrace interface, and replace transport
functions with ftrace interfaces.
- Get rid of duplicate codes from runtime.
- Modify translator to generate ftrace module.
- Push useful parts of runtime code to upstream kernel(might be "kernel APIs for
tracers").

ftrace and systemtap should be able to coexist on the kenrel.


-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=7042

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/7042] Make upstream kernel developers happy

[mhiramat at redhat dot com]

------- Additional Comments From mhiramat at redhat dot com  2009-01-27 01:21 -------
Here, I'd like to try to figure out technical details of my idea.

> - Modify ftrace to support kernel modules. this might have to involve
> Steven Rostedt.

This basically means exporting register_tracer() for kernel modules and adding
unregister_tracer().

> - Modify staprun/stapio to support ftrace interface, and replace transport
> functions with ftrace interfaces.

staprun may not be changed so much. stapio will be changed, but mainly the
change will be just about the path of debugfs file.

> - Get rid of duplicate codes from runtime.

this needs more investigation. I assume that most of autoconfs will be removed
and some of safety memory/register accessing code should be merged to upstream
kernel for debugging.

> - Modify translator to generate ftrace module.

This doesn't means using ftrace_printk(). Systemtap will generate a kernel
module which has a struct tracer and its own ring buffer file(because several
tracers will run in parallel).

> - Push useful parts of runtime code to upstream kernel(might be "kernel
> APIs for tracers").

As I said above, some of safety code should be merged, and other utility codes
including unwinder, map, statistics, taskfinder and so on.



-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=7042

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/7042] Make upstream kernel developers happy

[kosaki dot motohiro at jp dot fujitsu dot com]

------- Additional Comments From kosaki dot motohiro at jp dot fujitsu dot com  2009-01-27 10:46 -------
Hi

if Systemtap people works on upstream merging, Fujitsu might help some works and
activity.
please contact me.

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=7042

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/7042] Make upstream kernel developers happy

[fche at redhat dot com]

------- Additional Comments From fche at redhat dot com  2009-01-29 19:26 -------
(In reply to comment #11)
> > - Modify ftrace to support kernel modules. this might have to involve
> > Steven Rostedt.
> 
> This basically means exporting register_tracer() for kernel modules and adding
> unregister_tracer().

OK...

> > - Modify staprun/stapio to support ftrace interface, and replace transport
> > functions with ftrace interfaces.
> 
> staprun may not be changed so much. stapio will be changed, but mainly the
> change will be just about the path of debugfs file.

OK, bug #7043.

> > - Get rid of duplicate codes from runtime.
> this needs more investigation. I assume that most of autoconfs will be removed

(But they can't, if we wish to still run on non-bleeding-edge kernels.)

> and some of safety memory/register accessing code should be merged to upstream
> kernel for debugging.

Sure, if there is upstream interest in those small and stable bits.

> > - Modify translator to generate ftrace module.
> 
> This doesn't means using ftrace_printk(). Systemtap will generate a kernel
> module which has a struct tracer and its own ring buffer file(because several
> tracers will run in parallel).


> > - Push useful parts of runtime code to upstream kernel(might be "kernel
> > APIs for tracers").
> 
> As I said above, some of safety code should be merged, and other utility codes
> including unwinder, map, statistics, taskfinder and so on.

The unwinder is blocked on more experience with it.
The task-finder is blocked on utrace.

The map/statistics stuff could go, to see if kernel folks were interested in
having it.  (Since this part has not needed porting effort, the status quo does
not represent a cost to the project.)  Who wants to try proposing it?

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=7042

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/7042] Make upstream kernel developers happy

[ananth at in dot ibm dot com]

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |ananth at in dot ibm dot com


http://sourceware.org/bugzilla/show_bug.cgi?id=7042

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/7042] Make upstream kernel developers happy

[mhiramat at redhat dot com]

------- Additional Comments From mhiramat at redhat dot com  2009-02-10 17:30 -------
> > > - Get rid of duplicate codes from runtime.
> > this needs more investigation. I assume that most of autoconfs will be removed
> 
> (But they can't, if we wish to still run on non-bleeding-edge kernels.)

Hmm, so, how about just branching sources for newer kernels?
Fortunately, we already have -R option.

> > and some of safety memory/register accessing code should be merged to upstream
> > kernel for debugging.
> 
> Sure, if there is upstream interest in those small and stable bits.

Those stuffs will be useful for other developers who want to write a
custom tracer in plain c.

> > > - Push useful parts of runtime code to upstream kernel(might be "kernel
> > > APIs for tracers").
> > 
> > As I said above, some of safety code should be merged, and other utility codes
> > including unwinder, map, statistics, taskfinder and so on.
> 
> The unwinder is blocked on more experience with it.
> The task-finder is blocked on utrace.
> 
> The map/statistics stuff could go, to see if kernel folks were interested in
> having it.  (Since this part has not needed porting effort, the status quo does
> not represent a cost to the project.)  Who wants to try proposing it?

It's a part of runtime merging task. We may need to provide some ftrace-based
examples for these parts as Frank did for utrace.

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=7042

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/7042] Make upstream kernel developers happy

[mhiramat at redhat dot com]

------- Additional Comments From mhiramat at redhat dot com  2009-02-12 00:24 -------
I still doubt that just using ftrace's ring buffer can clear all of the
criticism from kernel developers. Perhaps, we need more interaction between us
and them. That means we may need to improve kernel itself by using some portion
of systemtap code or our knowledge. That is what they hope.

Therefore, I think systemtap-ftrace integration has big advantage for that purpose.
- Kernel can have only one trace infrastructure, it simplifies code maintenance.
- Kernel developers can easily make prototypes of their tracers by using
systemtap. When they think that tracer is enough stable and useful, they can
push the refined c code into kernel. So, it's good for ftrace too.
- systemtap can show its presence in the kernel community.


-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=7042

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/7042] Make upstream kernel developers happy

[fche at redhat dot com]

------- Additional Comments From fche at redhat dot com  2010-06-09 13:23 -------
This is probably moot, with more lkml attention on perf.

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |INVALID


http://sourceware.org/bugzilla/show_bug.cgi?id=7042

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/7042] Make upstream kernel developers happy

[fche at redhat dot com]

-- 
Bug 7042 depends on bug 7052, which changed state.

Bug 7052 Summary: branch upstream-only runtime code
http://sourceware.org/bugzilla/show_bug.cgi?id=7052

           What    |Old Value                   |New Value
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |WONTFIX

http://sourceware.org/bugzilla/show_bug.cgi?id=7042

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.