Re: [RFC] Splitting kernel headers and deprecating __KERNEL__

Re: [RFC] Splitting kernel headers and deprecating __KERNEL__

[Andreas Schwab]

Linus Torvalds <torvalds@osdl.org> writes:

> We used to compile with "-nostdinc" to make sure that you couldn't include 
> user files even by mistake, but that was removed for some reason I can't 
> for the life of me remember any more.

??? We still do that, see NOSTDINC_FLAGS.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: [RFC] Splitting kernel headers and deprecating __KERNEL__

[Andreas Schwab]

Linus Torvalds <torvalds@osdl.org> writes:

> This is a common issue with namespace pollution. For example, this program 
> is perfectly valid afaik (well, except for being _stupid_, but that's 
> another issue):
>
> 	#include <stdio.h>
>
> 	const char *int32_t(int i)
> 	{
> 		return i ? "non-zero" : "zero";
> 	}

Actually this is not allowed in POSIX.  _Any_ header may define any
identifier ending with "_t".

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: [RFC] Splitting kernel headers and deprecating __KERNEL__

[Alexandre Oliva]

On Nov 30, 2004, Linus Torvalds <torvalds@osdl.org> wrote:

> On Tue, 30 Nov 2004, Alexandre Oliva wrote:

>> On Nov 30, 2004, Linus Torvalds <torvalds@osdl.org> wrote:

>> > If you want to use a legal analogy, the ABI is not a contract, it's a
>> > public _license_.

>> I didn't mean to use a legal analogy.  I meant contract in the
>> software engineering sense.  Sorry if that wasn't clear.

> Then your definition of a "contract" is flawed or your world-view has 
> nothing to do with reality.

It's not my definition, it's a definition used in software
engineering.  Sure, if you ask a legal mind, you'll get a different
answer.  If you ask a bridge player, you'll get yet another
definition.  That's why we have jargons in which common-use words have
their meanings narrowed to the specifics of certain fields.  And
that's why it's so hard for specialists in different fields to talk to
each other at times when there's disagreement about the precise
meaning of certain words.

> An ABI is not a contract.

Not a legal contract, for sure.

An ABI is a definition of an interface, including operations with pre-
and post-conditions, data structures with their invariants, constants,
file formats, etc.  Most of that is covered by the software
engineering term `contract'.

Sure enough, the headers we're talking about describe only a very
small portion on that; pretty much only data structures and
constants.  It certainly doesn't *specify* the entire ABI, it only
highlights portions of it.



Anyhow, all of this is beyond the point.  I see you've decreed that
people can introduce `user' directories in the kernel now.  Would you
please reconsider and choose a dir name that would enable the same ABI
headers to be used by kernel and userland, without adding a directory
to /usr/include that has no indication that it comes from the kernel?

Thanks,

-- 
Alexandre Oliva             http://www.ic.unicamp.br/~oliva/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}

Re: [RFC] Splitting kernel headers and deprecating __KERNEL__

[Ralf Baechle]

On Mon, Nov 29, 2004 at 09:42:16AM +0000, David Woodhouse wrote:

> I've lost track of the number of times things have broken because of
> incorrect use of kernel headers from userspace. That's what we're trying
> to fix -- by putting only the bits which are _supposed_ to be visible
> into files which userspace sees, where we know they define part of the
> userspace API and hence we can be extremely careful when editing them. 
> 
> I don't think it makes sense at this point for us to bury our collective
> heads in the sand and pretend there isn't a problem here that's worth
> fixing.
> 
> I agree that it should be obviously correct though -- and that's why
> we're trying to end up with a structure that in the first pass would
> give us in userspace essentially what we already have in the various
> glibc-kernheaders packages, but without the constant and unnecessary
> need for some poor sod to keep those up to date by hand.

The concept of copying kernel headers into applications is even worse
when arch portability is affected.  I stopped counting how often I had
to fix that kind of crap - and the state of kernel headers and userspace
kernel header packages is really provoking that kind of mess.

  Ralf

Re: [RFC] Splitting kernel headers and deprecating __KERNEL__

[Alexandre Oliva]

On Nov 30, 2004, Al Viro <viro@parcelfarce.linux.theplanet.co.uk> wrote:

> WTF?  I've got a dozen kernel trees hanging around, which one (and WTF any,
> while we are at it) should be "linked to"?

Whichever you chose to install in your /usr/include, and use as the
kernel ABI definition as far as userland is concerned.

I don't think `make install' should touch /usr/include at all.  It
should be a separate step, such that one can build a kernel abi
headers package out of the kernel source tree, ideally without even
having to configure it first, and use that as the kernel ABI
definition.  You should only have to do this again *if* the ABI
changes (ideally no removals, only additions) *and* you want to take
advantage of them, *and* you're willing to rebuild userland pieces
that could take advantage of them.

On most systems, people won't do that, they'll just use whatever
kernel headers and glibc binaries their distro ships.  If they want to
change any of these, they have to know what they're doing.

-- 
Alexandre Oliva             http://www.ic.unicamp.br/~oliva/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}

Re: [RFC] Splitting kernel headers and deprecating __KERNEL__

[Alexandre Oliva]

On Nov 30, 2004, Linus Torvalds <torvalds@osdl.org> wrote:

> Wrong. They'll _still_ have to maintain duplicates, since they can't rely
> ont he end-user to have a recent enough kernel.

No.  They just won't have the headers for the newer kernel installed
on their system.  They stick with the kernel ABI headers provided by a
baseline kernel, and that's probably from an older kernel tarball.

> It's just that they can hopefully start _copying_ their dupliates more 
> easily.

It's still the headers from the kernel, copied or not.  From a
different, baseline version, maybe.

> At RH you may see only the case where people do a whole-system upgrade. 

I personally don't care about the case you perceive as the Red Hat
case, because I work on embedded systems development, in a different
organization within Red Hat than the people who develop Red Hat
Enterprise Linux and Fedora.  So yes, in the case that actually
benefits me, it's even more of a whole-system thing, since for our
group it's *the* kernel, not *a* random kernel version.  It's the
kernel we port and ship to the customer, and the corresponding
kernel<->userland ABI.

That said, I do understand the issues of kernel baseline ABI settings,
or at least part of them.  I know you don't want to assume the
presence of a certain system call just because you're running a kernel
that implements it.  In fact, I know you don't want to assume the
presence of a system call just because it's defined in linux/unistd.h.
As long as the userland software keeps this distinction in mind, I
don't see why taking the latest kernel<->userland ABI headers, that
describe the newest system calls and any corresponding data
structures, could hurt anything.  In fact, if you don't have the
corresponding system call numbers and data structures, you can't
possibly take advantage of the new syscalls even if they happen to be
available, unless you duplicate the syscall numbers and data structure
definitions in your own code, which, from a software engineering
standpoint, is bad.

Of course, badly written software can assume the syscalls are present
just because their numbers are given, and then people who rely on such
bad software lose, but such is the nature of bad software.

-- 
Alexandre Oliva             http://www.ic.unicamp.br/~oliva/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}

Re: [RFC] Splitting kernel headers and deprecating __KERNEL__

[Alexandre Oliva]

On Nov 30, 2004, Linus Torvalds <torvalds@osdl.org> wrote:

> On Tue, 30 Nov 2004, Alexandre Oliva wrote:

>> Then maybe this is the fundamental problem.  As long as the kernel
>> doesn't recognize that an ABI is a contract, rather than an
>> imposition, kernel developers won't care.

> That's a silly analogy. Worse, it's a very flawed analogy.

> If you want to use a legal analogy, the ABI is not a contract, it's a
> public _license_.

I didn't mean to use a legal analogy.  I meant contract in the
software engineering sense.  Sorry if that wasn't clear.

-- 
Alexandre Oliva             http://www.ic.unicamp.br/~oliva/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}

Re: [RFC] Splitting kernel headers and deprecating __KERNEL__

[Alexandre Oliva]

On Nov 30, 2004, Alexandre Oliva <aoliva@redhat.com> wrote:

>> I've suggested "include/user/" and "include/asm-xxx/user", which handles 
>> architecture-specific parts too. I'm ok with doing it the other way 
>> around, ie "include/user/" and "include/user/arch-xxxx".

> As I pointed out, `user' is a very bad name.  As you said yourself,
> we're talking about the *kernel* ABI.  So what's `user' supposed to
> mean?  Was I so successful in my arguments that you now see it as the
> userland ABI? :-)

I got carried away with joking and failed to repeat why I consider it
a bad name (assuming that, since you missed the beginning of the
thread, you probably missed the first reply I posted to the message
that started it): since we're going to install these headers in
/usr/include (where headers for userland live), /usr/include/user is
quite misleading.  /usr/include/kernel would be far more
appropriate for this purpose.

Sure, we could take headers from linux-*/include/user and install them
in /usr/include/kernel, but then includes in there that reference
other headers in user/ or in asm-<arch>/ will cease to work.

So we should come up with a name that makes sense for both users of
these headers, which is why I suggested ukabi.  linux/abi and
asm-<mach>/abi work just as well, and then we can soft-link `abi -> .'
in /usr/include/{linux,asm-<mach>} if needed.  Ideally, we wouldn't
have to.

-- 
Alexandre Oliva             http://www.ic.unicamp.br/~oliva/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}

Re: [RFC] Splitting kernel headers and deprecating __KERNEL__

[Alexandre Oliva]

On Nov 30, 2004, Linus Torvalds <torvalds@osdl.org> wrote:

> On Tue, 30 Nov 2004, Alexandre Oliva wrote:
>> 
>> - move anything that is not protected by #ifdef __KERNEL__ to the
>> ukabi header tree, adding an include in the beginning of the original
>> header that includes the ukabi header.

> No. I want stuff that goes into the ABI tree to be clearly _defined_ to be 
> user-visible. Not a "let's move it all there and then prune it". 

Right.  Which brings us to my second bullet.  I *knew* I should have
made it the first :-/

> Also, "ukabi" just isn't going to fly as a name. It's also not as simple 
> as you seem to think, since a lot of these ABI things are architecture- 
> dependent, which apparently all you guys have totally ignored. 

Looks like you missed the post that started this thread.  It *did*
mention machine-independent and machine-specific user headers.

> I've suggested "include/user/" and "include/asm-xxx/user", which handles 
> architecture-specific parts too. I'm ok with doing it the other way 
> around, ie "include/user/" and "include/user/arch-xxxx".

As I pointed out, `user' is a very bad name.  As you said yourself,
we're talking about the *kernel* ABI.  So what's `user' supposed to
mean?  Was I so successful in my arguments that you now see it as the
userland ABI? :-)

>  (a) it can't break anything (ie the old location still includes the new 
>      one, exactly the same way)

You mean it can't break anything in a kernel build, or it can't break
anything except for userland apps that abused kernel headers and used
to get away with that?

>  (b) there are people who will actually take _advantage_ of that 
>      particular file (ie "just because I think so" doesn't fly).

People who currently get to maintain duplicates of these header
contents will take immediate advantage of these changes, since they
will no longer have to maintain the duplicates.  This is a major step
forward in terms of maintainability for everybody else, at no expense
to the kernel, that now gets to explicitly (as opposed to implicitly)
document what is the ABI it's willing to comply with.  It's a win for
both ends.

-- 
Alexandre Oliva             http://www.ic.unicamp.br/~oliva/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}

Re: [RFC] Splitting kernel headers and deprecating __KERNEL__

[Alexandre Oliva]

On Nov 29, 2004, Linus Torvalds <torvalds@osdl.org> wrote:

> On Mon, 29 Nov 2004, Alexandre Oliva wrote:
>> 
>> I don't see it as obvious at all.  The need for an agreement between
>> two parties on an ABI doesn't imply that one party gets to define it
>> and the other gets to follow it. 

> Sorry, but that's not how it works.

Then maybe this is the fundamental problem.  As long as the kernel
doesn't recognize that an ABI is a contract, rather than an
imposition, kernel developers won't care.

But guess what?  Kernel developers *do* care about maintaining the
userland ABI.  They don't just go wild changing data types,
structures, renumbering system calls and all that sort of nasty stuff
because there's a contract with userland that they don't want to
break.  Because breaking this contract would be a stupid thing, that
would require all affected userland to be modified to cope with the
change, even if that's just a recompile using the new kernel
headers...  erhm...  using the result of applying the kernel ABI
change in the copy/extract/whatever of kernel headers that userland is
entitled to use.


As long as we recognize that the notion that the kernel<->userland ABI
is a separate entity, that enables kernel and userland to exchange
information, rather than something that the kernel defines and
userland must follow, we give strength to the argument that userland
is not a derived work from the kernel.  If this argument proves to be
false, then only works released under the GNU GPL or a compatible
license will be allowed to run atop of Linux, because libc would be a
derived work, its license would become full GPL, and any program
linked with it could only be distributed under the full GPL.  Oh, and
just in case, IANAL :-)

Again, I wouldn't mind that at all, but I'm sure others feel
otherwise.  Besides, as I wrote before, if this is the intent, I'd
rather have it exposed in clear, rather than seeing people being
misled to believe it's fair game to run non-GPLed software on
Linux-based operating systems.

> He who writes the code decides what it is.

I've been in projects in which I wrote ABI headers before the
corresponding kernel port had even started.  Sometimes, quite
unfortunately, the kernel engineer went ahead and wrote headers with
slightly different assumptions, and then we had to get together to
define what the ABI was going to be.

In fact, the way you seem to want things to be, you want both kernel
and userland to write the code, and somehow hope they magically match
each other.  That's not how things work.  The way things work is that
either you get agreement first, or each end goes on its way and then
at some point they figure out where they diverged and fix things up,
or one end goes first and the other takes the definitions and uses
them.  It's in no way a kernel->userland one-way street.  It's a
language (for lack of a better word) that kernel and userland choose
to exchange information, not one that the kernel invents and then
requires others interested in talking to it to learn (you know, that
``Do you speak English/my language?´´ phrase so common in Hollywood
movies in which an American character needs to talk to someone abroad
:-)

Sometimes you'll even find formal ABI specifications, written down
before either userland or kernel start being developed.  Claiming that
the kernel gets to set the ABI as it wishes, without worrying about
userland because it never calls userland is a very narrow view, and in
some cases it's even a lie.  Think signal handling and process
start-up, for example: if you don't follow the userland ABI to call
the function pointer given to the signal function, or if you don't
follow the userland ABI to start the program's entry point, it's not
userland that is broken, it's the kernel.

> In this case, if the kernel 
> does a new extension, it's the kernel that gets to decide what it is. 

As in, say, NPTL?  As if a number of userland considerations hadn't
driven most of the design decisions behind the then-defined ABI
extensions?

> If glibc wants to do something new, go wild. The kernel won't care.

This is not about glibc.  That's not the only userland component.
There are several different alternatives one can use.

And then, the kernel will care, because it has to.  A kernel without
userland is nothing.  A kernel that kept introducing incompatible ABI
changes would die a very quick death.  A kernel has to be very careful
about complying with the ABI it shares with userland, and you know
that, because you do care about it.

> And that's really the fundamental issue. The kernel does not care what
> user land does. The kernel exports functionality, the kernel does _not_
> ask user land to help.

I'm not so much talking about the hundreds of syscalls, the syscall
conventions and the number and type of arguments given to each
syscall.  This is not even well defined in kernel headers.  All that's
exported regarding syscalls is a mapping from syscall names to
numbers.

What I'm talking about is the data structures.  Sure, the kernel can
define these however it likes.  So can userland.  The point is that,
if they don't match, when you get userland and kernel together, in the
hopes of getting something useful done, it doesn't work.

Both ends can come up with extensions, and do things the old way until
the other end picks up the extension.  I'll give you that many
extensions make to the kernel before say glibc, but that's mostly just
like, when we create a new toolchain port, assembler and linker are
generally implemented before the compiler.  I guess it just makes more
sense.  It wastes less effort to make sure you got something right in
a lower layer before you build something atop of it.  It's unlikely
that a change would make to glibc if it used a feature that isn't
available in the kernel, just because most often features undergo
(minor or sometimes major) changes when they're integrated, and it's
easier to fix things up if you haven't committed to the way that
didn't make it there yet.

> That _does_ make it a one-way street. Sorry.

Good, we're getting somewhere.  This is surely better than a zero-way
street, i.e., one in which headers that specify the ABI don't flow in
any direction.  If we're agreeing to make it one way, let's make it
so: have headers that are maintained as part of the kernel, and that
are imposed, if you like, on userland.  Saying it's a one-way street
but denying others from taking what comes their way through this
one-way street hasn't been very effective :-)

-- 
Alexandre Oliva             http://www.ic.unicamp.br/~oliva/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}

Re: [RFC] Splitting kernel headers and deprecating __KERNEL__

[Alexandre Oliva]

On Nov 30, 2004, Linus Torvalds <torvalds@osdl.org> wrote:

> I object sternuously to your "the header files". If you can't even say 
> _which_ header file, I'm not interested.

How about this formulation then:

- move anything that is not protected by #ifdef __KERNEL__ to the
ukabi header tree, adding an include in the beginning of the original
header that includes the ukabi header.  Since ukabi stuff can't depend on
kernel-internal stuff, having this include as the first thing in the
existing header should work.

- figure out what should have been protected by #ifdef __KERNEL__ but
wasn't, and move it back

- wait for bug reports from the world and deal with them

- repeat until you feel you got it right


> See what I'm saying? Whole-sale "move things around because we want to" 
> I'm not interested in. Specific problems, yes.

The specific problem we're trying to address is the creation of header
files that define the ABI between userland and kernel, that both
kernel and userland can use.

>> Personally, I'd prefer us to move to using standard C99 types in lieu of u32
>> and co at least for the interface to userspace because they are just that:
>> standard.

> No. I told you why it cannot and MUST NOT be done. Repeat after me:

> 	WE MUST NOT POLLUTE THE NAMESPACE!

David, Linus is right.  Using standard types is not an option because
they're not built in type names; you have to include a header to bring
them in, and ideally you shouldn't gratuitously get the names defined
just because you include some unrelated header, although the C
Standard grants standard headers freedom to include other headers or
something along these lines; I don't recall the exact passage of the
Standard that says so, but we should strive to not pollute the user
namespace anyway.

-- 
Alexandre Oliva             http://www.ic.unicamp.br/~oliva/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}

Re: [RFC] Splitting kernel headers and deprecating __KERNEL__

[Alexandre Oliva]

On Nov 29, 2004, Linus Torvalds <torvalds@osdl.org> wrote:

> On Mon, 29 Nov 2004, Alexandre Oliva wrote:

>> - Linux gets to define the ABI between kernel and userland, and
>> userland must duplicate the contents of headers in which the kernel
>> defines the kernel<->userland ABI, tracking changes in them in the
>> hope that nothing falls through the cracks

> This is unquestionably true. The kernel obviously _does_ define the
> ABI, and userland just lives with it.

I don't see it as obvious at all.  The need for an agreement between
two parties on an ABI doesn't imply that one party gets to define it
and the other gets to follow it.  An ABI has to be set, and then
kernel and userland get to follow it, but I don't see why it should be
the kernel that determines it.  It might very well be userland that
sets the ABI, and the kernel is implemented so as to comply.  Stating
it the way you did is what I alluded to as being an arrogant attitude
in my previous e-mail.  It sounds as if the kernel was the center of
the universe, and everything else gravitated around it.

Just like userland needs a kernel to operate, a kernel without
any userland programs to run is pretty much useless.  Yes, I know one
can point out examples of exceptions to both statements, but those are
just that, exceptions.

> At some point you have to track things, just because new features
> etc just can't be sanely handled any other way.

This applies to both sides of the bridge.

>> - we move the kernel<->userland to a separate package, where it's
>> maintained such that it can be used by both kernel and userland, and
>> Linux will only build when given a pointer to the location of this
>> package.

> No. Quite frankly, I absolutely hate projects that do that.

I'm glad to see we agree on this.  I happen to find the current
proposition of duplicating contents of kernel headers in some other
package just because `kernel headers are just that' just as
disgusting.  If you want to get to set the ABI, which you appear to
take as a given (and, in spite of my argumentation above, I'm fine
with that), you should do so in a way in which others can sanely
comply with the ABI.  Duplicating header contents is not a sane way to
do it.  Just consider that the problem of tracking the dependencies
needed to build Linux, should the ukabi headers be moved out of it,
affects any project that purports to maintain this set of headers for
userland use outside the kernel, just in a more subtle way: you need
not only to make sure that you have the package that offers the
interfaces you want to use, but also you have to make sure you have a
specific version of that package that matches the kernel baseline your
userland is intended to work with.

Although I can see that, for established ports, the rate of change in
the ukabi is quite small (addition of syscalls, mostly), during the
development of a new port, the rate of change is very high, and that's
the point in which you need the most to make sure that the ABIs are in
sync, and maintaining them separately is exactly the right way to make
sure they aren't.

> Now, on trying to make the tracking _easier_, I would not mind at all to 
> move (well-defined) things around a bit to make it clearer what is 
> actually exported to user space. But on the other hand, I don't think it's 
> actively wrong either to just "mark" them in the headers some way, and 
> have tools to extract it automatically _without_ having to separate them 
> into some magic location.

As long as the marks are not ones that actually require C
preprocessing, which would render them useless to extract bits in a
form that makes them usable for a multi-ABI userland targets, I happen
to like this better as well.  #ifdef __KERNEL__ can be easily
machine-extracted (I know dwmw2 has a script for that, dunno whether
he wrote it himself or got it from someone else), and the extraction
of bits for userland could even be done as part of say a make
install-ukabi-headers.

There are problems, though: sometimes, someone will make a change such
as adding one #include to a header file that is needed because of
kernel-only bits, and if this include is nor protected by the same
guards and a possibly empty header file with that name is not
installed as part of the ukabi, userland builds fail.

So, again, we agree that the marking should be more explicit than
implicit export.  The advantage of __KERNEL__, however, is that it
makes the very same header files (presumably) usable for userland and
the kernel, which could save some space and speed up the installation
of such headers.  I'm not sure this advantage is worth the trouble.


Another important problem about sharing headers between kernel and
userland, that I don't recall anyone having brought up before, is that
of licensing.  If the kernel headers are licensed under the GNU GPL,
and say a libc that is GNU LGPL in the source form uses them, it might
inherit the full GPL protection and start imposing the GPL
requirements on binaries linked with it.  Since the intent of the
ukabi headers is to set the interface between kernel and userland and
AFAIK you copyright protection doesn't apply to interfaces (IANAL so
this is probably quite inaccurate), one might get away with that, but
since there's often more than just interfaces in headers that are used
by both kernel and userland, the argument might fail.  Some (myself
included) would probably consider this trap against proprietary
software a good thing, but others might consider it a huge problem.

If not for any other reason, at least for the sake of openness, it
would probably be a good thing to separate the ukabi contents from
headers that define other kernel-internal macros, inline functions,
data structures, etc, and perhaps use different licensing terms in
these split-out headers, explaining to what extent using them limits
the licensing choices available to programs that include them.

-- 
Alexandre Oliva             http://www.ic.unicamp.br/~oliva/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}

Re: [RFC] Splitting kernel headers and deprecating __KERNEL__

[Alexandre Oliva]

On Nov 28, 2004, Linus Torvalds <torvalds@osdl.org> wrote:

> On the bigger question of what to do with kernel headers in general, let's 
> just make one thing clear: the kernel headers are for the kernel, and big 
> and painful re-organizations that don't help _existing_ user space are not 
> going to happen.

So, of the following two options, which one you think everybody should
pick?

- Linux gets to define the ABI between kernel and userland, and
  userland must duplicate the contents of headers in which the kernel
  defines the kernel<->userland ABI, tracking changes in them in the
  hope that nothing falls through the cracks

- we move the kernel<->userland to a separate package, where it's
  maintained such that it can be used by both kernel and userland, and
  Linux will only build when given a pointer to the location of this
  package.


I don't think you can have it both ways.  As long as the kernel wants
freedom to set the ABI, either let people use the headers where you
define it, or maintain the ABI separately and let people use them.
Forcing others to maintain duplicates of your headers will just get
most people to use your headers for purposes they weren't meant to be
used; maintaining two similar but far from identical sets of headers
in sync is a major pain, that's why few people do it, and even fewer
people do it properly.

Refusing to help in the effort to maintain them for wishes of freedom
to tinker or allergy to external dependencies haven't worked as
excuses in the past, and I think they never will.  Duplicating the
effort is just stupid, and requiring others to do so is arrogant.  Why
shouldn't we just get together and have something that works for both?

Or how about we turn the table around and define an ABI that the
kernel must implement, and then have the kernel have to keep track of
the changes in such external ABI definition, instead of the way things
(fail to) work now?  How comfortable would you feel with this
arrangement?


My usual reply when I hear this argument that the kernel headers that
define the ABI with userland are to be used by the kernel only, and
someone else is supposed to create the headers for the same purpose to
be used by userland, is that you generally won't find the headers that
expose the interface of a library in a separate package (think
libz.{a,so} and zlib.h).  They're part of the same package, built out
of the same sources, and there's a reason for that: if they get out of
sync, it won't serve its purpose.  It makes far more sense to maintain
them together.

You won't find zlib.h in packages that just want to use zlib, unless
they contain zlib in its entirety, just in case it's not
pre-installed.  I think this makes perfect sense.  Requiring libc or a
separate package used by libc to contain copies of headers that define
the API/ABI exported by the kernel to userland is just the same sort
of nonsense, with the difference that it's a way bigger set of headers
we're talking about, which renders the problem even worse.

Sure you can take the easy way out and claim you told people they
shouldn't do what they're doing when they run into problems because of
including kernel headers from userland.  But as soon as you reject
proposals to clean things up such that we would have a set of headers
that both kernel and userland will be able to use, in a way that would
have the headers still conveniently (for you) sitting in the kernel
source tree, I can't see that you could say it's not your problem with
a straight face any more.

-- 
Alexandre Oliva             http://www.ic.unicamp.br/~oliva/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}

Re: [RFC] Splitting kernel headers and deprecating __KERNEL__

[Alexandre Oliva]

On Nov 25, 2004, David Howells <dhowells@redhat.com> wrote:

> 	SOURCE			INSTALLED AS
> 	======================	============
> 	include/user/		/usr/include/user/
> 	include/user-i386/	/usr/include/user-i386/
> 				/usr/include/linux -> user
> 				/usr/include/asm -> user-i386

Although user/ and user-* make a lot of sense within the kernel source
tree, I don't think these names would be very clear in /usr/include.
I'd rather use names in /usr/include that more clearly associate them
with the kernel.  Heck, even /usr/include/asm is inappropriate, but
it's been there for so long that we really shouldn't try to get rid of
it.

If I had it my way, we'd have, in the kernel tree, userland-aimed
headers in include/linux/user and include/asm-<machine>/user, and have
them installed in /usr/include/linux and /usr/include/asm-<machine>.

This means these headers shouldn't reference each other as
linux/user/something.h, but rather as linux/something.h, such that
they still work when installed in /usr/include/linux.  This may
require headers include/linux/something.h to include
linux/user/something.h, but that's already part of the proposal.

-- 
Alexandre Oliva             http://www.ic.unicamp.br/~oliva/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}