Questions on glibc submission for a new project

Questions on glibc submission for a new project

[Madhavan T. Venkataraman]

Hi all,

I am planning to submit an RFC for a glibc project. This is my first time. So, I have some
questions.

Description of the project
--------------------------

I am developing some API functions to create and use trampolines so that applications and
libraries don't have to each implement them (often using dynamic code).

	void	*tramp_alloc(int flags);
	void	tramp_set_parms(void *tramp, void *code, void *data);
	void	*tramp_get_addr(void *tramp);
	void	tramp_free(void *tramp);
	int	tramp_supported(void);

I will provide a full justification in the RFC. For now, I just have some questions.

Copyright assignment
--------------------

I am developing these functions under Linux. I see that the files under sysdeps/unix/sysv/linux
have a "GNU Lesser General Public License". It appears that I should just use that and not need
to apply for a new FSF Copyright assignment.

Am I correct in my assumption?

If I need to apply for something, could you please advise me on what exactly I need to do?

Placement of the code
----------------------

I have the following files in the project:

	- tramp.h (contains the function prototypes of the API functions)
	- tramp.c (contains the common code for creating and managing trampolines)
	- tramp_arch.c (contains architecture-specific C code)
	- tramp_code.S (contains architecture-specific assembly code)

It looks like I must place the files here:

	sysdeps/unix/sysv/linux/tramp.c
	sysdeps/unix/sysv/linux/tramp.h
	sysdeps/unix/sysv/linux/x86/tramp_arch.c (etc for each arch)
	sysdeps/unix/sysv/linux/x86/tramp_code.S (etc for each arch)

Did I get that right?

Other OSes
----------

These functions can potentially be implemented for other OSes as well. There are,
however, some OS-specific details such as mmap() details and traversing the list of
mapped objects in an address space.

I have two options:

1. Implement it for Linux. In the future, when it is implemented for another OS,
   deal with the changes required.

2. Implement it from the beginning as a common feature and provide OS-specific code
   for Linux.

Which one would you recommend? If it is (2), could you point me to an example within
glibc that I can follow for how to organize the code for a multi-OS feature?

Documentation
-------------

I do plan to submit a Linux manpage. Do I have to do that after my final patch gets accepted into
glibc? When exactly do I need to do that?

Self-tests
----------

I do plan to submit tests along with the patch. But do I need to submit tests for the RFC?
Or, is it enough for me to submit tests along with the actual patch later?

mailing list
-------------

I will send out my RFC (and an actual patch later) to libc-alpha. Do I need to include
anyone else in the review? How about folks for each architecture?

Public header
-------------

tramp.h would be a public header file. Is there any specific process I need to follow for
introducing a public header file?

Architecture support
--------------------

I will be introducing support for X86 and ARM, 32-bit and 64-bit. Is there any special
process I need to follow if my changes contain architecture-specific changes?

Thank you all in advance for your guidance. Much appreciated.

Madhavan

Re: Questions on glibc submission for a new project

[Carlos O'Donell]

On 10/15/20 3:05 PM, Madhavan T. Venkataraman via Libc-help wrote:
> Hi all,
> 
> I am planning to submit an RFC for a glibc project. This is my first time. So, I have some
> questions.

Glad to hear that!

I look forward to reading and reviewing the RFC.
 
> Description of the project
> --------------------------
> 
> I am developing some API functions to create and use trampolines so that applications and
> libraries don't have to each implement them (often using dynamic code).
> 
> 	void	*tramp_alloc(int flags);
> 	void	tramp_set_parms(void *tramp, void *code, void *data);
> 	void	*tramp_get_addr(void *tramp);
> 	void	tramp_free(void *tramp);
> 	int	tramp_supported(void);
> 
> I will provide a full justification in the RFC. For now, I just have some questions.

No problem.

> Copyright assignment
> --------------------
> 
> I am developing these functions under Linux. I see that the files under sysdeps/unix/sysv/linux
> have a "GNU Lesser General Public License". It appears that I should just use that and not need
> to apply for a new FSF Copyright assignment.
> 
> Am I correct in my assumption?

Please have a read through the "Contribution checklist"
https://sourceware.org/glibc/wiki/Contribution%20checklist

> If I need to apply for something, could you please advise me on what exactly I need to do?

I see you are submitting this email via a @linux.microsoft.com account.

I can't tell if this is a hobby for you or a work project, either way, the rules are the
same and they still apply.

I am TO'ing Ian Bearman and Roman Kiselev to see if they can assist you with this internally.

In summary:
- If Microsoft owns the copyright of the code you write then you'll need to go through legal
  to get a corporate assignment completed with the Free Software Foundation. This is the same
  process that companies like Intel, IBM, Red Hat, Amazon etc. have followed for contribution.
- If Microsoft does not own the copyright of the code you write then you'll need a personal
  assignment.

My suggestion is talk to Ian and Roman and see if there is already a process inside Microsoft
for this. I can tell you that Microsoft has signed an assignment for the GNU Debugger (gdb)
just last year.

I always suggest getting a future assignment in place which allows us to accept any and all
patches from not into the future:
http://git.savannah.gnu.org/cgit/gnulib.git/plain/doc/Copyright/request-assign.future

I also suggest getting GCC GLIBC GDB and BINUTILS assignments all done at the same time.
These 4 core projects constitute the core pieces of the GNU Toolchain and as such it's
sometimes required to synchronize headers across projects. However, to start with you'll
need GLIBC if you plan to contribute to glibc.

> Placement of the code
> ----------------------
> 
> I have the following files in the project:
> 
> 	- tramp.h (contains the function prototypes of the API functions)
> 	- tramp.c (contains the common code for creating and managing trampolines)
> 	- tramp_arch.c (contains architecture-specific C code)
> 	- tramp_code.S (contains architecture-specific assembly code)

OK.
 
> It looks like I must place the files here:
> 
> 	sysdeps/unix/sysv/linux/tramp.c
> 	sysdeps/unix/sysv/linux/tramp.h


Sure.

> 	sysdeps/unix/sysv/linux/x86/tramp_arch.c (etc for each arch)
> 	sysdeps/unix/sysv/linux/x86/tramp_code.S (etc for each arch)


Sure.

> Did I get that right?

Yes, that's reasonable.

You'll also need a sysdeps/unis/sysv/linux/tramp_[arch,code].[c,s]
that implements a stub, because I expect you won't add support for all arches
right away?

> Other OSes
> ----------
> 
> These functions can potentially be implemented for other OSes as well. There are,
> however, some OS-specific details such as mmap() details and traversing the list of
> mapped objects in an address space.
> 
> I have two options:
> 
> 1. Implement it for Linux. In the future, when it is implemented for another OS,
>    deal with the changes required.

This.

> 2. Implement it from the beginning as a common feature and provide OS-specific code
>    for Linux.

This is made complicated by the fact that you have to get the design right the first
time for Linux and Hurd.

> Which one would you recommend? If it is (2), could you point me to an example within
> glibc that I can follow for how to organize the code for a multi-OS feature?

I recommend (1).

> Documentation
> -------------
> 
> I do plan to submit a Linux manpage. Do I have to do that after my final patch gets accepted into
> glibc? When exactly do I need to do that?

Documentation should be added to glibc/manual/*.texi in the appropriate place.

This includes considerations for threaded safety, signal safety, and cancellation safety.

> Self-tests
> ----------
> 
> I do plan to submit tests along with the patch. But do I need to submit tests for the RFC?
> Or, is it enough for me to submit tests along with the actual patch later?

You don't need to submit tests for the RFC.
 
> mailing list
> -------------
> 
> I will send out my RFC (and an actual patch later) to libc-alpha. Do I need to include
> anyone else in the review? How about folks for each architecture?

The machine maintainers should be reading the list.

If there is anyone in particular you want to call out you can do so from the MAINTAINERS list:
https://sourceware.org/glibc/wiki/MAINTAINERS

> Public header
> -------------
> 
> tramp.h would be a public header file. Is there any specific process I need to follow for
> introducing a public header file?

No. You just have to convince the maintainers that the API belongs in glibc.
 
> Architecture support
> --------------------
> 
> I will be introducing support for X86 and ARM, 32-bit and 64-bit. Is there any special
> process I need to follow if my changes contain architecture-specific changes?
 
You will be adding a new symbol to the ABI baseline for all architectures.

You will need to update all of the symbol baselines with `make update-all-abi` (IIRC).
You will have to explain what other machine maintainers need to do to complete the support.

Please consider the following:
- Can the functionality be provided by putting the code in another library? Why glibc?
- Is it tightly coupled to PLTs? LD_AUDIT of PLTs? Will LD_AUDIT support the trampolines?
- What do other C library authors think of the interface? You could post to libc-coord.
  https://www.openwall.com/lists/libc-coord/2020/01/30/1
- What are the parallelism and concurrency requirements for the interface? What happens
  if multiple threads use the interface and what hardware atomics are required for it?
- How does this trampoline relate to libffi's trampolines?
- Does the trampoline require executable memory? How do you plan to get that under
  conditions like SELinux?

I'm also TO'ing Florian Weimer who has in the past suggested some kind of
trampoline creation interfaces for the dynamic loader.

-- 
Cheers,
Carlos.

Re: Questions on glibc submission for a new project

[Madhavan T. Venkataraman]

Thanks for the answers. Responses inline...

On 10/15/20 3:22 PM, Carlos O'Donell wrote:
> On 10/15/20 3:05 PM, Madhavan T. Venkataraman via Libc-help wrote:
>> Hi all,
>>
>> I am planning to submit an RFC for a glibc project. This is my first time. So, I have some
>> questions.
> 
> Glad to hear that!
> 
> I look forward to reading and reviewing the RFC.
>  
>> Description of the project
>> --------------------------
>>
>> I am developing some API functions to create and use trampolines so that applications and
>> libraries don't have to each implement them (often using dynamic code).
>>
>> 	void	*tramp_alloc(int flags);
>> 	void	tramp_set_parms(void *tramp, void *code, void *data);
>> 	void	*tramp_get_addr(void *tramp);
>> 	void	tramp_free(void *tramp);
>> 	int	tramp_supported(void);
>>
>> I will provide a full justification in the RFC. For now, I just have some questions.
> 
> No problem.
> 
>> Copyright assignment
>> --------------------
>>
>> I am developing these functions under Linux. I see that the files under sysdeps/unix/sysv/linux
>> have a "GNU Lesser General Public License". It appears that I should just use that and not need
>> to apply for a new FSF Copyright assignment.
>>
>> Am I correct in my assumption?
> 
> Please have a read through the "Contribution checklist"
> https://sourceware.org/glibc/wiki/Contribution%20checklist
> 

OK. Will do.

>> If I need to apply for something, could you please advise me on what exactly I need to do?
> 
> I see you are submitting this email via a @linux.microsoft.com account.
> 
> I can't tell if this is a hobby for you or a work project, either way, the rules are the
> same and they still apply.
> 

This is part of my work at Microsoft.

> I am TO'ing Ian Bearman and Roman Kiselev to see if they can assist you with this internally.
> 

Thanks.

> In summary:
> - If Microsoft owns the copyright of the code you write then you'll need to go through legal
>   to get a corporate assignment completed with the Free Software Foundation. This is the same
>   process that companies like Intel, IBM, Red Hat, Amazon etc. have followed for contribution.
> - If Microsoft does not own the copyright of the code you write then you'll need a personal
>   assignment.
> 
> My suggestion is talk to Ian and Roman and see if there is already a process inside Microsoft
> for this. I can tell you that Microsoft has signed an assignment for the GNU Debugger (gdb)
> just last year.
> 

OK.

> I always suggest getting a future assignment in place which allows us to accept any and all
> patches from not into the future:
> http://git.savannah.gnu.org/cgit/gnulib.git/plain/doc/Copyright/request-assign.future
> 

OK.

> I also suggest getting GCC GLIBC GDB and BINUTILS assignments all done at the same time.
> These 4 core projects constitute the core pieces of the GNU Toolchain and as such it's
> sometimes required to synchronize headers across projects. However, to start with you'll
> need GLIBC if you plan to contribute to glibc.
> 

OK.

>> Placement of the code
>> ----------------------
>>
>> I have the following files in the project:
>>
>> 	- tramp.h (contains the function prototypes of the API functions)
>> 	- tramp.c (contains the common code for creating and managing trampolines)
>> 	- tramp_arch.c (contains architecture-specific C code)
>> 	- tramp_code.S (contains architecture-specific assembly code)
> 
> OK.
>  
>> It looks like I must place the files here:
>>
>> 	sysdeps/unix/sysv/linux/tramp.c
>> 	sysdeps/unix/sysv/linux/tramp.h
> 
> 
> Sure.
> 
>> 	sysdeps/unix/sysv/linux/x86/tramp_arch.c (etc for each arch)
>> 	sysdeps/unix/sysv/linux/x86/tramp_code.S (etc for each arch)
> 
> 
> Sure.
> 
>> Did I get that right?
> 
> Yes, that's reasonable.
> 

Thanks.

> You'll also need a sysdeps/unis/sysv/linux/tramp_[arch,code].[c,s]
> that implements a stub, because I expect you won't add support for all arches
> right away?
> 

Yes. I will be addressing X64, i386, ARM64 and ARM to beging with.

>> Other OSes
>> ----------
>>
>> These functions can potentially be implemented for other OSes as well. There are,
>> however, some OS-specific details such as mmap() details and traversing the list of
>> mapped objects in an address space.
>>
>> I have two options:
>>
>> 1. Implement it for Linux. In the future, when it is implemented for another OS,
>>    deal with the changes required.
> 
> This.
> 
>> 2. Implement it from the beginning as a common feature and provide OS-specific code
>>    for Linux.
> 
> This is made complicated by the fact that you have to get the design right the first
> time for Linux and Hurd.
> 
>> Which one would you recommend? If it is (2), could you point me to an example within
>> glibc that I can follow for how to organize the code for a multi-OS feature?
> 
> I recommend (1).
> 

OK. Great!

>> Documentation
>> -------------
>>
>> I do plan to submit a Linux manpage. Do I have to do that after my final patch gets accepted into
>> glibc? When exactly do I need to do that?
> 
> Documentation should be added to glibc/manual/*.texi in the appropriate place.
> 
> This includes considerations for threaded safety, signal safety, and cancellation safety.
> 

OK.

>> Self-tests
>> ----------
>>
>> I do plan to submit tests along with the patch. But do I need to submit tests for the RFC?
>> Or, is it enough for me to submit tests along with the actual patch later?
> 
> You don't need to submit tests for the RFC.
>  

OK. Thanks.

>> mailing list
>> -------------
>>
>> I will send out my RFC (and an actual patch later) to libc-alpha. Do I need to include
>> anyone else in the review? How about folks for each architecture?
> 
> The machine maintainers should be reading the list.
> 
> If there is anyone in particular you want to call out you can do so from the MAINTAINERS list:
> https://sourceware.org/glibc/wiki/MAINTAINERS
> 

OK.

>> Public header
>> -------------
>>
>> tramp.h would be a public header file. Is there any specific process I need to follow for
>> introducing a public header file?
> 
> No. You just have to convince the maintainers that the API belongs in glibc.
>  

OK.

>> Architecture support
>> --------------------
>>
>> I will be introducing support for X86 and ARM, 32-bit and 64-bit. Is there any special
>> process I need to follow if my changes contain architecture-specific changes?
>  
> You will be adding a new symbol to the ABI baseline for all architectures.
> 
> You will need to update all of the symbol baselines with `make update-all-abi` (IIRC).
> You will have to explain what other machine maintainers need to do to complete the support.
> 

OK.

> Please consider the following:
> - Can the functionality be provided by putting the code in another library? Why glibc?
> - Is it tightly coupled to PLTs? LD_AUDIT of PLTs? Will LD_AUDIT support the trampolines?
> - What do other C library authors think of the interface? You could post to libc-coord.
>   https://www.openwall.com/lists/libc-coord/2020/01/30/1
> - What are the parallelism and concurrency requirements for the interface? What happens
>   if multiple threads use the interface and what hardware atomics are required for it?
> - How does this trampoline relate to libffi's trampolines?
> - Does the trampoline require executable memory? How do you plan to get that under
>   conditions like SELinux?
> 


Yes. I am thinking about all of these things.

> I'm also TO'ing Florian Weimer who has in the past suggested some kind of
> trampoline creation interfaces for the dynamic loader.
> 

I had originally sent out an RFC to solve the problem with a kernel API. Florian
was one of the reviewers. It was agreed that a user-level solution in a library
is preferred. Hence, this project.

Again, thanks for the information. I will study this more.

Madhavan

Re: Questions on glibc submission for a new project

[Madhavan T. Venkataraman]

Sorry for the delay.

So, after thinking about this a little, I think libffi is the right place for this work.
I am discussing the possibility with Florian.

But thanks for all the help and information. It will definitely be useful in the future
when I want to submit something to glibc.

Madhavan

On 10/15/20 3:22 PM, Carlos O'Donell wrote:
> On 10/15/20 3:05 PM, Madhavan T. Venkataraman via Libc-help wrote:
>> Hi all,
>>
>> I am planning to submit an RFC for a glibc project. This is my first time. So, I have some
>> questions.
> 
> Glad to hear that!
> 
> I look forward to reading and reviewing the RFC.
>  
>> Description of the project
>> --------------------------
>>
>> I am developing some API functions to create and use trampolines so that applications and
>> libraries don't have to each implement them (often using dynamic code).
>>
>> 	void	*tramp_alloc(int flags);
>> 	void	tramp_set_parms(void *tramp, void *code, void *data);
>> 	void	*tramp_get_addr(void *tramp);
>> 	void	tramp_free(void *tramp);
>> 	int	tramp_supported(void);
>>
>> I will provide a full justification in the RFC. For now, I just have some questions.
> 
> No problem.
> 
>> Copyright assignment
>> --------------------
>>
>> I am developing these functions under Linux. I see that the files under sysdeps/unix/sysv/linux
>> have a "GNU Lesser General Public License". It appears that I should just use that and not need
>> to apply for a new FSF Copyright assignment.
>>
>> Am I correct in my assumption?
> 
> Please have a read through the "Contribution checklist"
> https://sourceware.org/glibc/wiki/Contribution%20checklist
> 
>> If I need to apply for something, could you please advise me on what exactly I need to do?
> 
> I see you are submitting this email via a @linux.microsoft.com account.
> 
> I can't tell if this is a hobby for you or a work project, either way, the rules are the
> same and they still apply.
> 
> I am TO'ing Ian Bearman and Roman Kiselev to see if they can assist you with this internally.
> 
> In summary:
> - If Microsoft owns the copyright of the code you write then you'll need to go through legal
>   to get a corporate assignment completed with the Free Software Foundation. This is the same
>   process that companies like Intel, IBM, Red Hat, Amazon etc. have followed for contribution.
> - If Microsoft does not own the copyright of the code you write then you'll need a personal
>   assignment.
> 
> My suggestion is talk to Ian and Roman and see if there is already a process inside Microsoft
> for this. I can tell you that Microsoft has signed an assignment for the GNU Debugger (gdb)
> just last year.
> 
> I always suggest getting a future assignment in place which allows us to accept any and all
> patches from not into the future:
> http://git.savannah.gnu.org/cgit/gnulib.git/plain/doc/Copyright/request-assign.future
> 
> I also suggest getting GCC GLIBC GDB and BINUTILS assignments all done at the same time.
> These 4 core projects constitute the core pieces of the GNU Toolchain and as such it's
> sometimes required to synchronize headers across projects. However, to start with you'll
> need GLIBC if you plan to contribute to glibc.
> 
>> Placement of the code
>> ----------------------
>>
>> I have the following files in the project:
>>
>> 	- tramp.h (contains the function prototypes of the API functions)
>> 	- tramp.c (contains the common code for creating and managing trampolines)
>> 	- tramp_arch.c (contains architecture-specific C code)
>> 	- tramp_code.S (contains architecture-specific assembly code)
> 
> OK.
>  
>> It looks like I must place the files here:
>>
>> 	sysdeps/unix/sysv/linux/tramp.c
>> 	sysdeps/unix/sysv/linux/tramp.h
> 
> 
> Sure.
> 
>> 	sysdeps/unix/sysv/linux/x86/tramp_arch.c (etc for each arch)
>> 	sysdeps/unix/sysv/linux/x86/tramp_code.S (etc for each arch)
> 
> 
> Sure.
> 
>> Did I get that right?
> 
> Yes, that's reasonable.
> 
> You'll also need a sysdeps/unis/sysv/linux/tramp_[arch,code].[c,s]
> that implements a stub, because I expect you won't add support for all arches
> right away?
> 
>> Other OSes
>> ----------
>>
>> These functions can potentially be implemented for other OSes as well. There are,
>> however, some OS-specific details such as mmap() details and traversing the list of
>> mapped objects in an address space.
>>
>> I have two options:
>>
>> 1. Implement it for Linux. In the future, when it is implemented for another OS,
>>    deal with the changes required.
> 
> This.
> 
>> 2. Implement it from the beginning as a common feature and provide OS-specific code
>>    for Linux.
> 
> This is made complicated by the fact that you have to get the design right the first
> time for Linux and Hurd.
> 
>> Which one would you recommend? If it is (2), could you point me to an example within
>> glibc that I can follow for how to organize the code for a multi-OS feature?
> 
> I recommend (1).
> 
>> Documentation
>> -------------
>>
>> I do plan to submit a Linux manpage. Do I have to do that after my final patch gets accepted into
>> glibc? When exactly do I need to do that?
> 
> Documentation should be added to glibc/manual/*.texi in the appropriate place.
> 
> This includes considerations for threaded safety, signal safety, and cancellation safety.
> 
>> Self-tests
>> ----------
>>
>> I do plan to submit tests along with the patch. But do I need to submit tests for the RFC?
>> Or, is it enough for me to submit tests along with the actual patch later?
> 
> You don't need to submit tests for the RFC.
>  
>> mailing list
>> -------------
>>
>> I will send out my RFC (and an actual patch later) to libc-alpha. Do I need to include
>> anyone else in the review? How about folks for each architecture?
> 
> The machine maintainers should be reading the list.
> 
> If there is anyone in particular you want to call out you can do so from the MAINTAINERS list:
> https://sourceware.org/glibc/wiki/MAINTAINERS
> 
>> Public header
>> -------------
>>
>> tramp.h would be a public header file. Is there any specific process I need to follow for
>> introducing a public header file?
> 
> No. You just have to convince the maintainers that the API belongs in glibc.
>  
>> Architecture support
>> --------------------
>>
>> I will be introducing support for X86 and ARM, 32-bit and 64-bit. Is there any special
>> process I need to follow if my changes contain architecture-specific changes?
>  
> You will be adding a new symbol to the ABI baseline for all architectures.
> 
> You will need to update all of the symbol baselines with `make update-all-abi` (IIRC).
> You will have to explain what other machine maintainers need to do to complete the support.
> 
> Please consider the following:
> - Can the functionality be provided by putting the code in another library? Why glibc?
> - Is it tightly coupled to PLTs? LD_AUDIT of PLTs? Will LD_AUDIT support the trampolines?
> - What do other C library authors think of the interface? You could post to libc-coord.
>   https://www.openwall.com/lists/libc-coord/2020/01/30/1
> - What are the parallelism and concurrency requirements for the interface? What happens
>   if multiple threads use the interface and what hardware atomics are required for it?
> - How does this trampoline relate to libffi's trampolines?
> - Does the trampoline require executable memory? How do you plan to get that under
>   conditions like SELinux?
> 
> I'm also TO'ing Florian Weimer who has in the past suggested some kind of
> trampoline creation interfaces for the dynamic loader.
>