Using Py_SetPythonHome

Using Py_SetPythonHome

[Meador Inge]

Hi All,

I noticed in Nov 2010 Joel proposed adding a call to Py_SetPythonHome in this
thread [1].  However, it seems that got dropped.  I would like to bring this
topic up again.

The use case I am having trouble with is the one where gdb gets shipped with:
(1) its own pythonxy.so, (2) a zipped standard library, and (3) a directory for
the standard library module shared libraries.  In this case, Py_SetPythonHome
needs to be called before Py_Initialize so that the standard library pieces can
even be found (Py_Initialize does some imports).

So, I was playing around with something like what Joel proposed, but before
Py_Initiliaze:


diff --git a/gdb/python/python.c b/gdb/python/python.c
index 482f53e..fa86e61 100644
--- a/gdb/python/python.c
+++ b/gdb/python/python.c
@@ -1353,6 +1353,7 @@ message == an error message without a stack will be printe
      This must be done before calling Py_Initialize.  */
   Py_SetProgramName (concat (ldirname (python_libdir), SLASH_STRING, "bin",
                             SLASH_STRING, "python", NULL));
+  Py_SetPythonHome (ldirname (python_libdir));
 #endif

   Py_Initialize ();


Is there another way around the problem I am having?  Have the feelings
concerning Py_SetPythonHome changed since comments in [1] ?

[1] http://sources.redhat.com/ml/gdb-patches/2010-11/msg00328.html

-- 
Meador Inge
CodeSourcery / Mentor Embedded
http://www.mentor.com/embedded-software

Re: Using Py_SetPythonHome

[Joel Brobecker]

> The use case I am having trouble with is the one where gdb gets
> shipped with: (1) its own pythonxy.so, (2) a zipped standard library,
> and (3) a directory for the standard library module shared libraries.
> In this case, Py_SetPythonHome needs to be called before Py_Initialize
> so that the standard library pieces can even be found (Py_Initialize
> does some imports).

Not sure what (2) and (3) are...

> Is there another way around the problem I am having?  Have the feelings
> concerning Py_SetPythonHome changed since comments in [1] ?
> 
> [1] http://sources.redhat.com/ml/gdb-patches/2010-11/msg00328.html

... The proposed patch has been working flawlessly for us since
I installed it in AdaCore's tree, but as far as I know, there has
been no change of heart regarding its inclusion in the FSF tree.
I believe Jan contacted the Python developers, and their answer
seemed to go in Jan's direction.

-- 
Joel

Re: Using Py_SetPythonHome

[Jan Kratochvil]

On Mon, 17 Sep 2012 19:06:57 +0200, Joel Brobecker wrote:
> I believe Jan contacted the Python developers, and their answer
> seemed to go in Jan's direction.

Sorry for not replying it myself.

The thread mails in each month:
	[RFA] ignore PYTHONHOME environment variable.
	http://sourceware.org/ml/gdb-patches/2010-11/msg00328.html
	http://sourceware.org/ml/gdb-patches/2010-12/msg00214.html
	http://sourceware.org/ml/gdb-patches/2011-01/msg00307.html
	http://sourceware.org/ml/gdb-patches/2012-05/msg00710.html

and the last one with Python upstream decision:
	http://sourceware.org/ml/gdb-patches/2012-05/msg01041.html


Regards,
Jan

Re: Using Py_SetPythonHome

[Doug Evans]

On Mon, Sep 17, 2012 at 10:46 AM, Jan Kratochvil
<jan.kratochvil@redhat.com> wrote:
> On Mon, 17 Sep 2012 19:06:57 +0200, Joel Brobecker wrote:
>> I believe Jan contacted the Python developers, and their answer
>> seemed to go in Jan's direction.
>
> Sorry for not replying it myself.
>
> The thread mails in each month:
>         [RFA] ignore PYTHONHOME environment variable.
>         http://sourceware.org/ml/gdb-patches/2010-11/msg00328.html
>         http://sourceware.org/ml/gdb-patches/2010-12/msg00214.html
>         http://sourceware.org/ml/gdb-patches/2011-01/msg00307.html
>         http://sourceware.org/ml/gdb-patches/2012-05/msg00710.html
>
> and the last one with Python upstream decision:
>         http://sourceware.org/ml/gdb-patches/2012-05/msg01041.html

It's not clear to me that upstream has made a decision (or that they
sufficiently understand our use-case).

btw, for reference sake, I'm not opposed to [1].
http://sources.redhat.com/ml/gdb-patches/2010-11/msg00328.html

If we need to provide some flexibility (via GDB_PYTHONHOME or some
command-line option) we could do so later when the need is justified.

Re: Using Py_SetPythonHome

[Jan Kratochvil]

On Tue, 18 Sep 2012 21:37:47 +0200, Doug Evans wrote:
> On Mon, Sep 17, 2012 at 10:46 AM, Jan Kratochvil <jan.kratochvil@redhat.com> wrote:
> > On Mon, 17 Sep 2012 19:06:57 +0200, Joel Brobecker wrote:
> > and the last one with Python upstream decision:
> >         http://sourceware.org/ml/gdb-patches/2012-05/msg01041.html
> 
> It's not clear to me that upstream has made a decision (or that they
> sufficiently understand our use-case).

It is important to make clear what is "our use-case".  My use-case is standard
GNU/Linux distribution.  Joel's use-case is separate unconfigured directory
with bundled various libraries incl. Python.  Not sure which one is yours.

I believe the default configuration should be suitable for GNU/Linux
distribution and not for bundled uninstalled pack of files.  We probably do
not agree in this item I guess.

I (plus all GNU/Linux distros should) already have to disable
relocate_gdb_directory to make 'gdb' suitable for GNU/Linux distribution:
	http://pkgs.fedoraproject.org/cgit/gdb.git/commit/?id=a5cb59e730e456e3dbf1a834027453e282046b45

For the bundled uninstalled pack of files I find OK to use Py_SetPythonHome
although only in the case $PYTHONHOME is not set.  If $PYTHONHOME is set then
GDB should not override it.  This override is what was decided in:
	http://bugs.python.org/issue14956#msg161959

There should be decided - also for relocate_gdb_directory:
 * What is the target installation layout for: ./configure; make; make install
 * Whether there are other installation layouts which should be made available
   by some: ./configure --enable-xxx=yyy; make; make install
 * For Python: upstream mentions some virtualenv/venv which should
   reportedly obsolete all the Python-specific hacks we discuss here.
   http://bugs.python.org/issue14956
   But that also means Python-3.x requirement which GDB is not compatible with.


Regards,
Jan

Re: Using Py_SetPythonHome

[Joel Brobecker]

> For the bundled uninstalled pack of files I find OK to use Py_SetPythonHome
> although only in the case $PYTHONHOME is not set.  If $PYTHONHOME is set then
> GDB should not override it.  This override is what was decided in:
> 	http://bugs.python.org/issue14956#msg161959

FWIW, I read the discussion, and I find it inconclusive in the sense
that they seem to lean towards letting each application decide for
themselves what to do. Certainly, I am still in favor of the proposed
patch, but IIRC, this is causing problems for you (?). This is why
I dropped its inclusion in the FSF tree.

-- 
Joel

Re: Using Py_SetPythonHome

[Jan Kratochvil]

On Fri, 21 Sep 2012 17:36:45 +0200, Joel Brobecker wrote:
> > For the bundled uninstalled pack of files I find OK to use Py_SetPythonHome
> > although only in the case $PYTHONHOME is not set.  If $PYTHONHOME is set then
> > GDB should not override it.  This override is what was decided in:
> > 	http://bugs.python.org/issue14956#msg161959
> 
> FWIW, I read the discussion, and I find it inconclusive in the sense
> that they seem to lean towards letting each application decide for
> themselves what to do. Certainly, I am still in favor of the proposed
> patch, but IIRC, this is causing problems for you (?). This is why
> I dropped its inclusion in the FSF tree.

I cannot answer Py_SetPythonHome in standard build before I find out what is
the meaning of relocate_gdb_directory in standard build.  Py_SetPythonHome
AFAIK does not have a meaning without relocate_gdb_directory and
relocate_gdb_directory is IMO already incorrect.

This is creating OS-in-an-application designs reimplementing what OS should do.


Regards,
Jan

Re: Using Py_SetPythonHome

[Joel Brobecker]

> I cannot answer Py_SetPythonHome in standard build before I find out what is
> the meaning of relocate_gdb_directory in standard build.  Py_SetPythonHome
> AFAIK does not have a meaning without relocate_gdb_directory and
> relocate_gdb_directory is IMO already incorrect.

The question is: should PYTHONHOME override the work of
relocate_gdb_directory? My answer is: No. Auxillary answer: We can
create a GDB_PYTHONHOME if needed, too.

> This is creating OS-in-an-application designs reimplementing what OS
> should do.

Really? You keep dismissing the model where everything is managed by
a GNU/Linux distribution, and this has been a regular source of
fustration.  Some users, myself included (and I am not speaking as
an AdaCore employee), routinely install software from sources, and
I keep everything separate from the system installs, because it's
easier to manage.  Are you going to deny that this is a legitimate
situation?

-- 
Joel

Re: Using Py_SetPythonHome

[Jan Kratochvil]

On Fri, 21 Sep 2012 17:57:58 +0200, Joel Brobecker wrote:
> > This is creating OS-in-an-application designs reimplementing what OS
> > should do.
> 
> Really? You keep dismissing the model where everything is managed by
> a GNU/Linux distribution, and this has been a regular source of
> fustration.  Some users, myself included (and I am not speaking as
> an AdaCore employee), routinely install software from sources, and
> I keep everything separate from the system installs, because it's
> easier to manage.  Are you going to deny that this is a legitimate
> situation?

It may be a perfectly legitimate situation, I do not argue with this part.
I argue that no other system application does what GDB does.  If some changes
should be done they should be done in a system, not in one application.

The question is whether such behavior is at least native for GNU Toolchain
which has various specifics.

I have discovered now that default build of GCC really uses something like
relocate_gdb_directory for its own components (like cc1), it does not use
constant compiled-in path like normal applications do.  This justifies
relocate_gdb_directory for the default build of GDB, it was not clear to me.
This also means GCC is violating standard system packaging practices.

But gcj still loads external dependency --with-ecj-jar=/path/to/eclipse-ecj.jar
from its original location, without any relocation:

2833  stat("/home/jkratoch/redhat/gcchead2-root-moved/bin/../lib/gcc/../../share/java/libgcj-4.8.0.jar", {st_mode=S_IFREG|0644, st_size=10619110, ...}) = 0
2834  execve("/home/jkratoch/redhat/gcchead2-root-moved/bin/../libexec/gcc/x86_64-unknown-linux-gnu/4.8.0/ecj1", ...], [/* 54 vars */] <unfinished ...>
 - relocated
2834  access("/home/jkratoch/redhat/gcchead2-root/eclipse-ecj.jar", F_OK) = -1 ENOENT (No such file or directory)
 - not relocated

Is this gcj behavior considered a bug or not?  eclipse-ecj.jar is external
(==part of a different system rpm/deb) to GCC like Python is external to GDB.


I did mind about relocate_gdb_directory, when that part seems native to GNU
Toolchain I leave the Python configuration to Python maintainers.  There is
already an IMO-incorrect part of GDB setting Py_SetProgramName so
Py_SetPythonHome may be a part of it, I do not know Python to say more.
Python itself is already violating packaging standards (single directory for
both source .py and arch-dependent .pyc/pyo, incorrect use of -rdynamic, there
may be more) so I understand applications using Python may also violate it.



Regards,
Jan

Re: Using Py_SetPythonHome

[Joel Brobecker]

> I have discovered now that default build of GCC really uses something like
> relocate_gdb_directory for its own components (like cc1), it does not use
> constant compiled-in path like normal applications do.  This justifies
> relocate_gdb_directory for the default build of GDB, it was not clear to me.
> This also means GCC is violating standard system packaging practices.

I am starting to see where you are coming from. GCC has had this feature
longer than GDB, IIRC. And to me, this is not violating "standard system
packagainng practices" (please point me to a document that would claim
to explain what the standard practices are), but providing a very
important feature, which, if not standard, definitely should be (IMO).
If I have built a tool somewhere with a given prefix, and then I want
to install a copy somewhere else, it would be a real pain in the neck
to have to rebuild it.

Going back to the actual subject of this discussion, would it cause
a problem to call Py_SetPythonHome in your situation where everything
is static and installed at the default location?

-- 
Joel

Re: Using Py_SetPythonHome

[Jan Kratochvil]

On Tue, 02 Oct 2012 15:08:54 +0200, Joel Brobecker wrote:
> And to me, this is not violating "standard system
> packagainng practices" (please point me to a document that would claim
> to explain what the standard practices are),

I follow
	http://fedoraproject.org/wiki/Packaging:Guidelines
	http://fedoraproject.org/wiki/Packaging:Guidelines#Beware_of_Rpath
which is sure Fedora specific but in this part it can be considered as
a general GNU/Linux behavior, I am willing to discuss packaging guidelines of
other distros.


> but providing a very
> important feature, which, if not standard, definitely should be (IMO).

I really do not say anything whether this feature should be standard or not.
Maybe it should be standard, indeed.  That is completely offtopic here.

Show me any common GNU/Linux distro with >= 1% of packages using this feature.
There is not any.  Which proves to me it is not a standard feature.

If it should be a standard feature:
 * These relocation functions should be moved as a GNU extension to glibc.
 * GNU Coding Standards should be extended for it.
 * etc.
I do not see any attempt to do this part, which is what I find inappropriate
on this feature.


> If I have built a tool somewhere with a given prefix, and then I want
> to install a copy somewhere else, it would be a real pain in the neck
> to have to rebuild it.

There already exist various methods to deal with it, a standardized one is
/usr/sbin/alternatives (chkconfig package) which requires no specific upstream
support from a package.  Then there are per-package hacks like ENV variables
documented in a man page but they are still more clear than this surprising
relocation.


> Going back to the actual subject of this discussion, would it cause
> a problem to call Py_SetPythonHome in your situation where everything
> is static and installed at the default location?

Yes, it is a problem because 99.9% of other Python-using packages behave
differently.


Thanks,
Jan

Re: Using Py_SetPythonHome

[Jan Kratochvil]

On Wed, 03 Oct 2012 17:12:44 +0200, Jan Kratochvil wrote:
> On Tue, 02 Oct 2012 15:08:54 +0200, Joel Brobecker wrote:
> > Going back to the actual subject of this discussion, would it cause
> > a problem to call Py_SetPythonHome in your situation where everything
> > is static and installed at the default location?
> 
> Yes, it is a problem because 99.9% of other Python-using packages behave
> differently.

That is I do not mind anymore as it just fits the current GDB relocation
feature already in place.  I am just against this GDB relocation feature.
But that is also off-topic here, this mail thread discusses only the
Py_SetPythonHome part.


Thanks,
Jan

Re: Using Py_SetPythonHome

[Joel Brobecker]

> Show me any common GNU/Linux distro [...]

There lies the problem. You are being focused on distros, whereas I am
saying that we should cater to the needs of people who do not provide
a distribution, but just a binary package. This is why we disagree
so strongly. You need to understand that there are uses of Free
Software other than distribution-provided binaries.

It may be a lot less common today, compared to 15 years ago, but to
this day, I still build a lot of software from sources, and I move it
around from time to time.

Example: For GDB development, I build specific versions of the auto
tools, as well as flex, etc. I install them at a specific location
to start somewhere in my home directory. But then, the sysadmin asked
me to move it elsewhere because it takes too much room. Should I have
to recompile everything just because the world is now distro-centric?

Should every company out there that provides binary packages deal
with the problem on their own rather than share the feature just because
it isn't a necessary feature in distro-style binaries?

Yes, it would be great if glibc dealt with it automatically for us.
But what about Solaris, HP-UX, IRIX, Windows? Right now, there is
no standard cross-platform way to deal with the problem. So each
project is on its own. Not ideal, but still a fact that we have to
deal with.

> > Going back to the actual subject of this discussion, would it cause
> > a problem to call Py_SetPythonHome in your situation where everything
> > is static and installed at the default location?
> 
> Yes, it is a problem because 99.9% of other Python-using packages behave
> differently.

With this reasoning, would people ever inovate?

-- 
Joel

Re: Using Py_SetPythonHome

[Eli Zaretskii]

> Date: Wed, 3 Oct 2012 08:38:54 -0700
> From: Joel Brobecker <brobecker@adacore.com>
> Cc: Doug Evans <dje@google.com>, Meador Inge <meadori@codesourcery.com>,	gdb@sourceware.org
> 
> It may be a lot less common today, compared to 15 years ago, but to
> this day, I still build a lot of software from sources, and I move it
> around from time to time.

So do I, FWIW.

Re: Using Py_SetPythonHome

[Jan Kratochvil]

Hi Joel,

please take this mail "with a grain of salt", although only a bit.


On Wed, 03 Oct 2012 17:38:54 +0200, Joel Brobecker wrote:
> that we should cater to the needs of people who do not provide
> a distribution, but just a binary package.

There are not any such people.


> You need to understand that there are uses of Free
> Software other than distribution-provided binaries.

It is already a history.


> I still build a lot of software from sources,

Binaries outside of package management no longer exists and they should be
deleted ASAP if found as it is both a security hole and a too expensive
software management issue.


> to start somewhere in my home directory. But then, the sysadmin asked
> me to move it elsewhere because it takes too much room.

There do not exist any multi-user systems anymore.  Each developer has her own
virtual machine (in fact many of them), therefore sure with root access and
with proper normal automatic package management there.


> Should I have
> to recompile everything just because the world is now distro-centric?

Nobody is compiling software, this is happenning automatically in build farms.


> Should every company out there that provides binary packages deal
> with the problem on their own rather than share the feature just because
> it isn't a necessary feature in distro-style binaries?

There is no problem, all files and their locations are under the control of
package management of each GNU/Linux distro.


> Yes, it would be great if glibc dealt with it automatically for us.
> But what about Solaris, HP-UX, IRIX, Windows? Right now, there is
> no standard cross-platform way to deal with the problem. So each
> project is on its own. Not ideal, but still a fact that we have to
> deal with.

Please withstand those few remaining years on those proprietary systems and do
not try to reinvent GNU/Linux package management on top of them, that has been
tried already uncountable times and it does not work.  These proprietary
systems are doomed, their missing package management is a part of this fate.


> > > Going back to the actual subject of this discussion, would it cause
> > > a problem to call Py_SetPythonHome in your situation where everything
> > > is static and installed at the default location?
> > 
> > Yes, it is a problem because 99.9% of other Python-using packages behave
> > differently.
> 
> With this reasoning, would people ever inovate?

That is a great idea.  Packaging rules changes get proposed and discussed
first at:
	http://fedoraproject.org/wiki/Packaging_Committee
Or sure an appropriate body in some other major GNU/Linux distro featuring
qualified people who can contribute to your idea.


Thanks,
Jan

Re: Using Py_SetPythonHome

[Paul_Koning]

On Oct 3, 2012, at 1:53 PM, Jan Kratochvil wrote:

> ...
> There do not exist any multi-user systems anymore.  Each developer has her own
> virtual machine (in fact many of them), therefore sure with root access and
> with proper normal automatic package management there.

That is not true.

	paul

Re: Using Py_SetPythonHome

[Eli Zaretskii]

> From: <Paul_Koning@Dell.com>
> CC: <gdb@sourceware.org>
> Date: Wed, 3 Oct 2012 17:57:12 +0000
> 
> 
> On Oct 3, 2012, at 1:53 PM, Jan Kratochvil wrote:
> 
> > ...
> > There do not exist any multi-user systems anymore.  Each developer has her own
> > virtual machine (in fact many of them), therefore sure with root access and
> > with proper normal automatic package management there.
> 
> That is not true.

Of course, it isn't.  Example: fencepost.gnu.org.

Re: Using Py_SetPythonHome

[Jan Kratochvil]

On Wed, 03 Oct 2012 20:35:45 +0200, Eli Zaretskii wrote:
> > On Oct 3, 2012, at 1:53 PM, Jan Kratochvil wrote:
> > > ...
> > > There do not exist any multi-user systems anymore.  Each developer has her own
> > > virtual machine (in fact many of them), therefore sure with root access and
> > > with proper normal automatic package management there.
> > 
> > That is not true.
> 
> Of course, it isn't.  Example: fencepost.gnu.org.

I hope you do not mean this seriously.  I understand there still exist some
such systems but they are a relict needing to be replaced.  And at least from
the systems around me (not just Red Hat) they already have been replaced.

Specifically logging into fencepost.gnu.org for checking FSF assignments is
always a pain as the interactive remote shell is slow, there are unconfigured
and missing tools there, one is anxious not to leave there accidentally some
CPU-intensive crashed processes etc.

The right solution would sure be if I could just 'git pull' an update to the
assignments database/files.


Thanks,
Jan

Re: Using Py_SetPythonHome

[Eli Zaretskii]

> Date: Wed, 3 Oct 2012 20:43:40 +0200
> From: Jan Kratochvil <jan.kratochvil@redhat.com>
> Cc: Paul_Koning@Dell.com, gdb@sourceware.org
> 
> On Wed, 03 Oct 2012 20:35:45 +0200, Eli Zaretskii wrote:
> > > On Oct 3, 2012, at 1:53 PM, Jan Kratochvil wrote:
> > > > ...
> > > > There do not exist any multi-user systems anymore.  Each developer has her own
> > > > virtual machine (in fact many of them), therefore sure with root access and
> > > > with proper normal automatic package management there.
> > > 
> > > That is not true.
> > 
> > Of course, it isn't.  Example: fencepost.gnu.org.
> 
> I hope you do not mean this seriously.

Actually, I do.

> I understand there still exist some
> such systems but they are a relict needing to be replaced.  And at least from
> the systems around me (not just Red Hat) they already have been replaced.
> 
> Specifically logging into fencepost.gnu.org for checking FSF assignments is
> always a pain as the interactive remote shell is slow, there are unconfigured
> and missing tools there, one is anxious not to leave there accidentally some
> CPU-intensive crashed processes etc.

All that might be true, but is not relevant to this discussion.  If
there's something on fencepost that needs to be improved or fixed,
mail the sysadmins.  I did that several times, and everything I need
for my work on GNU projects was fixed sooner than I expected.

> The right solution would sure be if I could just 'git pull' an update to the
> assignments database/files.

Who or what prevents you from setting up a git repo on fencepost, or
asking savannah admins to add copyright.list to their repo, or ...?

Re: Using Py_SetPythonHome

[Jan Kratochvil]

On Wed, 03 Oct 2012 20:52:37 +0200, Eli Zaretskii wrote:
> > The right solution would sure be if I could just 'git pull' an update to the
> > assignments database/files.
> 
> Who or what prevents you from setting up a git repo on fencepost, or
> asking savannah admins to add copyright.list to their repo, or ...?

 * The GIT repo would have to be used read/write by all its users.
 * GIT may not be welcome at gnu.org so I would have to learn some GNU
   equivalent of GIT.
 * Life is too short to fix every little itch, interactive SSH works somehow.

As a summary from my point of view the relocations do not make sense, I also
I understand you have reasons to keep them, I have already removed these hacks
in Fedora where they IMO do not make sense.  There only still AFAIK remains
a disagreement in what should be the default './configure; make' GDB behavior.
Maybe the current state of all deployments (except non-Fedora GNU/Linux) is OK.


Thanks,
Jan

Re: Using Py_SetPythonHome

[Eli Zaretskii]

> Date: Wed, 3 Oct 2012 21:03:10 +0200
> From: Jan Kratochvil <jan.kratochvil@redhat.com>
> Cc: Paul_Koning@Dell.com, gdb@sourceware.org
> 
>  * GIT may not be welcome at gnu.org so I would have to learn some GNU
>    equivalent of GIT.

Savannah hosts gobs of projects maintained in git repositories.  So
much for git "not being welcome" at gnu.org.

Re: Using Py_SetPythonHome

[Paul_Koning]

On Oct 3, 2012, at 2:43 PM, Jan Kratochvil wrote:

> On Wed, 03 Oct 2012 20:35:45 +0200, Eli Zaretskii wrote:
>>> On Oct 3, 2012, at 1:53 PM, Jan Kratochvil wrote:
>>>> ...
>>>> There do not exist any multi-user systems anymore.  Each developer has her own
>>>> virtual machine (in fact many of them), therefore sure with root access and
>>>> with proper normal automatic package management there.
>>> 
>>> That is not true.
>> 
>> Of course, it isn't.  Example: fencepost.gnu.org.
> 
> I hope you do not mean this seriously. 

I can't speak for Eli, but as for my own statement, yes of course I meant it seriously.

I have no idea at all how you can claim to speak authoritatively about the software development and capital equipment usage practices of software development teams worldwide.

If you want to write notes like this for comic relief, that's ok (other than not fitting the intended use of the list).  But if you are trying to derive serious conclusions, you have to start out with valid and defensible assumptions.  What you wrote does not fit that definition.

	paul

Re: Using Py_SetPythonHome

[Jan Kratochvil]

On Wed, 03 Oct 2012 21:54:17 +0200, Paul_Koning@Dell.com wrote:
> If you want to write notes like this for comic relief, that's ok (other than
> not fitting the intended use of the list).  But if you are trying to derive
> serious conclusions, you have to start out with valid and defensible
> assumptions.  What you wrote does not fit that definition.

I really do not see multi-user systems anymore.  They are used only as web
hosting services but for any serious shell use with USD 5 / month for VPS
hosting I do not see a reason for asking admins for this or that like before.
Not speaking for any specific companies here.

Thanks for the advice this may not be (yet?) everywhere.

BTW you can see I also no longer try to make any changes based on my claims as
I am aware there is no agreement for it, for whatever reason.


Thanks,
Jan

Re: Using Py_SetPythonHome

[Alfred M. Szmidt]

   > > > There do not exist any multi-user systems anymore.  Each
   > > > developer has her own virtual machine (in fact many of them),
   > > > therefore sure with root access and with proper normal
   > > > automatic package management there.
   > > 
   > > That is not true.
   > 
   > Of course, it isn't.  Example: fencepost.gnu.org.

   I hope you do not mean this seriously.  I understand there still
   exist some such systems but they are a relict needing to be
   replaced.  And at least from the systems around me (not just Red
   Hat) they already have been replaced.

I work at a place where we have 100+ multi-user systems per machine
(and hundreds of machines) that have no plans on being replaced, more
like extended and expanded; we recently finished a migration from i386
boxen to x86_64 boxen, switching OS (to RHEL from SuSE) and other
stuff.

Relocation is a dream of anyone who futzes with special deliviers for
weirdo platforms; even plain old GNU/Linux boxen.

   The right solution would sure be if I could just 'git pull' an
   update to the assignments database/files.

Setting up a simple cronjob to copy copyright into a vcs repo that you
pull from fp is quite trivial.  I have a small Makefile that copies
copyright.list (amongst other things) to my latop, and commits it to
VCS.  If you want, I can bundle up that script for you.

Cheers!

Re: Using Py_SetPythonHome

[Eli Zaretskii]

> Date: Wed, 3 Oct 2012 19:53:43 +0200
> From: Jan Kratochvil <jan.kratochvil@redhat.com>
> Cc: Doug Evans <dje@google.com>, Meador Inge <meadori@codesourcery.com>,        gdb@sourceware.org
> 
> please take this mail "with a grain of salt", although only a bit.

You too ;-)

> > that we should cater to the needs of people who do not provide
> > a distribution, but just a binary package.
> 
> There are not any such people.

You are talking to them.

> > I still build a lot of software from sources,
> 
> Binaries outside of package management no longer exists and they should be
> deleted ASAP if found as it is both a security hole and a too expensive
> software management issue.

On the contrary.  I trust binaries I produced myself much more than
some package downloaded from somewhere.

> > Yes, it would be great if glibc dealt with it automatically for us.
> > But what about Solaris, HP-UX, IRIX, Windows? Right now, there is
> > no standard cross-platform way to deal with the problem. So each
> > project is on its own. Not ideal, but still a fact that we have to
> > deal with.
> 
> Please withstand those few remaining years on those proprietary systems and do
> not try to reinvent GNU/Linux package management on top of them, that has been
> tried already uncountable times and it does not work.  These proprietary
> systems are doomed, their missing package management is a part of this fate.

I'm afraid that's a pipe dream.

I really suggest that we do not base our design decisions on such
extreme assumptions.

RE: Using Py_SetPythonHome

[Terekhov, Mikhail]

Hi Jan,

You are living on another planet for sure.

Regards,
Mikhail

P.S. The same note about "grain of salt" applies.

> -----Original Message-----
> From: gdb-owner@sourceware.org [mailto:gdb-owner@sourceware.org] On
> Behalf Of Jan Kratochvil
> Sent: Wednesday, October 03, 2012 1:54 PM
> To: Joel Brobecker
> Cc: Doug Evans; Meador Inge; gdb@sourceware.org
> Subject: Re: Using Py_SetPythonHome
> 
> Hi Joel,
> 
> please take this mail "with a grain of salt", although only a bit.
> 
> 
> On Wed, 03 Oct 2012 17:38:54 +0200, Joel Brobecker wrote:
> > that we should cater to the needs of people who do not provide
> > a distribution, but just a binary package.
> 
> There are not any such people.
> 
> 
> > You need to understand that there are uses of Free
> > Software other than distribution-provided binaries.
> 
> It is already a history.
> 
> 
> > I still build a lot of software from sources,
> 
> Binaries outside of package management no longer exists and they should be
> deleted ASAP if found as it is both a security hole and a too expensive
> software management issue.
> 
> 
> > to start somewhere in my home directory. But then, the sysadmin asked
> > me to move it elsewhere because it takes too much room.
> 
> There do not exist any multi-user systems anymore.  Each developer has her
> own
> virtual machine (in fact many of them), therefore sure with root access and
> with proper normal automatic package management there.
> 
> 
> > Should I have
> > to recompile everything just because the world is now distro-centric?
> 
> Nobody is compiling software, this is happenning automatically in build
> farms.
> 
> 
> > Should every company out there that provides binary packages deal
> > with the problem on their own rather than share the feature just because
> > it isn't a necessary feature in distro-style binaries?
> 
> There is no problem, all files and their locations are under the control of
> package management of each GNU/Linux distro.
> 
> 
> > Yes, it would be great if glibc dealt with it automatically for us.
> > But what about Solaris, HP-UX, IRIX, Windows? Right now, there is
> > no standard cross-platform way to deal with the problem. So each
> > project is on its own. Not ideal, but still a fact that we have to
> > deal with.
> 
> Please withstand those few remaining years on those proprietary systems
> and do
> not try to reinvent GNU/Linux package management on top of them, that
> has been
> tried already uncountable times and it does not work.  These proprietary
> systems are doomed, their missing package management is a part of this
> fate.
> 
> 
> > > > Going back to the actual subject of this discussion, would it cause
> > > > a problem to call Py_SetPythonHome in your situation where
> everything
> > > > is static and installed at the default location?
> > >
> > > Yes, it is a problem because 99.9% of other Python-using packages
> behave
> > > differently.
> >
> > With this reasoning, would people ever inovate?
> 
> That is a great idea.  Packaging rules changes get proposed and discussed
> first at:
> 	http://fedoraproject.org/wiki/Packaging_Committee
> Or sure an appropriate body in some other major GNU/Linux distro featuring
> qualified people who can contribute to your idea.
> 
> 
> Thanks,
> Jan

Re: Using Py_SetPythonHome

[John Gilmore]

Package management is a sinkhole, unfortunately.  The OLPC project has
unfortunately discovered that despite the great support in the GNU
tools for cross-compilation, the Fedora package management tools are
completely incapable of cross-compilation.  So now that they are
making hardware with three architectures to build software for (i386,
i686, and ARM), they need to dedicate three kinds of hardware to
building their Fedora-based releases.  They can't make an OS image on
a fast x86 machine that will install or boot on an ARM.(*)

(I think Debian/Ubuntu package managers suffer from the same problem;
they all assume they're running "native", they run package-specific
shell scripts that think they're running in the target environment,
etc.)

I recommend NOT assuming that package managers are the cat's pajamas
and that therefore we can all skip the ability to usefully build from
source.

Having seen this Py_SetPythonHome discussion drag on for what seems
months (I think it's the most frequent subject line in the mailing
list), and yet I still don't understand why y'all care, perhaps someone
should try to write up a solid proposal that explains what the hell is
going on, with pros and cons listed and generally agreed upon.  That
might help point a path to making a decision that sticks for a while.

	John

(*): They can run builds under QEMU on x86, emulating the ARM
instruction set, using a set of native ARM compilers and a full ARM
GNU/Linux virtual machine, and make the ARM builds that way.  Indeed they
do -- it's only a 2- to 3-times slowdown, which is far easier than
rewriting the package management subsystem for cross-compilation and
then getting the changes adopted "upstream" into Fedora.  And far, far
easier than building fast hardware based on an available ARM chip.

Re: Using Py_SetPythonHome

[Jan Kratochvil]

On Thu, 04 Oct 2012 09:32:50 +0200, John Gilmore wrote:
> Package management is a sinkhole, unfortunately.  The OLPC project has
> unfortunately discovered that despite the great support in the GNU
> tools for cross-compilation, the Fedora package management tools are
> completely incapable of cross-compilation.  So now that they are
> making hardware with three architectures to build software for (i386,
> i686, and ARM), they need to dedicate three kinds of hardware to
> building their Fedora-based releases.  They can't make an OS image on
> a fast x86 machine that will install or boot on an ARM.(*)
> 
> (I think Debian/Ubuntu package managers suffer from the same problem;

They have:
	http://wiki.debian.org/Multiarch


> I recommend NOT assuming that package managers are the cat's pajamas
> and that therefore we can all skip the ability to usefully build from
> source.

This was also my goal, to be able to build from source.  Currently you
"cannot", as you will build something doing unexpected things (the
relocations).


> and yet I still don't understand why y'all care,

Because GDB introduces serious change in where its files are located without
it being relevant to its debugger subject.  The same way every other package
on system could be self-relocated; it is not.  Therefore GDB also should not.

If anybody needs it for GDB it should be introduced first as a general package
relocation feature.


> perhaps someone
> should try to write up a solid proposal that explains what the hell is
> going on, with pros and cons listed and generally agreed upon.  That
> might help point a path to making a decision that sticks for a while.

$ set | grep /var/lib/mock/fedora-16-x86_64/root
$ echo foo >>/var/lib/mock/fedora-16-x86_64/root/usr/share/gdb/python/gdb/__init__.py
$ /var/lib/mock/fedora-16-x86_64/root/usr/bin/gdb
Traceback (most recent call last):
  File "<string>", line 70, in <module>
  File "<string>", line 67, in GdbSetPythonDirectory
  File "/var/lib/mock/fedora-16-x86_64/root/usr/share/gdb/python/gdb/__init__.py", line 42, in <module>
    foo

This is apparently broken, there is no reason why
/var/lib/mock/fedora-16-x86_64/root/usr/bin/gdb should touch anything under
/var/lib/mock/fedora-16-x86_64/root , no other package does so.


And this leads to the issues what everything should be self-relocated as no
other system package is self-relocated.  So then GDB tries to self-relocate
also Python etc. and it can never work completely.


Regards,
Jan

Re: Using Py_SetPythonHome

[Doug Evans]

On Thu, Oct 11, 2012 at 1:42 AM, Jan Kratochvil
<jan.kratochvil@redhat.com> wrote:
> The same way every other package
> on system could be self-relocated; it is not.  Therefore GDB also should not.

I disagree.
I've found it sufficiently useful that I do not want to see the capability go.

Re: Using Py_SetPythonHome

[Meador Inge]

On 09/17/2012 12:06 PM, Joel Brobecker wrote:

>> The use case I am having trouble with is the one where gdb gets
>> shipped with: (1) its own pythonxy.so, (2) a zipped standard library,
>> and (3) a directory for the standard library module shared libraries.
>> In this case, Py_SetPythonHome needs to be called before Py_Initialize
>> so that the standard library pieces can even be found (Py_Initialize
>> does some imports).
> 
> Not sure what (2) and (3) are...

We distribute the toolchain with a layout that looks like:

   bin/
      $cross-gdb
   lib/
      libpython2.6.so.1.0
      python2.6/
      python26.zip

Py_Initialize and the Python import machinery can handle a zipped standard
library if the the python<major><minor>.zip naming convention is followed.
However, we don't ship the actual python executable, thus the system paths
aren't computed correctly unless Py_SetPythonHome is called (hence the patch).

This is why I moved Py_SetPythonHome before Py_Initialize in my patch.

>> Is there another way around the problem I am having?  Have the feelings
>> concerning Py_SetPythonHome changed since comments in [1] ?
>>
>> [1] http://sources.redhat.com/ml/gdb-patches/2010-11/msg00328.html
> 
> ... The proposed patch has been working flawlessly for us since
> I installed it in AdaCore's tree, but as far as I know, there has
> been no change of heart regarding its inclusion in the FSF tree.
> I believe Jan contacted the Python developers, and their answer
> seemed to go in Jan's direction.

Hmmm, I missed that issue on python-bugs.  I will follow up on the Python side
in the Python tracker.

-- 
Meador Inge
CodeSourcery / Mentor Embedded
http://www.mentor.com/embedded-software

Re: Using Py_SetPythonHome

[Jan Kratochvil]

On Fri, 21 Sep 2012 17:54:47 +0200, Meador Inge wrote:
> We distribute the toolchain with a layout that looks like:
> 
>    bin/
>       $cross-gdb
>    lib/
>       libpython2.6.so.1.0
>       python2.6/
>       python26.zip

Sorry but this is very custom setup of GDB.  In such case GDB should be
configured with special configure option for it (if any, IMO it should be
rather a vendor patch).  Unaware why such custom setup should complicate usage
of GDB by all the users on regular GNU/Linux systems.  It is normal nowadays
to use various virtualizations and chroot setups (like I also do) and one does
not expect called GDB does not follow standard
PYTHONHOME/PYTHONPATH/LD_LIBRARY_PATH environment variables like all the other
system binaries do.

I believe these arguments just repeat the discussion several months old,
without any new facts brought in.


Regards,
Jan

Re: Using Py_SetPythonHome

[Doug Evans]

On Fri, Sep 21, 2012 at 9:01 AM, Jan Kratochvil
<jan.kratochvil@redhat.com> wrote:
> On Fri, 21 Sep 2012 17:54:47 +0200, Meador Inge wrote:
>> We distribute the toolchain with a layout that looks like:
>>
>>    bin/
>>       $cross-gdb
>>    lib/
>>       libpython2.6.so.1.0
>>       python2.6/
>>       python26.zip
>
> Sorry but this is very custom setup of GDB.  In such case GDB should be
> configured with special configure option for it

It doesn't feel that "custom".
I don't mind handling it by a configure option though.