A problem while building xapian-bindindgs for python 3.6 on i686

A problem while building xapian-bindindgs for python 3.6 on i686

[Lemures Lemniscati]

Hi!

I'm trying to build xapian-core and xapian-bindings 1.4.18 with cygport
files:
https://github.com/cygwin-lem/xapian-core-cygport/tree/n_1.4.18-0.1
https://github.com/cygwin-lem/xapian-bindings-cygport/tree/n_1.4.18-0.1

But, on i686, I've failed to build xapian-bindings for python 3.6.


----

I've built xapian-core 1.4.18 successfully on x86_64 and on i686.
And on x86_64, I've succefully built xapian-bindings.


On i686, I've succefully built xapian-bindings for python 3.7 and 3.8, 
but failed to build xapian-bindings for python 3.6 with a message
'*** stack smashing detected ***: terminated'.

https://github.com/cygwin-lem/xapian-bindings-cygport/blob/n_1.4.18-0.1_log/xapian-bindings-1.4.18-0.1.i686/log/xapian-bindings-1.4.18-0.1-compile.log#L1618


I've no idea what is wrong.
Please, help me.

Lem

Re: A problem while building xapian-bindindgs for python 3.6 on i686

[Achim Gratz]

Lemures Lemniscati via Cygwin-apps writes:
> On i686, I've succefully built xapian-bindings for python 3.7 and 3.8, 
> but failed to build xapian-bindings for python 3.6 with a message
> '*** stack smashing detected ***: terminated'.

You might need to do an (ephemeral) rebase on the just-built DLL.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf rackAttack:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: A problem while building xapian-bindindgs for python 3.6 on i686

[Lemures Lemniscati]

On Mon, 12 Apr 2021 20:13:57 +0200, Achim Gratz
> Lemures Lemniscati via Cygwin-apps writes:
> > On i686, I've succefully built xapian-bindings for python 3.7 and 3.8, 
> > but failed to build xapian-bindings for python 3.6 with a message
> > '*** stack smashing detected ***: terminated'.
> 
> You might need to do an (ephemeral) rebase on the just-built DLL.
> 

Thank you, Achim.

I tried it by 'rebase -s $@'
https://github.com/cygwin-lem/xapian-bindings-cygport/commit/0c50eee43e018dcaaea6b27b147790e437c85c14

But, sadly, no improvement.
https://github.com/cygwin-lem/xapian-bindings-cygport/blob/166fbc8c7f092f71a68d5c186dad9dc9640aefb1/xapian-bindings-1.4.18-0.1.i686/log/xapian-bindings-1.4.18-0.1-compile.log#L233

Any advices are welcome.
I still have no idea.

Regards,

Lem

Re: A problem while building xapian-bindindgs for python 3.6 on i686

[ASSI]

Lemures Lemniscati via Cygwin-apps writes:
> I tried it by 'rebase -s $@'

Make that 'rebase -sO $@' since you don't want to rebase the system DLL.

> But, sadly, no improvement.

Well, you'd generally start with figuring out where the error message
comes from and then try to work your way back as to why the conditions
arise that trigger it.  Also, I have at least one build (maxima) where I
need to increase the Python heap size so it won't crash, so if Xapian
uses lots of heap it might run into a problem as well.  I don't know the
layout of where stack and heap are located and if the heap could grow
into the stack (which would explain the error, but it might have an
entirely different cause).


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Q+, Q and microQ:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: A problem while building xapian-bindindgs for python 3.6 on i686

[Lemures Lemniscati]

On Thu, 15 Apr 2021 08:13:17 +0200, ASSI
> Lemures Lemniscati via Cygwin-apps writes:
> > I tried it by 'rebase -s $@'
> 
> Make that 'rebase -sO $@' since you don't want to rebase the system DLL.
> 
> > But, sadly, no improvement.
> 
> Well, you'd generally start with figuring out where the error message
> comes from and then try to work your way back as to why the conditions
> arise that trigger it.  Also, I have at least one build (maxima) where I
> need to increase the Python heap size so it won't crash, so if Xapian
> uses lots of heap it might run into a problem as well.  I don't know the
> layout of where stack and heap are located and if the heap could grow
> into the stack (which would explain the error, but it might have an
> entirely different cause).

Thank you Achim.

(1) Debugging stack and heap seems too much for me now...

(2) The situation occurs at testing to load just-built modules
	/usr/bin/python3.6 -c 'import xapian'
	/usr/bin/python3.6 -O -c 'import xapian'

And weirdly, only first runs will crash.

That is, just after failure of building, these commands will
successfully exit, in an appropriate working directory
(xapian-bindings-1.4.18-0.1.i686/build/python3.6/python3/).

(3) So I've added dummy runs just after building and rebasing modules,
like this [1]:

--- origsrc/xapian-bindings-1.4.18/python3/Makefile.am
+++ src/xapian-bindings-1.4.18/python3/Makefile.am
@@ -92,6 +92,9 @@
 ## ksh requires a path on the sourced file.
 	. ./libtoolconfig.tmp; cp $$objdir/_xapian$(PYTHON3_SO) xapian
 	rm -f libtoolconfig.tmp
+	rebase -sO $@
+	-$(PYTHON3) -c 'import xapian'    # Without this, weirdly, it might fail to build for python 3.6 on i686
+	-$(PYTHON3) -O -c 'import xapian' # Without this, weirdly, it might fail to build for python 3.6 on i686
 
 CLEANFILES = \
     xapian/_xapian$(PYTHON3_SO) \

[1]: https://github.com/cygwin-lem/xapian-bindings-cygport/commit/ef980c028c1018f27c13e66b0cf682bd35b16a82#diff-d0fb27992df3bfdc93dcef2469e6378d62dda513c730b4872ef25dd80199a63bR7-R9

(4) With this patch, modules for python 3.6 on i686 will be built
successfully.
    Although both two dummy runs in [1] are expected to crash,
    they don't crash, weirdly.


Regards,

Lem

Re: A problem while building xapian-bindindgs for python 3.6 on i686

[Achim Gratz]

Lemures Lemniscati via Cygwin-apps writes:
> (2) The situation occurs at testing to load just-built modules
> 	/usr/bin/python3.6 -c 'import xapian'
> 	/usr/bin/python3.6 -O -c 'import xapian'
>
> And weirdly, only first runs will crash.

That symptom indicates that probably BLODA injects itself into the image
when it detectes a new DLL and then stops doing that when it has
determined that DLL to be benign.  For many reasons you should exclude
your build directories from getting subjected to that sort of behaviour
(if you can do that), this is just one of them.

> (4) With this patch, modules for python 3.6 on i686 will be built
> successfully.
>     Although both two dummy runs in [1] are expected to crash,
>     they don't crash, weirdly.

AS long as the workaround gets you a successful build I'd not worry too
much about it.  Just document somehwere that it's a workaround and what
problem it was trying to solve.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: A problem while building xapian-bindindgs for python 3.6 on i686

[Lemures Lemniscati]

On Sat, 17 Apr 2021 15:54:38 +0200, Achim Gratz
> Lemures Lemniscati via Cygwin-apps writes:
> > (2) The situation occurs at testing to load just-built modules
> > 	/usr/bin/python3.6 -c 'import xapian'
> > 	/usr/bin/python3.6 -O -c 'import xapian'
> >
> > And weirdly, only first runs will crash.
> 
> That symptom indicates that probably BLODA injects itself into the image
> when it detectes a new DLL and then stops doing that when it has
> determined that DLL to be benign.  For many reasons you should exclude
> your build directories from getting subjected to that sort of behaviour
> (if you can do that), this is just one of them.
> 
> > (4) With this patch, modules for python 3.6 on i686 will be built
> > successfully.
> >     Although both two dummy runs in [1] are expected to crash,
> >     they don't crash, weirdly.
> 
> AS long as the workaround gets you a successful build I'd not worry too
> much about it.  Just document somehwere that it's a workaround and what
> problem it was trying to solve.

Thank you, Achim.

Now, I've prepared cygport files for xapian-core and xapian-bindings:

  https://github.com/cygwin-lem/xapian-core-cygport/tree/n_1.4.18-1
  https://github.com/cygwin-lem/xapian-bindings-cygport/tree/n_1.4.18-1

And their test packages
  (added python 3.8 modules, dropped python 2.7 modules):

  https://cygwin-lem.github.io/xapian-core-cygport/
  https://cygwin-lem.github.io/xapian-bindings-cygport/

Regards,

Lem