From: Eric Feng <ef2648@columbia.edu>
To: David Malcolm <dmalcolm@redhat.com>
Cc: gcc@gcc.gnu.org
Subject: Re: Update and Questions on CPython Extension Module -fanalyzer plugin development
Date: Mon, 7 Aug 2023 14:31:08 -0400 [thread overview]
Message-ID: <CANGHATXT3SNuFmPHG6mGCNooxJ+GPXKJe_0OFGMJyZOvq+S9Ug@mail.gmail.com> (raw)
In-Reply-To: <1c5026021761abd92ded93950e31e4ff77ca250a.camel@redhat.com>
On Fri, Aug 4, 2023 at 6:46 PM David Malcolm <dmalcolm@redhat.com> wrote:
>
> On Fri, 2023-08-04 at 18:42 -0400, David Malcolm wrote:
> > On Fri, 2023-08-04 at 16:48 -0400, Eric Feng wrote:
> > > On Fri, Aug 4, 2023 at 11:39 AM David Malcolm <dmalcolm@redhat.com>
> > > wrote:
> > > >
> > > > On Fri, 2023-08-04 at 11:02 -0400, Eric Feng wrote:
> > > > > Hi Dave,
> > > > >
> > > > > Tests related to our plugin which depend on Python-specific
> > > > > definitions have been run by including /* { dg-options "-
> > > > > fanalyzer
> > > > > -I/usr/include/python3.9" } */. This is undoubtedly not ideal;
> > > > > is
> > > > > it
> > > > > best to approach this problem by adapting a subset of relevant
> > > > > definitions like in gil.h?
> > > >
> > > > That might be acceptable in the very short-term, but to create a
> > > > plugin
> > > > that's useful to end-user (authors of CPython extension modules)
> > > > we
> > > > want to be testing against real Python headers.
> > > >
> > > > As I understand it, https://peps.python.org/pep-0394/ allows for
> > > > distributors of Python to symlink "python3-config" in the PATH to
> > > > a
> > > > python3.X-config script (for some X).
> > > >
> > > > So on such systems running:
> > > > python3-config --includes
> > > > should emit the correct -I option. On my box it emits:
> > > >
> > > > -I/usr/include/python3.8 -I/usr/include/python3.8
> > > >
> > > >
> > > > It's more complicated, but I believe:
> > > > python3-config --cflags
> > > > should emit the build flags that C/C++ extensions ought to use
> > > > when
> > > > building. On my box this emits:
> > > >
> > > > -I/usr/include/python3.8 -I/usr/include/python3.8 -Wno-unused-
> > > > result -
> > > > Wsign-compare -O2 -g -pipe -Wall -Werror=format-security -Wp,-
> > > > D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -
> > > > fstack-
> > > > protector-strong -grecord-gcc-switches -m64 -mtune=generic -
> > > > fasynchronous-unwind-tables -fstack-clash-protection -fcf-
> > > > protection -
> > > > D_GNU_SOURCE -fPIC -fwrapv -DDYNAMIC_ANNOTATIONS_ENABLED=1 -
> > > > DNDEBUG -
> > > > O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2
> > > > -
> > > > Wp,-
> > > > D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -
> > > > grecord-
> > > > gcc-switches -m64 -mtune=generic -fasynchronous-unwind-tables -
> > > > fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -
> > > > fwrapv
> > > >
> > > > and it's likely going to vary from distribution to distribution.
> > > > Some
> > > > of those options *are* going to affect the gimple that -fanalyzer
> > > > "sees".
> > > >
> > > > Does your installation of Python have such a script?
> > > >
> > > > So in the short term you could hack in a minimal subset of the
> > > > decls/defns from Python.h, but I'd prefer it if target-
> > > > supports.exp
> > > > gained a DejaGnu directive that invokes python3-config, captures
> > > > the
> > > > result (or fails with UNSUPPORTED for systems without python3
> > > > development headers), and then adds the result to the build flags
> > > > of
> > > > the file being tested. The .exp files are implemented in Tcl,
> > > > alas;
> > > > let me know if you want help with that.
> > > >
> > > > Dave
> > > Sounds good; thanks! Following existing examples in
> > > target-supports.exp, the following works as expected in terms of
> > > extracting the build flags we are interested in.
> > >
> > > In target-supports.exp:
> > > proc check_python_flags { } {
> > > set result [remote_exec host "python3-config --cflags"]
> > > set status [lindex $result 0]
> > > if { $status == 0 } {
> > > return [lindex $result 1]
> > > } else {
> > > return "UNSUPPORTED"
> > > }
> > > }
> > >
> > > However, I'm having some trouble figuring out the specifics as to
> > > how
> > > we may add the build flags to our test cases. My intuition looks
> > > like
> > > something like the following:
> > >
> > > In plugin.exp:
> > > foreach plugin_test $plugin_test_list {
> > > if {[lindex $plugin_test 0] eq "analyzer_cpython_plugin.c"} {
> > > set python_flags [check_python_flags]
> > > if { $python_flags ne "UNSUPPORTED" } {
> > > // append $python_flags to build flags here
> > > }
> > > }
> > > ....
> > > }
> > >
> > > How might we do so?
> >
> > Good question.
> >
> > Looking at plugin.exp I see it uses plugin-test-execute, which is
> > defined in gcc/testsuite/lib/plugin-support.exp.
> >
> > Looking there, I see it attempts to build the plugin, and then if it
> > succeeds, it calls
> > dg-runtest $plugin_tests $plugin_enabling_flags $default_flags
> > where $plugin_tests is the list of source files to be compiled using
> > the plugin. So one way to do this would be to modify that code from
> > plugin.exp to pass in a different value, rather than $default_flags.
> > Though it seems hackish to special-case this.
>
> Sorry, I think I misspoke here; that line that uses $default_flags is
> from plugin-support.exp, not from plugin.exp; $default_flags is a
> global variable.
>
> So I think my 2nd approach below may be the one to try:
>
> >
> > As another way, that avoids adding special-casing to plugin.exp,
> > there's an existing directive:
> > dg-additional-options
> > implemented in gcc/testsuite/lib/gcc-defs.exp which appends options
> > to
> > the default options. Unfortunately, it works via:
> > upvar dg-extra-tool-flags extra-tool-flags
> > which is a nasty Tcl hack meaning access the local variable named
> > "dg-
> > extra-tool-flags" in *the frame above*, referring to it as "extra-
> > tool-
> > flags". (this is why I don't like Tcl)
> >
> > So I think what could be done is to invoke your "check_python_flags"
> > test as a directive from the test case, so that in target-
> > supports.exp
> > you'd have something like:
> >
> > proc dg-require-python-h {} {
> >
> > which could do the invocation/output-capture of python3-config, and
> > would also have code similar to that in dg-additional-options to
> > append
> > to the options (or it could possibly just call dg-additional-options
> > provided there's an "upvar" before the callsite to make the nested
> > stack manipulation work).
> >
> > The individual test cases could then have:
> >
> > /* { dg-require-python-h } */
> >
> > in them.
> >
> > That way the Tcl stack at the point where the new directive runs
> > should
> > be similar enough to how dg-additional-options gets run for similar
> > option-injection code to work (yuck!).
Gotcha, thanks for the tip! I've been trying to make this approach
work, but despite trying many things just having /* {
dg-require-python-h } */ in the relevant test cases does not seem to
invoke proc dg-require-python-h {} { ... } in target-supports.exp. Am
I missing something else? Do we need to also "register" it somewhere
else for it to recognize the command? For context, previously I was
able to see the results of "check_python_flags" (i.e the output of
python3-config) by invoking it in plugin.exp.
> >
> > Maybe someone else on the list can see a less hackish way to get this
> > to work?
> >
> > Let me know if any of the above is unclear.
> > Dave
> >
> >
> >
> >
> >
> >
> > > >
> > > >
> > > > >
> > > > > Best,
> > > > > Eric
> > > > >
> > > > > On Tue, Aug 1, 2023 at 1:06 PM David Malcolm
> > > > > <dmalcolm@redhat.com>
> > > > > wrote:
> > > > > >
> > > > > > On Tue, 2023-08-01 at 09:57 -0400, Eric Feng wrote:
> > > > > > > >
> > > > > > > > My guess is that you were trying to do it from the
> > > > > > > > PLUGIN_ANALYZER_INIT
> > > > > > > > hook rather than from the plugin_init function, but it's
> > > > > > > > hard
> > > > > > > > to be
> > > > > > > > sure without seeing the code.
> > > > > > > >
> > > > > > >
> > > > > > > Thanks Dave, you are entirely right — I made the mistake of
> > > > > > > trying to
> > > > > > > do it from PLUGIN_ANALYZER_INIT hook and not from the
> > > > > > > plugin_init
> > > > > > > function. After following your suggestion, the callbacks
> > > > > > > are
> > > > > > > getting
> > > > > > > registered as expected.
> > > > > >
> > > > > > Ah, good.
> > > > > >
> > > > > > > I submitted a patch to review for this feature
> > > > > > > on gcc-patches; please let me know if it looks OK.
> > > > > >
> > > > > > Thanks Eric; I've posted a reply to your email there, so
> > > > > > let's
> > > > > > discuss
> > > > > > the details there.
> > > > > >
> > > > > > Dave
> > > > > >
> > > > >
> > > >
> > >
> >
>
next prev parent reply other threads:[~2023-08-07 18:31 UTC|newest]
Thread overview: 50+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-07-25 4:49 Eric Feng
2023-07-25 14:41 ` David Malcolm
2023-07-27 22:13 ` Eric Feng
2023-07-27 22:35 ` David Malcolm
2023-07-30 17:52 ` Eric Feng
2023-07-30 23:44 ` David Malcolm
2023-08-01 13:57 ` Eric Feng
2023-08-01 17:06 ` David Malcolm
2023-08-04 15:02 ` Eric Feng
2023-08-04 15:39 ` David Malcolm
2023-08-04 20:48 ` Eric Feng
2023-08-04 22:42 ` David Malcolm
2023-08-04 22:46 ` David Malcolm
2023-08-07 18:31 ` Eric Feng [this message]
2023-08-07 23:16 ` David Malcolm
2023-08-08 16:51 ` [PATCH] WIP for dg-require-python-h [PR107646] Eric Feng
2023-08-08 18:08 ` David Malcolm
2023-08-08 18:51 ` David Malcolm
2023-08-09 19:22 ` [PATCH v2] analyzer: More features for CPython analyzer plugin [PR107646] Eric Feng
2023-08-09 21:36 ` David Malcolm
2023-08-11 17:47 ` [COMMITTED] " Eric Feng
2023-08-11 20:23 ` Eric Feng
2023-08-16 19:17 ` Update on CPython Extension Module -fanalyzer plugin development Eric Feng
2023-08-16 21:28 ` David Malcolm
2023-08-17 1:47 ` Eric Feng
2023-08-21 14:05 ` Eric Feng
2023-08-21 15:04 ` David Malcolm
2023-08-23 21:15 ` Eric Feng
2023-08-23 23:16 ` David Malcolm
2023-08-24 14:45 ` Eric Feng
2023-08-25 12:50 ` Eric Feng
2023-08-25 19:50 ` David Malcolm
2023-08-29 4:31 ` [PATCH] analyzer: implement reference count checking for CPython plugin [PR107646] Eric Feng
2023-08-29 4:35 ` Eric Feng
2023-08-29 17:28 ` Eric Feng
2023-08-29 21:14 ` David Malcolm
2023-08-30 22:15 ` Eric Feng
2023-08-31 17:01 ` David Malcolm
2023-08-31 19:09 ` Eric Feng
2023-08-31 20:19 ` David Malcolm
2023-09-01 1:25 ` Eric Feng
2023-09-01 11:57 ` David Malcolm
2023-09-05 2:13 ` [PATCH] analyzer: implement symbolic value support for CPython plugin's refcnt checker [PR107646] Eric Feng
2023-09-07 17:28 ` David Malcolm
2023-09-11 2:12 ` Eric Feng
2023-09-11 19:00 ` David Malcolm
2023-08-29 21:08 ` [PATCH] analyzer: implement reference count checking for CPython plugin [PR107646] David Malcolm
2023-09-01 2:49 ` Hans-Peter Nilsson
2023-09-01 14:51 ` David Malcolm
2023-09-01 21:07 ` Eric Feng
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CANGHATXT3SNuFmPHG6mGCNooxJ+GPXKJe_0OFGMJyZOvq+S9Ug@mail.gmail.com \
--to=ef2648@columbia.edu \
--cc=dmalcolm@redhat.com \
--cc=gcc@gcc.gnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).