How to create new mi commands via python / get current interpreter in python

How to create new mi commands via python / get current interpreter in python

[Simon Sobisch]

It is possible to send `interpreter-exec mi "command"` from a python 
extension, but how can we check the _current_ interpreter a python 
create gdb command was executed in?

The reason: depending on the interpreter the return value should be 
different

interpreter-exec console "break"
--> returns user output (which may be parsed)

vs.
interpreter-exec mi "-break-insert"
--> returns interpreter output intended to be parsed


Additional to this question: is there an option to create new MI 
commands from python (instead of "console commands"), or python helper 
functions which provide better ways to return mi-messages than plain 
"print" with self-formatting?

Thanks for any insights,
Simon

Re: How to create new mi commands via python / get current interpreter in python

[Jan Vrany]

Hi,

On Tue, 2021-11-02 at 17:40 +0100, Simon Sobisch via Gdb wrote:
> Additional to this question: is there an option to create new MI 
> commands from python (instead of "console commands")

Sort of. This has been started by Didier Nadeau and then continued by
me but sadly, I got carried away and have not done the last bits so it
can be pushed. 

However, I'm using this feature heavily and keep more-or-less 
up-to-date GDB with this support (and some other hacks/fixes waiting
to be polished and submitted, see

https://github.com/janvrany/binutils-gdb/tree/users/jv%2Fvdb

Here you may find example how I use it:

https://swing.fit.cvut.cz/hg/jv-vdb/file/tip/python/vdb/__init__.py#l329

Let me know if you have any questions, let me know.

I'd like to take the opportunity and apologise to all reviewers
for not finishing it yet - hopefuly I'll find long-enough period of
quiter time to finish this.

Jan

Re: How to create new mi commands via python / get current interpreter in python

[Simon Sobisch]

That's exactly I was looking for, so sad that this isn't available in 
GDB 11 :-(
In any case that looks quite promising, would be nice to see it at least 
in a feature-branch in the official gdb repo until it is ready to be merged.

For the "hack without a patched GDB version": do you know of a good way 
to create the mi-expected message with different parameters in plain 
python and/or a way to know which interpreter is active in the current 
context of a gdb.Command?

Simon

Am 02.11.2021 um 17:56 schrieb Jan Vrany:
> Hi,
> 
> On Tue, 2021-11-02 at 17:40 +0100, Simon Sobisch via Gdb wrote:
>> Additional to this question: is there an option to create new MI
>> commands from python (instead of "console commands")
> 
> Sort of. This has been started by Didier Nadeau and then continued by
> me but sadly, I got carried away and have not done the last bits so it
> can be pushed.
> 
> However, I'm using this feature heavily and keep more-or-less
> up-to-date GDB with this support (and some other hacks/fixes waiting
> to be polished and submitted, see
> 
> https://github.com/janvrany/binutils-gdb/tree/users/jv%2Fvdb
> 
> Here you may find example how I use it:
> 
> https://swing.fit.cvut.cz/hg/jv-vdb/file/tip/python/vdb/__init__.py#l329
> 
> Let me know if you have any questions, let me know.
> 
> I'd like to take the opportunity and apologise to all reviewers
> for not finishing it yet - hopefuly I'll find long-enough period of
> quiter time to finish this.
> 
> Jan
> 
> 

Re: How to create new mi commands via python / get current interpreter in python

[Jan Vrany]

On Tue, 2021-11-02 at 20:11 +0100, Simon Sobisch via Gdb wrote:
> That's exactly I was looking for, so sad that this isn't available in
> GDB 11 :-(

I know...

> In any case that looks quite promising, would be nice to see it at
> least 
> in a feature-branch in the official gdb repo until it is ready to be
> merged.

Good idea, I can do that. I'll post here when done.

> 
> For the "hack without a patched GDB version": do you know of a good
> way 
> to create the mi-expected message with different parameters in plain 
> python and/or a way to know which interpreter is active in the
> current 
> context of a gdb.Command?
> 

I'm not sure I 100% understand what you want to achieve, but I'm afraid
one cannot tell which interprerer was active (as of now). 
But, if inside python you do "intepreter-exec mi <some custom py
command>" you may pass down to the command option - say - `--mi` no?

HTH

> Simon
> 
> Am 02.11.2021 um 17:56 schrieb Jan Vrany:
> > Hi,
> > 
> > On Tue, 2021-11-02 at 17:40 +0100, Simon Sobisch via Gdb wrote:
> > > Additional to this question: is there an option to create new MI
> > > commands from python (instead of "console commands")
> > 
> > Sort of. This has been started by Didier Nadeau and then continued
> > by
> > me but sadly, I got carried away and have not done the last bits so
> > it
> > can be pushed.
> > 
> > However, I'm using this feature heavily and keep more-or-less
> > up-to-date GDB with this support (and some other hacks/fixes
> > waiting
> > to be polished and submitted, see
> > 
> > https://github.com/janvrany/binutils-gdb/tree/users/jv%2Fvdb
> > 
> > Here you may find example how I use it:
> > 
> >  
> > https://swing.fit.cvut.cz/hg/jv-vdb/file/tip/python/vdb/__init__.py#l329
> > 
> > Let me know if you have any questions, let me know.
> > 
> > I'd like to take the opportunity and apologise to all reviewers
> > for not finishing it yet - hopefuly I'll find long-enough period of
> > quiter time to finish this.
> > 
> > Jan
> > 
> > 

How to get the full source location of a frame via python

[Simon Sobisch]

With the Python API we can do nearly everything necessary related to 
frames and
	st = frame.find_sal().symtab
	if st:
		filename = st.filename

provides the filename as noted in the debug info.

If we now need to get the full filename as used in GDB (with source path 
and similar applied) the only version I've found so far involves GDB 
text parsing:

	# Get the full path for the source file
	old_frame = gdb.selected_frame()
	frame.select()
	info_source = gdb.execute("info source", False, True)
	pattern = re.compile(r"Located in (.*)\n")
	match = re.search(pattern, info_source)
	if match:
		self.current_full_path = match.group(1)
	else:
		self.current_full_path = filename_frame	
	old_frame.select()

Which seems not very robust as it parses text which may have a different 
format in other GDB versions and likely a different text when localized; 
additional it needs a regex which is not that fast.

Questions:
Is there a direct way to get the full source location for a frame?
If not: any suggestions to improve the code (especially the reliability?)

Also: if there isn't a way via GDBs python API please consider this as a 
feature request, maybe as
	gdb.Frame.resolve_filename()

Simon

Re: How to get the full source location of a frame via python

[Jan Vrany]

On Wed, 2021-11-03 at 17:31 +0100, Simon Sobisch via Gdb wrote:
> With the Python API we can do nearly everything necessary related to 
> frames and
>         st = frame.find_sal().symtab
>         if st:
>                 filename = st.filename
> 
> provides the filename as noted in the debug info.
> 
> If we now need to get the full filename as used in GDB (with source
> path 
> and similar applied) the only version I've found so far involves GDB 
> text parsing:
> 
>         # Get the full path for the source file
>         old_frame = gdb.selected_frame()
>         frame.select()
>         info_source = gdb.execute("info source", False, True)
>         pattern = re.compile(r"Located in (.*)\n")
>         match = re.search(pattern, info_source)
>         if match:
>                 self.current_full_path = match.group(1)
>         else:
>                 self.current_full_path = filename_frame 
>         old_frame.select()
> 
> Which seems not very robust as it parses text which may have a
> different 
> format in other GDB versions and likely a different text when
> localized; 
> additional it needs a regex which is not that fast.
> 
> Questions:
> Is there a direct way to get the full source location for a frame?

By glancing at the code, it seems to me that using

  st.fullname() # instead of st.filename

should return the path printed by `info source`. Both return
fullname member from struct symtab* IIUC.

Did you try `fullname()`? 

Jan

Re: How to get the full source location of a frame via python

[Simon Sobisch]

Am 03.11.2021 um 21:07 schrieb Jan Vrany:
> On Wed, 2021-11-03 at 17:31 +0100, Simon Sobisch via Gdb wrote:
>> [...]
>> Questions:
>> Is there a direct way to get the full source location for a frame?
> 
> By glancing at the code, it seems to me that using
> 
>    st.fullname() # instead of st.filename
> 
> should return the path printed by `info source`. Both return
> fullname member from struct symtab* IIUC.
> 
> Did you try `fullname()`?
> 
> Jan

No, I overlooked this and feel a bit embarrassed as it is in the 
documentation. Works perfectly fine, both with old GDB 7.6+Python2.7 and 
  with current GDB + Python - thank you Jan!

UnicodeDecodeError on gdb.execute

[Simon Sobisch]

For some special file I need to look at the source code from within the 
GDB extension.

I did this with the reasonable and obious
	output = gdb.execute("list *" + hex(sal.pc), False, True)
(and get more lines with a follow-up "list" [not all are needed, 
otherwise the gdb.parameter("listsize") could be adjusted).

I _think_ the problem I expect now is because of a system with Python3 
which has default utf8 encoding, but it _may_ was also in before: 
There's a python exception UnicodeDecodeError in this line whenever it 
contains "extended" ascii.

"list" in GDB shows the code correctly; also
	(gdb) py gdb.execute("list 14")
shows the correct text, but as soon as python has to internally decode 
it to store a string:
	(gdb) py gdb.execute("list 14", False, True)
	Traceback (most recent call last):
	  File "<string>", line 1, in <module>
	UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfc in
	position 18: invalid start byte
	Error while executing Python code.

is there any way I could adjust the encoding used for storing 
gdb.execute as string?
Is there a reason that this isn't by default set to match 
gdb.target_charset() ?

Thanks for insights to this issue, too,
Simon

Is "forward-search" also possible case insensitive?

[Simon Sobisch]

fo and rev commands take a REGEX, with escaping like

   \d\+\s*\(SOME\|WORD\)

can we also search case-insensitive directly or is this only possible 
with character sets?

   \d\+\s*\([sS][oO][mM][eE]\|[wW][oO][rR][dD]\)

it would be nice to be able to specify a case-modifier

Is there a way to know about current "until"/"advance" execution?

[Simon Sobisch]

For a silent breakpoint the following commands are very useful

break something
command
   silent
   do_something
   continue
end

or the similar version in a python script using the silent attribute and 
continue as a command.

I've recently seen a problem with this the first time - it break any 
active "until"


break 120
command
   silent
   continue
end
break 140
until 123


will execute until 140 (with a function that just executes each line), 
while the user "intended" to go until line 123.

Similar applies to "advance".


The workaround as a user:
instead of using "until": create a temporary maintenance breakpoint in 
line 123 and "continue".
... but as the break 120 in the sample above is scripted and not 
executed by a user this cannot be applied.

Questions:
* Is there a more direct way to "preserve" the running "until"/"advance" 
in GDB scripting or with Python?

* Is there a way in Python to know about the current active "until" (so 
this can be re-done instead of "continue"/"until" in the above example)?

* If there isn't an option for this, can you please consider this as a 
Feature Request for the Python API of GDB?

Thanks,
Simon

Re: How to create new mi commands via python / get current interpreter in python

[Jan Vrany]

> 
> > In any case that looks quite promising, would be nice to see it at
> > least 
> > in a feature-branch in the official gdb repo until it is ready to
> > be
> > merged.
> 
> Good idea, I can do that. I'll post here when done.
> 
> 

Done. Rebased on a recent master and pushed into branch:
"users/jv/patches/feature-python-mi":

https://sourceware.org/git/?p=binutils-gdb.git;a=shortlog;h=refs/heads/users/jv/patches/feature-python-mi

HTH, Jan

feature branch for creating MI commands with python (was: How to create new mi commands via python / get current interpreter in python)

[Simon Sobisch]

Am 05.11.2021 um 19:51 schrieb Jan Vrany:
>>
>>> In any case that looks quite promising, would be nice to see it at
>>> least
>>> in a feature-branch in the official gdb repo until it is ready to
>>> be
>>> merged.
>>
>> Good idea, I can do that. I'll post here when done.
>>
>>
> 
> Done. Rebased on a recent master and pushed into branch:
> "users/jv/patches/feature-python-mi":
> 
> https://sourceware.org/git/?p=binutils-gdb.git;a=shortlog;h=refs/heads/users/jv/patches/feature-python-mi
> 
> HTH, Jan

Thanks, I think this discussion should now move to gdb-patches.
I'd be interested what is missing to integrate this for GDB 12 
(obviously a "stable" ABI would be needed).

I'm looking forward to read about that in gdb-patches (hopefully sooner 
than later).

Thanks again for taking the time to answer and for your work on 
improving GDB's Python API.

Simon

Is there a way to get a function's end address other than `disassemble`?

[Simon Sobisch]

`info address func_name` will show the *start address* of the given 
function, but is there any built-in way to get the *end address* other 
than executing the `disassemble` command?

The use case here is "if the size of the current function is too big, 
don't use `disassemble` but limit to N instructions around $pc";
using a fixed calculation like
     disassemble $pc - 50, $pc + 150
may get "over" the last instruction of the function...

Simon

Re: Is there a way to get a function's end address other than `disassemble`?

[Andreas Schwab]

On Nov 22 2021, Simon Sobisch via Gdb wrote:

> `info address func_name` will show the *start address* of the given
> function, but is there any built-in way to get the *end address* other 
> than executing the `disassemble` command?

The end address of a function isn't well defined.  Part of the code
could be split off into a separate section, for example.

Even the start address isn't well defined.  The entry address does not
have to point to the lowest address of the function's code.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Is there a way to get a function's end address other than `disassemble`?

[Simon Sobisch]

Am 22.11.2021 um 16:14 schrieb Andreas Schwab:
> On Nov 22 2021, Simon Sobisch via Gdb wrote:
> 
>> `info address func_name` will show the *start address* of the given
>> function, but is there any built-in way to get the *end address* other
>> than executing the `disassemble` command?
> 
> The end address of a function isn't well defined.  Part of the code
> could be split off into a separate section, for example.
> 
> Even the start address isn't well defined.  The entry address does not
> have to point to the lowest address of the function's code.
> 
> Andreas.
> 

That's an interesting point. Does this mean that `disassemble` may not 
show the complete function, too?
If it does - how to get the correct function ranges(!) as `disassemble` 
does?

Simon

Re: feature branch for creating MI commands with python (was: How to create new mi commands via python / get current interpreter in python)

[Jan Vrany]

Hi Simon, Andrew

> > 
> > Done. Rebased on a recent master and pushed into branch:
> > "users/jv/patches/feature-python-mi":
> > 
> > https://sourceware.org/git/?p=binutils-gdb.git;a=shortlog;h=refs/heads/users/jv/patches/feature-python-mi
> > 

I have rebased this branch on to of Andrew's patches and
wanted to update the feature branch on sourceware.org but
it failed:

remote: error: denying non-fast-forward
refs/heads/users/jv/patches/feature-python-mi (you should pull first)

What's the correct way of updating rebased / edited feature branch? 

Andrew, you mentioned you have some thoughts on a different / way of 
implemeting Python MI commands. Can you elaborate? I can try to address
your comments while waiting for your submitted series to be merged
(thanks for doing that!)

Thanks! 

Jan

Re: feature branch for creating MI commands with python (was: How to create new mi commands via python / get current interpreter in python)

[Lancelot SIX]

On Tue, Nov 23, 2021 at 12:29:41PM +0000, Jan Vrany wrote:
> Hi Simon, Andrew
> 
> > > 
> > > Done. Rebased on a recent master and pushed into branch:
> > > "users/jv/patches/feature-python-mi":
> > > 
> > > https://sourceware.org/git/?p=binutils-gdb.git;a=shortlog;h=refs/heads/users/jv/patches/feature-python-mi
> > > 
> 
> I have rebased this branch on to of Andrew's patches and
> wanted to update the feature branch on sourceware.org but
> it failed:
> 
> remote: error: denying non-fast-forward
> refs/heads/users/jv/patches/feature-python-mi (you should pull first)
> 
> What's the correct way of updating rebased / edited feature branch? 
> 

Hi,

When I have to do this and cannot force push to a feature branch, I
first delete the remote branch, and then push to re-create it.
 
This could look like something like this (assuming you push the remove
called origin):

    git push origin :users/jv/patches/feature-python-mi
    git push origin the_name_of_the_local_branch:users/jv/patches/feature-python-mi

Hopefully, this does the trick.

Best,
Lancelot.

Re: feature branch for creating MI commands with python (was: How to create new mi commands via python / get current interpreter in python)

[Jan Vrany]

On Tue, 2021-11-23 at 12:48 +0000, Lancelot SIX wrote:
> On Tue, Nov 23, 2021 at 12:29:41PM +0000, Jan Vrany wrote:
> > Hi Simon, Andrew
> > 
> > > > 
> > > > Done. Rebased on a recent master and pushed into branch:
> > > > "users/jv/patches/feature-python-mi":
> > > > 
> > > > https://sourceware.org/git/?p=binutils-
> > > > gdb.git;a=shortlog;h=refs/heads/users/jv/patches/feature-
> > > > python-mi
> > > > 
> > 
> > I have rebased this branch on to of Andrew's patches and
> > wanted to update the feature branch on sourceware.org but
> > it failed:
> > 
> > remote: error: denying non-fast-forward
> > refs/heads/users/jv/patches/feature-python-mi (you should pull
> > first)
> > 
> > What's the correct way of updating rebased / edited feature branch?
> > 
> 
> Hi,
> 
> When I have to do this and cannot force push to a feature branch, I
> first delete the remote branch, and then push to re-create it.
>  
> This could look like something like this (assuming you push the
> remove
> called origin):
> 
>     git push origin :users/jv/patches/feature-python-mi
>     git push origin
> the_name_of_the_local_branch:users/jv/patches/feature-python-mi
> 
> Hopefully, this does the trick.

Thanks, that indeed did the trick!

Feature branch updated. 

Jan
> 
> Best,
> Lancelot.

Broken source view with Pygments and non-UTF-8 encoded source

[Simon Sobisch]

I _really_ like the Pygments option, but it sadly doesn't work when the 
source code as invalid UTF8 data in it.
As soon as this is the case "list" does not show a nice highlighted code 
any more but instead Python Exception is raised (UnicodeDecodeError) and 
then non-colored lines are shown.

This is especially bad when you "step" through the code because you get 
the python exception for every line.

I _think_ this is related to GDB not passing the source encoding it 
knows of (likely get from LANG) to Pygments.

More details about that: https://pygments.org/docs/unicode/


Additional to the kind request of fixing that in GDB 11.x/12

Is there a way to disable the source highlighting for a single file or 
only globally?


Simon

Re: Broken source view with Pygments and non-UTF-8 encoded source

[Andrew Burgess]

* Simon Sobisch via Gdb <gdb@sourceware.org> [2021-11-25 20:28:18 +0100]:

> I _really_ like the Pygments option, but it sadly doesn't work when the
> source code as invalid UTF8 data in it.
> As soon as this is the case "list" does not show a nice highlighted code any
> more but instead Python Exception is raised (UnicodeDecodeError) and then
> non-colored lines are shown.
> 
> This is especially bad when you "step" through the code because you get the
> python exception for every line.

Could you post here, or provide a link to, an example source file that
triggers the error please.

Thanks,
Andrew


> 
> I _think_ this is related to GDB not passing the source encoding it knows of
> (likely get from LANG) to Pygments.
> 
> More details about that: https://pygments.org/docs/unicode/
> 
> 
> Additional to the kind request of fixing that in GDB 11.x/12
> 
> Is there a way to disable the source highlighting for a single file or only
> globally?
> 
> 
> Simon

Re: Broken source view with Pygments and non-UTF-8 encoded source

[Andrew Burgess]

* Andrew Burgess via Gdb <gdb@sourceware.org> [2021-11-26 11:16:21 +0000]:

> * Simon Sobisch via Gdb <gdb@sourceware.org> [2021-11-25 20:28:18 +0100]:
> 
> > I _really_ like the Pygments option, but it sadly doesn't work when the
> > source code as invalid UTF8 data in it.
> > As soon as this is the case "list" does not show a nice highlighted code any
> > more but instead Python Exception is raised (UnicodeDecodeError) and then
> > non-colored lines are shown.
> > 
> > This is especially bad when you "step" through the code because you get the
> > python exception for every line.
> 
> Could you post here, or provide a link to, an example source file that
> triggers the error please.

Never mind, I figured it out.  I have a fix which I'll post to
gdb-patches shortly.

Thanks,
Andrew