Extending vfprintf and family

Extending vfprintf and family

[Philip Prindeville]

Hi.

I work a lot with routing, protocols, and firewalls so I’m constantly having to dump packets for tracing, logging, auditing, etc.

Common types I need to format are:

* IPv4 dotted quads
* MAC addresses (not just Ethernet/Wifi)
* IPv6 addresses
* uint16_t and uint32_t in network (BigEndian) byte order
* n-byte hex dump (BigEndian) of an arbitrary blob
* safe n-byte string formatting (printable characters get printed, others [including NUL] get escaped into \xHH or \OOO)

It’s a bit tedious (not to mention inefficient) to have to call inet_ntop() to format a machine address into a readable format (via a temporary buffer), then interpolate that string into a logging message via syslog() or snprintf(), etc.  What would be really great would be adding extensions (which can be turned off or on, depending on whether strict compliance with C99, ANSI C, etc is required) as format specifiers.

When I was at Cisco in the IOS group, there was a way to register on-the-fly additional format specifiers that *printf() could then leverage; some of the formatting helpers were quite sophisticated, understanding ASN.1 or X.509 certs, etc.  Not sure we need to be that complex, though I wouldn’t reject it if it were offered, either.

One would be able to specify both precision and width, e.g. “I have a 20 byte safe string I want to format, but not into more than 32 columns” so that if there was a lot of escaping and the string overflowed the 32-byte output count, it would be truncated.

Does anyone else see the value in doing this?

Thanks,

-Philip

Re: Extending vfprintf and family

[Russ Allbery]

Philip Prindeville <philipp_subx@redfish-solutions.com> writes:

> When I was at Cisco in the IOS group, there was a way to register
> on-the-fly additional format specifiers that *printf() could then
> leverage; some of the formatting helpers were quite sophisticated,
> understanding ASN.1 or X.509 certs, etc.  Not sure we need to be that
> complex, though I wouldn’t reject it if it were offered, either.

glibc supports exactly this.  See:

https://www.gnu.org/software/libc/manual/html_node/Customizing-Printf.html#Customizing-Printf

-- 
Russ Allbery (eagle@eyrie.org)              <http://www.eyrie.org/~eagle/>

Re: Extending vfprintf and family

[Rich Felker]

On Thu, Aug 14, 2014 at 11:45:07AM -0700, Russ Allbery wrote:
> Philip Prindeville <philipp_subx@redfish-solutions.com> writes:
> 
> > When I was at Cisco in the IOS group, there was a way to register
> > on-the-fly additional format specifiers that *printf() could then
> > leverage; some of the formatting helpers were quite sophisticated,
> > understanding ASN.1 or X.509 certs, etc.  Not sure we need to be that
> > complex, though I wouldn’t reject it if it were offered, either.
> 
> glibc supports exactly this.  See:
> 
> https://www.gnu.org/software/libc/manual/html_node/Customizing-Printf.html#Customizing-Printf

But this feature really should not be used except possibly for
implementing in-progress extensions to the base language, like
decimal-float. There's a very small space of characters available for
use, risk of colliding with use by other code, and various
thread-safety, dlclose-safety, etc. issues with registration.

There's a very simple solution to the temp buffer problem: compound
literals. Doing something like this works:

static char *do_fmt_sockaddr(void *sa, socklen_t sl, char *buf, size_t size)
{
	getnameinfo(sa, sl, buf, size, 0, 0, NI_NUMERICHOST);
	return buf;
}
#define fmt_sockaddr(sa, sl)  do_fmt_sockaddr(sa, sl, (char[HOSTMAX]){0}, HOSTMAX)

...

printf("%s\n", fmt_sockaddr(sa, sl));

Perhaps most importantly, this allows you to keep the logic for how a
particular type should be formatted local to the calling code rather
than imposing global state for the choice of formatting.

Note that for this example a socklen was needed, which is ugly; you
could fix that by just having a lookup table of the right lengths for
each address family. I've used getnameinfo rather than inet_ntop
because the latter fails to show link ids for link-local ipv6
addresses despite their being mandatory, thus making it unsuitable for
general use.

Rich

Re: Extending vfprintf and family

[Philip Prindeville]

On Aug 14, 2014, at 12:45 PM, Russ Allbery <eagle@eyrie.org> wrote:

> Philip Prindeville <philipp_subx@redfish-solutions.com> writes:
> 
>> When I was at Cisco in the IOS group, there was a way to register
>> on-the-fly additional format specifiers that *printf() could then
>> leverage; some of the formatting helpers were quite sophisticated,
>> understanding ASN.1 or X.509 certs, etc.  Not sure we need to be that
>> complex, though I wouldn’t reject it if it were offered, either.
> 
> glibc supports exactly this.  See:
> 
> https://www.gnu.org/software/libc/manual/html_node/Customizing-Printf.html#Customizing-Printf


Thanks, that was helpful.  One thing that I don’t think was clear, though, is how the same specifier could handle a uint16_t, uint32_t, or uint64_t value.  For instance, for a ‘Q’ specifier:

%hq
%q
%llq

as examples.  What’s a portable way to retrieve a variable-sized scalar argument?

Thanks,

-Philip

Re: Extending vfprintf and family

[Ryan Arnold]

On Mon, Aug 18, 2014 at 8:23 PM, Philip Prindeville
<philipp_subx@redfish-solutions.com> wrote:
>
>
> On Aug 14, 2014, at 12:45 PM, Russ Allbery <eagle@eyrie.org> wrote:
>
> > Philip Prindeville <philipp_subx@redfish-solutions.com> writes:
> >
> >> When I was at Cisco in the IOS group, there was a way to register
> >> on-the-fly additional format specifiers that *printf() could then
> >> leverage; some of the formatting helpers were quite sophisticated,
> >> understanding ASN.1 or X.509 certs, etc.  Not sure we need to be that
> >> complex, though I wouldn’t reject it if it were offered, either.
> >
> > glibc supports exactly this.  See:
> >
> > https://www.gnu.org/software/libc/manual/html_node/Customizing-Printf.html#Customizing-Printf
>
>
> Thanks, that was helpful.  One thing that I don’t think was clear, though, is how the same specifier could handle a uint16_t, uint32_t, or uint64_t value.  For instance, for a ‘Q’ specifier:
>
> %hq
> %q
> %llq
>
> as examples.  What’s a portable way to retrieve a variable-sized scalar argument?
>
> Thanks,
>
> -Philip
>

Hi Philip,

Libvecpf uses these hooks, and perhaps it does something similar to
what you're trying to accomplish:

https://github.com/Libvecpf/libvecpf
-- 
Ryan S. Arnold
Linaro Toolchain Working Group
www.linaro.org

Re: Extending vfprintf and family

[Philip Prindeville]

On Aug 18, 2014, at 7:40 PM, Ryan Arnold <ryan.arnold@linaro.org> wrote:

> 
> Hi Philip,
> 
> Libvecpf uses these hooks, and perhaps it does something similar to
> what you're trying to accomplish:
> 
> https://github.com/Libvecpf/libvecpf


Thanks, that helped a bit.  Now I just need to figure out how to make -Wformat work with custom format specifiers.

I was looking at the gcc __attribute__ list, but can’t find one that informs gcc of additional formatters.

Re: Extending vfprintf and family

[Philip Prindeville]

Hmmm… Actually, there already is a standard for format specifiers for common network objects, or at least there is in Linux.

https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/tree/lib/vsprintf.c?id=refs/tags/v3.16.1#n1608

It doesn’t look like most of this could be implemented using register_printf_*(). It doesn’t seem to me that the state machine which parses the format could pick up characters past the actual conversion specifier itself:

   Format of the format string
       The  format  string  is a character string, beginning and ending in its
       initial shift state, if any.  The format string is composed of zero  or
       more   directives:  ordinary  characters  (not  %),  which  are  copied
       unchanged to the output stream; and conversion specifications, each  of
       which results in fetching zero or more subsequent arguments.  Each con‐
       version specification is introduced by the character %, and ends with a
       conversion  specifier.

Looks like libbacktrace would be required, at a minimum.  And the printf_function() would need to be able to peek ahead in the format string and consume extra characters… which it’s not able to do.

Looks like it would be easier just to port the vsnprintf code from the Linux kernel. Sigh.

-Philip