malloc’ing strcat

malloc’ing strcat

[Alexandre François Garreau]

Hi,

strcat needs a buffer already wide enough to contain concatenation of both 
strings, hence I deduce idiomatic use is to first malloc(strlen(str1) + 
strlen(str2) + 1)…

But then the usage must always be something like:
#define strcat(a, b) strcat (strcpy (malloc (strlen(a) + strlen (b) + 1) 
a), b)
(which, of course, requires further free())…

So is there already a function or macro which does that? if not so, why? 
asprintf&cie already exist, after all…

Re: malloc’ing strcat

[Florian Weimer]

* Alexandre François Garreau:

> strcat needs a buffer already wide enough to contain concatenation of both 
> strings, hence I deduce idiomatic use is to first malloc(strlen(str1) + 
> strlen(str2) + 1)…
>
> But then the usage must always be something like:
> #define strcat(a, b) strcat (strcpy (malloc (strlen(a) + strlen (b) + 1) 
> a), b)
> (which, of course, requires further free())…
>
> So is there already a function or macro which does that? if not so, why? 

One reason could be that before C11 (and C++11), it was difficult to
implement in a portable manner.  You would have to use a variadic
function with a sentinel argument, and NULL as the most obvious (and
most portable) choice for the sentinel could silently truncate the
argument list if one of the arguments is specified incorrectly as NULL.

Thanks,
Florian

Re: malloc’ing strcat

[Alexandre François Garreau]

Le lundi 24 février 2020, 12:16:54 CET Florian Weimer a écrit :
> * Alexandre François Garreau:
> > strcat needs a buffer already wide enough to contain concatenation of
> > both strings, hence I deduce idiomatic use is to first
> > malloc(strlen(str1) + strlen(str2) + 1)…
> > 
> > But then the usage must always be something like:
> > #define strcat(a, b) strcat (strcpy (malloc (strlen(a) + strlen (b) +
> > 1) a), b)
> > (which, of course, requires further free())…
> > 
> > So is there already a function or macro which does that? if not so,
> > why?
>
> One reason could be that before C11 (and C++11), it was difficult to
> implement in a portable manner.  You would have to use a variadic
> function with a sentinel argument, and NULL as the most obvious (and
> most portable) choice for the sentinel could silently truncate the
> argument list if one of the arguments is specified incorrectly as NULL.

I didn’t even talk about something variadic (my example macro has a fixed 
(2) valence)… but even so taking care about NULL to make variadicity to 
works looks reasonable to me…

But I’m curious, what did C11 change about that?

And still, why wasn’t a macro such as the one I presented already 
introduced?

Re: malloc’ing strcat

[Florian Weimer]

* Alexandre François Garreau:

> Le lundi 24 février 2020, 12:16:54 CET Florian Weimer a écrit :
>> * Alexandre François Garreau:
>> > strcat needs a buffer already wide enough to contain concatenation of
>> > both strings, hence I deduce idiomatic use is to first
>> > malloc(strlen(str1) + strlen(str2) + 1)…
>> > 
>> > But then the usage must always be something like:
>> > #define strcat(a, b) strcat (strcpy (malloc (strlen(a) + strlen (b) +
>> > 1) a), b)
>> > (which, of course, requires further free())…
>> > 
>> > So is there already a function or macro which does that? if not so,
>> > why?
>>
>> One reason could be that before C11 (and C++11), it was difficult to
>> implement in a portable manner.  You would have to use a variadic
>> function with a sentinel argument, and NULL as the most obvious (and
>> most portable) choice for the sentinel could silently truncate the
>> argument list if one of the arguments is specified incorrectly as NULL.
>
> I didn’t even talk about something variadic (my example macro has a fixed 
> (2) valence)… but even so taking care about NULL to make variadicity to 
> works looks reasonable to me…

The two-argument restriction seems rather arbitrary.

> But I’m curious, what did C11 change about that?

It's possible to use a variadic macro and a compound literal to
construct an array in a macro, and pass that (along with the array size)
to the actual implementation in a function (whose name would be an
implementation detail).

I don't think this could be done in a portable manner before C11.

> And still, why wasn’t a macro such as the one I presented already 
> introduced?

strdup was only recently added to ISO C, and POSIX is somewhat dormant
these days.

Thanks,
Florian

Re: malloc’ing strcat

[Alexandre François Garreau]

Le lundi 24 février 2020, 13:38:32 CET Florian Weimer a écrit :
> * Alexandre François Garreau:
> > Le lundi 24 février 2020, 12:16:54 CET Florian Weimer a écrit :
> >> * Alexandre François Garreau:
> >> > strcat needs a buffer already wide enough to contain concatenation
> >> > of
> >> > both strings, hence I deduce idiomatic use is to first
> >> > malloc(strlen(str1) + strlen(str2) + 1)…
> >> > 
> >> > But then the usage must always be something like:
> >> > #define strcat(a, b) strcat (strcpy (malloc (strlen(a) + strlen (b)
> >> > +
> >> > 1) a), b)
> >> > (which, of course, requires further free())…
> >> > 
> >> > So is there already a function or macro which does that? if not so,
> >> > why?
> >> 
> >> One reason could be that before C11 (and C++11), it was difficult to
> >> implement in a portable manner.  You would have to use a variadic
> >> function with a sentinel argument, and NULL as the most obvious (and
> >> most portable) choice for the sentinel could silently truncate the
> >> argument list if one of the arguments is specified incorrectly as
> >> NULL.
> > 
> > I didn’t even talk about something variadic (my example macro has a
> > fixed (2) valence)… but even so taking care about NULL to make
> > variadicity to works looks reasonable to me…
> 
> The two-argument restriction seems rather arbitrary.

Not any more than the current standard strcat functions, which more or 
less behaves the same (except my macro doesn’t buffer overflow, and is not 
destructive, but I could as well make a version that is destructive as 
well using realloc that would fail for a statically allocated first 
argument):
#define astrcat(a, b) strcat (realloc (a, strlen(a) + strlen (b) + 1), b)

The most often use is to concatenate two strings at a time anyway, so it’s 
already a big step forward to have a self contained interface, rather than 
always repetitively calling malloc with the same arguments and often with 
strcpy combined…

> > But I’m curious, what did C11 change about that?
> 
> It's possible to use a variadic macro and a compound literal to
> construct an array in a macro, and pass that (along with the array size)
> to the actual implementation in a function (whose name would be an
> implementation detail).

Ohhhh, you mean the function-arg-separators commas would become arrays-
separators commas, like that?
#define astrcat(...) _astrcat((char *[]) {__VA_ARGS__, NULL})
char *
_astrcat (char *strarray[])
{
  char * result = malloc(1);
  size_t len = 1;
  off_t i = -1;
  result[0] = '\0';
  while (strarray[++i])
  result = strcat(realloc(result, len += strlen(strarray[i])), 
strarray[i]);
  return result;
}

> I don't think this could be done in a portable manner before C11.

Why couldn’t a standard variadic function do? it’s only longer:
#define astrcat(...) _astrcat (__VA_ARGS__, NULL)
char *
_astrcat (char * str, ...)
{
  va_list ap;
  size_t len = strlen(str)+1;
  char *arg, *result = malloc(len);
  strcpy(result, str);
  va_start(ap, str);
  while (arg = va_arg(ap, char*))
    result = strcat(realloc(result, len += strlen(arg)), arg);
  va_end(ap);
  return result;
}

I tried these and they work, without appearing to be specially non-
standard to me…

> > And still, why wasn’t a macro such as the one I presented already
> > introduced?
> 
> strdup was only recently added to ISO C, and POSIX is somewhat dormant
> these days.

but there were already convenient functions before standardizations 
weren’t they? such as the scanf “m” (or “a” I forgot) conversion modifier, 
or vasprintf… right?

I mean, glibc have extensions and doesn’t have to be the standard and only 
the standard, afaik

Re: malloc’ing strcat

[Florian Weimer]

* Alexandre François Garreau:

> Not any more than the current standard strcat functions, which more or 
> less behaves the same (except my macro doesn’t buffer overflow, and is not 
> destructive, but I could as well make a version that is destructive as 
> well using realloc that would fail for a statically allocated first 
> argument):
> #define astrcat(a, b) strcat (realloc (a, strlen(a) + strlen (b) + 1), b)
>
> The most often use is to concatenate two strings at a time anyway, so it’s 
> already a big step forward to have a self contained interface, rather than 
> always repetitively calling malloc with the same arguments and often with 
> strcpy combined…

That doesn't match what I see.  After all, without garbage collection,
you can't do this:

  astrcat (astract (a, b), c)

And multi-concatenations are fairly common:

  <https://codesearch.debian.net/search?q=stpcpy.*stpcpy.*stpcpy&literal=0>

>> > But I’m curious, what did C11 change about that?
>> 
>> It's possible to use a variadic macro and a compound literal to
>> construct an array in a macro, and pass that (along with the array size)
>> to the actual implementation in a function (whose name would be an
>> implementation detail).
>
> Ohhhh, you mean the function-arg-separators commas would become arrays-
> separators commas, like that?
> #define astrcat(...) _astrcat((char *[]) {__VA_ARGS__, NULL})
> char *
> _astrcat (char *strarray[])
> {
>   char * result = malloc(1);
>   size_t len = 1;
>   off_t i = -1;
>   result[0] = '\0';
>   while (strarray[++i])
>   result = strcat(realloc(result, len += strlen(strarray[i])), 
> strarray[i]);
>   return result;
> }

I think I wouldn't use the NULL sentinel for the reasons indicated
(silent truncation after memory allocation or other failure, even under
valgrind), and use

  sizeof ((const char *[]) { __VA_ARGS__ }) / sizeof (const char *)

to compute the number of arguments, and pass that separately from the
pointer.

>> I don't think this could be done in a portable manner before C11.
>
> Why couldn’t a standard variadic function do? it’s only longer:
> #define astrcat(...) _astrcat (__VA_ARGS__, NULL)

Yes, I meant C99.  Compound literals and __VA_ARGS__ were added in C99.

>> > And still, why wasn’t a macro such as the one I presented already
>> > introduced?
>> 
>> strdup was only recently added to ISO C, and POSIX is somewhat dormant
>> these days.
>
> but there were already convenient functions before standardizations 
> weren’t they? such as the scanf “m” (or “a” I forgot) conversion modifier, 
> or vasprintf… right?

asprintf is still a GNU extension.  The m scanf modifier is only in
POSIX.

Thanks,
Florian

Re: malloc’ing strcat

[Alexandre François Garreau]

Le lundi 24 février 2020, 22:17:10 CET Florian Weimer a écrit :
> * Alexandre François Garreau:
> > Not any more than the current standard strcat functions, which more or
> > less behaves the same (except my macro doesn’t buffer overflow, and is
> > not destructive, but I could as well make a version that is
> > destructive as well using realloc that would fail for a statically
> > allocated first argument):
> > #define astrcat(a, b) strcat (realloc (a, strlen(a) + strlen (b) + 1),
> > b)
> > 
> > The most often use is to concatenate two strings at a time anyway, so
> > it’s already a big step forward to have a self contained interface,
> > rather than always repetitively calling malloc with the same
> > arguments and often with strcpy combined…
> 
> That doesn't match what I see.  After all, without garbage collection,
> you can't do this:
> 
>   astrcat (astract (a, b), c)
> 
> And multi-concatenations are fairly common:
> <https://codesearch.debian.net/search?
q=stpcpy.*stpcpy.*stpcpy&literal=0>

Ohhh, that explains why there is that standard function not allocating 
anything… because you do the allocation before to use your commonly user-
supplied number of concatenations…

> >> > But I’m curious, what did C11 change about that?
> >> 
> >> It's possible to use a variadic macro and a compound literal to
> >> construct an array in a macro, and pass that (along with the array
> >> size) to the actual implementation in a function (whose name would
> >> be an implementation detail).
> > 
> > Ohhhh, you mean the function-arg-separators commas would become
> > arrays-
> > separators commas, like that?
> > #define astrcat(...) _astrcat((char *[]) {__VA_ARGS__, NULL})
> > char *
> > _astrcat (char *strarray[])
> > {
> > 
> >   char * result = malloc(1);
> >   size_t len = 1;
> >   off_t i = -1;
> >   result[0] = '\0';
> >   while (strarray[++i])
> >   result = strcat(realloc(result, len += strlen(strarray[i])),
> > 
> > strarray[i]);
> > 
> >   return result;
> > 
> > }
> 
> I think I wouldn't use the NULL sentinel for the reasons indicated
> (silent truncation after memory allocation or other failure, even under
> valgrind), and use
> 
>   sizeof ((const char *[]) { __VA_ARGS__ }) / sizeof (const char *)
> 
> to compute the number of arguments, and pass that separately from the
> pointer.

you mean like that?
#define anstrcat(...) \
astrncat(sizeof((char *[]) {__VA_ARGS__}), \
                (char *[]) {__VA_ARGS__})
char * anstrcat (size_t n; char *strarray[])
{
   char * result = malloc(1);
   size_t len = 1;
   result[0] = '\0';
   for (off_t i = 0; i < n; ++i)
   result = strcat(realloc(result, len += strlen(strarray[i])), 
strarray[i]);
   return result;
}

Yet this only works with a macro, and is it really unreasonable to expect 
not to give NULL args?

> >> I don't think this could be done in a portable manner before C11.
> > 
> > Why couldn’t a standard variadic function do? it’s only longer:
> > #define astrcat(...) _astrcat (__VA_ARGS__, NULL)
> 
> Yes, I meant C99.  Compound literals and __VA_ARGS__ were added in C99.

Oh! that’s way different.  Not all C11 is implemented but C99 is, afaik, 
right? also since recently, C99 got as distant from now as it is from C 
creation.

> >> > And still, why wasn’t a macro such as the one I presented already
> >> > introduced?
> >> 
> >> strdup was only recently added to ISO C, and POSIX is somewhat
> >> dormant
> >> these days.
> > 
> > but there were already convenient functions before standardizations
> > weren’t they? such as the scanf “m” (or “a” I forgot) conversion
> > modifier, or vasprintf… right?
> 
> asprintf is still a GNU extension.  The m scanf modifier is only in
> POSIX.

Anyway it is not common for non-standard interfaces to be provided… so why 
not one now? like one of the few I’ve shown?

Re: malloc’ing strcat

[Florian Weimer]

* Alexandre François Garreau:

>> I think I wouldn't use the NULL sentinel for the reasons indicated
>> (silent truncation after memory allocation or other failure, even under
>> valgrind), and use
>> 
>>   sizeof ((const char *[]) { __VA_ARGS__ }) / sizeof (const char *)
>> 
>> to compute the number of arguments, and pass that separately from the
>> pointer.
>
> you mean like that?
> #define anstrcat(...) \
> astrncat(sizeof((char *[]) {__VA_ARGS__}), \
>                 (char *[]) {__VA_ARGS__})
> char * anstrcat (size_t n; char *strarray[])
> {
>    char * result = malloc(1);
>    size_t len = 1;
>    result[0] = '\0';
>    for (off_t i = 0; i < n; ++i)
>    result = strcat(realloc(result, len += strlen(strarray[i])), 
> strarray[i]);
>    return result;
> }

Sort of, except with overflow checking and without the quadratic
behavior.

> Yet this only works with a macro, and is it really unreasonable to expect 
> not to give NULL args?

I think so.

But there is another complication: The NULL sentinel is not portable
because NULL can be defined as 0, which does not have the correct type
for use in an argument list of character pointers.

Thanks,
Florian

Re: malloc’ing strcat

[Alexandre François Garreau]

Le mardi 25 février 2020, 10:41:30 CET Florian Weimer a écrit :
> * Alexandre François Garreau:
> >> I think I wouldn't use the NULL sentinel for the reasons indicated
> >> (silent truncation after memory allocation or other failure, even
> >> under
> >> valgrind), and use
> >> 
> >>   sizeof ((const char *[]) { __VA_ARGS__ }) / sizeof (const char *)
> >> 
> >> to compute the number of arguments, and pass that separately from the
> >> pointer.
> > 
> > you mean like that?
> > #define anstrcat(...) \
> > astrncat(sizeof((char *[]) {__VA_ARGS__}), \
> > 
> >                 (char *[]) {__VA_ARGS__})
> > 
> > char * anstrcat (size_t n; char *strarray[])
> > {
> > 
> >    char * result = malloc(1);
> >    size_t len = 1;
> >    result[0] = '\0';
> >    for (off_t i = 0; i < n; ++i)
> >    result = strcat(realloc(result, len += strlen(strarray[i])),
> > 
> > strarray[i]);
> > 
> >    return result;
> > 
> > }
> 
> Sort of, except with overflow checking

I’m curious: how would it overflow if strlen already checks the end (the 
\0) of each string (so that it knows how much strcat will copy)?

> and without the quadratic
> behavior.

Why is it quadratic? with a single loop… are you talking about calling 
several times realloc, or strlen?

Or simply about using strcpy or stpcpy (or memcpy or mempcpy so not to 
check for \0-position a second time) instead because you could get the 
NULL-char*position and hence the end of a string only once?

char * anstrcat (size_t n; char *strarray[])
{
  char *result = NULL, *pos = result;
  size_t total_len = 1;
  for (off_t i = 0; i < n; ++i)
    {
      size_t len = strlen (strarray[i]);
      result = realloc (result, total_len += len);
      pos = mempcpy (pos, strarray[i], len);
    }
  result[total_len-1] = '\0';
  return result;
}

I don’t see how to do less than n strlen (except by asking to provide an 
array of lengths but to me it’s overkill and not really C-ish to me… we 
need each string’s size…) , nor how to call malloc only once without first 
doing twice as much strlens… except by storing them in an VLA array (or 
mallocated, for compatibility sake) but isn’t also that overkill? well, 
“memory’s almost free”… Is calling realloc n times a bad thing?

> > Yet this only works with a macro, and is it really unreasonable to
> > expect not to give NULL args?
> 
> I think so.

For the sake of not doing every single time the same malloc’s oneself, I 
believe a such function would be too useful for a such requirement to 
hinder its benefits…

Also I like simple interfaces, and an interface that doesn’t require to 
count the arguments (the same way you don’t have to always provide the 
size of a string) looks cool to me… because I wonder about the cases 
people will have to use it without the macro… 

I mean such usage looks so basic and likely what everybody would do since 
for much time that it ought to get a general interface so to factorize a 
lot…

> But there is another complication: The NULL sentinel is not portable
> because NULL can be defined as 0, which does not have the correct type
> for use in an argument list of character pointers.

And would (char*)NULL work as a replacement? otherwise how one is to 
specify a null pointer, a null object, etc. in a portable manner?

Re: malloc’ing strcat

[Florian Weimer]

* Alexandre François Garreau:

>> Sort of, except with overflow checking
>
> I’m curious: how would it overflow if strlen already checks the end (the 
> \0) of each string (so that it knows how much strcat will copy)?

The pointers can alias, so that the combined length of the strings is
larger than the entire addresspace.

>> and without the quadratic
>> behavior.
>
> Why is it quadratic? with a single loop… are you talking about calling 
> several times realloc, or strlen?

realloc and strcat look problematic.

>> But there is another complication: The NULL sentinel is not portable
>> because NULL can be defined as 0, which does not have the correct type
>> for use in an argument list of character pointers.
>
> And would (char*)NULL work as a replacement? otherwise how one is to 
> specify a null pointer, a null object, etc. in a portable manner?

Yes, you would have to write (char *) 0 or something like that.  Of
course, no one does.

Thanks,
Florian

Re: malloc’ing strcat

[Alexandre François Garreau]

Le mardi 25 février 2020, 13:53:32 CET Florian Weimer a écrit :
> * Alexandre François Garreau:
> >> Sort of, except with overflow checking
> > 
> > I’m curious: how would it overflow if strlen already checks the end
> > (the \0) of each string (so that it knows how much strcat will copy)?
>
> The pointers can alias, so that the combined length of the strings is
> larger than the entire addresspace.

Alias? what does it mean? I don’t see the additional risk in respect to 
strcat…

The only thing “alias” made me think were two pointers of the array 
pointing to the same string… but that shouldn’t be that…

> >> and without the quadratic
> >> behavior.
> > 
> > Why is it quadratic? with a single loop… are you talking about calling
> > several times realloc, or strlen?
> 
> realloc and strcat look problematic.

So how’s this?
char * anstrcat (size_t n; char *strarray[])
{
  char *result, *pos = result;
  size_t total_len = 0;
  size_t *len = malloc (sizeof (size_t) * n);
  for (off_t i = 0; i < n; ++i)
    total_len += len[i] = strlen (strarray[i]);
  result = malloc (total_len + 1);
  for (off_t i = 0; i < n; ++i)
    pos = mempcpy (pos, strarray[i], len[i]);
  free(len);
  result[total_len] = '\0';
  return result;
}

To me it feels so basic to always use strcat in combination with strlen, 
that an abstraction layer above that seems necessary… don’t you think so? 
isn’t that otherwise the cause of much effort duplication?

> >> But there is another complication: The NULL sentinel is not portable
> >> because NULL can be defined as 0, which does not have the correct
> >> type
> >> for use in an argument list of character pointers.
> > 
> > And would (char*)NULL work as a replacement? otherwise how one is to
> > specify a null pointer, a null object, etc. in a portable manner?
> 
> Yes, you would have to write (char *) 0 or something like that.  Of
> course, no one does.

You said it was necessary for portability… but well I understand you don’t 
want to use NULL at all.