Using libcurl in another library, when/if to call curl_global_init?

Using libcurl in another library, when/if to call curl_global_init?

[Mark Wielaard]

Hi,

We we are using libcurl in our own library now to (optionally) fetch
resources from http(s) locations. It is an implementation detail we use
libcurl for this and don't expose any Curl handles outside our own
library. It works great, thanks!

But we are struggling a bit with how to safely/correctly initialize
libcurl. Our own library doesn't need any global initialization, so
users won't use any global init like function. We have therefore added
a constructor function to the library that is called as soon as the
library is loaded which simply calls
curl_global_init(CURL_GLOBAL_DEFAULT);

We added it in the constructor so that it is called as early as
possible (hopefully) before the program created any threads because the
curl_global_init documentation says to call it before any threads are
created by the program.

This seems to work, but it means that curl_global_init is always called
even if our own library happens to not use libcurl to fetch any remote
resources. And this is causing some problems for our users because (in
FIPS mode) libcurl does significant initialization work (to check the
ssl implementation?)

The documentation seems to imply that you can also call curl_easy_init
without calling curl_global_init first. And doing that seems to work
fine. But our own testsuite doesn't contain many multi-threaded
examples.

Since the documentation does imply that doing without curl_global_init
might not be thread-safe we wonder how other libraries that use libcurl
are doing this. Or whether there is a thread-safe way to call
curl_global_init at a later time (to get rid of the library constructor
init function).

Thanks,

Mark

Re: Using libcurl in another library, when/if to call curl_global_init?

[Mark Wielaard]

Hi,

On Thu, Mar 31, 2022 at 04:00:16PM +0300, Catalin Raceanu via curl-library wrote:
> On 31-Mar-22 15:04, Mark Wielaard wrote:
> > whether there is a thread-safe way to call
> > curl_global_init at a later time (to get rid of the library constructor
> > init function).
> 
> I believe that this is an exact fit for C==11's std::call_once(). Boost also
> has an equivalent, that most likely predates the other, in case older c++
> standard is used.

Thanks. Our library is pure C, but we can probably rely on
pthread_once if it is allowable to call curl_global_init at a later
time when multiple threads are already running. The reason we aren't
doing that now is because the curl_global_init documentation
explicitly states "You must not call it when any other thread in the
program is running". But maybe we are interpreting the documentation
too strictly?

Are there examples of other libraries using libcurl which do this
global initialization lazily from which we can steal some code?

Thanks,

Mark

Re: Using libcurl in another library, when/if to call curl_global_init?

[Florian Weimer]

* Mark Wielaard:

> Hi,
>
> On Thu, Mar 31, 2022 at 04:00:16PM +0300, Catalin Raceanu via curl-library wrote:
>> On 31-Mar-22 15:04, Mark Wielaard wrote:
>> > whether there is a thread-safe way to call
>> > curl_global_init at a later time (to get rid of the library constructor
>> > init function).
>> 
>> I believe that this is an exact fit for C==11's std::call_once(). Boost also
>> has an equivalent, that most likely predates the other, in case older c++
>> standard is used.
>
> Thanks. Our library is pure C, but we can probably rely on
> pthread_once if it is allowable to call curl_global_init at a later
> time when multiple threads are already running.

The problem is that every caller of pthread_once needs to use the same
pthread_once_t synchronization object, so you still have the same
problem.

I strongly believe that library-safe code needs to perform lazy
initialization, that is, initialize the library on first use, and do
that in a thread-safe manner.  It solves the synchronization issue
between different users of the API, and it's the only way to report
errors properly (for example, a failure in an ELF constructor can only
be reported via process termination).

At least on Linux, the need to support multiple different threading
libraries is a thing of the past.

Thanks,
Florian

Re: Using libcurl in another library, when/if to call curl_global_init?

[Mark Wielaard]

Hi,

On Thu, Mar 31, 2022 at 03:19:51PM +0200, Mark Wielaard via curl-library wrote:
> On Thu, Mar 31, 2022 at 04:00:16PM +0300, Catalin Raceanu via curl-library wrote:
> > On 31-Mar-22 15:04, Mark Wielaard wrote:
> > > whether there is a thread-safe way to call
> > > curl_global_init at a later time (to get rid of the library constructor
> > > init function).
> > 
> > I believe that this is an exact fit for C==11's std::call_once(). Boost also
> > has an equivalent, that most likely predates the other, in case older c++
> > standard is used.
> 
> Thanks. Our library is pure C, but we can probably rely on
> pthread_once if it is allowable to call curl_global_init at a later
> time when multiple threads are already running. The reason we aren't
> doing that now is because the curl_global_init documentation
> explicitly states "You must not call it when any other thread in the
> program is running". But maybe we are interpreting the documentation
> too strictly?

Since it does seem we were interpreting the documentation a bit too
strictly I did propose a patch for elfutils that does lazy
initialization using pthread_once.

https://sourceware.org/pipermail/elfutils-devel/2022q2/004934.html

This makes sure we don't race against ourselves. And for programs
using our library we can assume they will call curl_global_init before
creating a multi-threaded environment and calling our own handle
initialization functions.

It would still be great if curl_global_init itself could be made
thread-safe to call.

Thanks,

Mark

Re: Using libcurl in another library, when/if to call curl_global_init?

[Daniel Stenberg]

On Thu, 31 Mar 2022, Mark Wielaard via curl-library wrote:

> But we are struggling a bit with how to safely/correctly initialize libcurl.

Are you struggling to meet the requirement as per the documentation or are you 
seeing actual runtime issues?

There isn't much left in the third party libraries that isn't threadsafe so 
I'm quite interested in knowing. The "not thread-safe" part is mostly 
theoretic now with modern versions of libraries (such as OpenSSL).

-- 

  / daniel.haxx.se
  | Commercial curl support up to 24x7 is available!
  | Private help, bug fixes, support, ports, new features
  | https://curl.se/support.html

Re: Using libcurl in another library, when/if to call curl_global_init?

[Mark Wielaard]

On Tue, 2022-04-05 at 17:36 +0200, Daniel Stenberg wrote:
> On Thu, 31 Mar 2022, Mark Wielaard via curl-library wrote:
> 
> > But we are struggling a bit with how to safely/correctly initialize
> > libcurl.
> 
> Are you struggling to meet the requirement as per the documentation
> or are you seeing actual runtime issues?

Struggling with the requirements as per the documentation "You must not
call it [curl_global_init] when any other thread in the program is
running".

Especially combined with having users who would not like to pay the
cost of initializing libcurl (in FIPS mode) when the program isn't
explicitly using the remote resource functionality of our library (but
is still linked to it).

We are a library that is (sometimes) dlopened, so we cannot guarantee
that when we call curl_global_init no other thread in the program is
running. But we try to mitigate that by having a
__attribute__((constructor)) ctor that just does
curl_global_init(CURL_GLOBAL_DEFAULT);
Which in most cases (especially if we aren't dlopened) should make sure
we run before the program has any chance to start any new threads.

But that of course also makes sure that we (or the program linking to
our library) always pays the cost for calling curl_global_init, which
at least in FIPS mode does some non-trivial checks.

So we are wondering if instead we could do lazy initialization just
before we know we will actually need to load a remote resource. We
could wrap such an initialization inside a pthread_once, so at least
with multiple threads we don't race against ourselves given that
curl_global_init itself is not thread-safe (it uses a unguarded
initialized int).

But Florian just pointed out that we would still race against other
uses of libcurl in the program, in case they also didn't call
curl_global_init before starting any other threads.

Cheers,

Mark