From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 6152 invoked by alias); 12 Aug 2007 19:52:19 -0000 Received: (qmail 5459 invoked by uid 22791); 12 Aug 2007 19:52:14 -0000 X-Spam-Check-By: sourceware.org Received: from sunsite.ms.mff.cuni.cz (HELO sunsite.mff.cuni.cz) (195.113.15.26) by sourceware.org (qpsmtpd/0.31) with ESMTP; Sun, 12 Aug 2007 19:52:02 +0000 Received: from sunsite.mff.cuni.cz (localhost.localdomain [127.0.0.1]) by sunsite.mff.cuni.cz (8.13.8/8.13.8) with ESMTP id l7CJvaJ9005374; Sun, 12 Aug 2007 21:57:36 +0200 Received: (from jakub@localhost) by sunsite.mff.cuni.cz (8.13.8/8.13.8/Submit) id l7CJvaiV005373; Sun, 12 Aug 2007 21:57:36 +0200 Date: Sun, 12 Aug 2007 19:52:00 -0000 From: Jakub Jelinek To: Ulrich Drepper Cc: Andreas Jaeger , Glibc hackers Subject: [PATCH] ldconfig speedup Message-ID: <20070812195735.GK4603@sunsite.mff.cuni.cz> Reply-To: Jakub Jelinek References: <876455cgf1.fsf@mid.deneb.enyo.de> <20070703112849.GE4603@sunsite.mff.cuni.cz> <468AB559.2030801@redhat.com> <20070727121434.GO4603@sunsite.mff.cuni.cz> <20070727165326.GP4603@sunsite.mff.cuni.cz> <20070727183626.GQ4603@sunsite.mff.cuni.cz> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20070727183626.GQ4603@sunsite.mff.cuni.cz> User-Agent: Mutt/1.4.2.2i Mailing-List: contact libc-hacker-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: libc-hacker-owner@sourceware.org X-SW-Source: 2007-08/txt/msg00010.txt.bz2 On Fri, Jul 27, 2007 at 08:36:26PM +0200, Jakub Jelinek wrote: > On Fri, Jul 27, 2007 at 07:08:55PM +0200, Andreas Jaeger wrote: > > > > What kind of testing have you done? I compared the /etc/ld.so.cache > > From the old ldconfig with the new one - and run the new ldconfig > > without the aux cache and then several times with the new one. I first > > had a bug where the results were not exactly the same and I lost some > > entries. ;-( > > Yeah, that's exactly what I tested, compare /etc/ld.so.cache generated > by non-patched ldconfig with patched ldconfig when > /var/cache/ldconfig/aux-cache was missing, then with the cache, then with > -i. I also looked at strace output to see how the caching was effective. > When working on the patch I had a buglet in it too which caused all > SONAMEs returned from the aux cache contain unitialized memory data, > it created wonderful symlinks all over my */lib{,64} directories. > But the version I posted seemed to work well. During testing we found problems with the error messages and non-zero exit when some problem was detected during aux-cache creation, e.g. when ldconfig is ran in some chroot etc. The aux-cache is only an optimization, so if anything fails during creation of the aux-cache, it shouldn't be fatal nor we should output any diagnostics. The following updated patch does that and we are successfully using it in Fedora development for almost a fortnight. 2007-08-01 Andreas Jaeger Jakub Jelinek * elf/ldconfig.c (opt_ignore_aux_cache): Add new option. (options): Add option. (parse_opt): Handle option. (manual_link): Adjust process_file caller. Call implicit_soname. (search_dir): Formatting. Use and populate auxiliary cache. (main): Load and save auxiliary cache. * elf/readlib.c (process_file): Add stat_buf argument. Pass struct stat64 from fstat64 to caller. (implicit_soname): New function. * elf/readelflib.c (process_elf_file): If DT_SONAME is not present, leave *soname as NULL. * elf/cache.c: Include libgen.h. (print_entry, print_cache, compare, save_cache, add_to_cache): Formatting and cleanups. (aux_cache_entry_id, aux_cache_entry, aux_cache_file_entry, aux_cache_file): New structures. (AUX_CACHEMAGIC): Define. (primes): New array. (aux_hash_size, aux_hash): New variables. (aux_cache_entry_id_hash, nextprime, init_aux_cache, search_aux_cache, insert_to_aux_cache, add_to_aux_cache, load_aux_cache, save_aux_cache): New functions. * sysdeps/generic/ldconfig.h (_PATH_LDCONFIG_AUX_CACHE): Define. (init_aux_cache, search_aux_cache, add_to_aux_cache, load_aux_cache, save_aux_cache, implicit_soname): New prototypes. (process_file): Adjust prototype. --- libc/elf/ldconfig.c.jj 2007-07-16 09:58:46.000000000 +0200 +++ libc/elf/ldconfig.c 2007-08-01 18:54:26.000000000 +0200 @@ -112,6 +112,9 @@ static char *opt_chroot; /* Manually link given shared libraries. */ static int opt_manual_link; +/* Should we ignore an old auxiliary cache file? */ +static int opt_ignore_aux_cache; + /* Cache file to use. */ static char *cache_file; @@ -142,6 +145,7 @@ static const struct argp_option options[ { NULL, 'n', NULL, 0, N_("Only process directories specified on the command line. Don't build cache."), 0}, { NULL, 'l', NULL, 0, N_("Manually link individual libraries."), 0}, { "format", 'c', N_("FORMAT"), 0, N_("Format to use: new, old or compat (default)"), 0}, + { "ignore-aux-cache", 'i', NULL, 0, N_("Ignore auxiliary cache file"), 0}, { NULL, 0, NULL, 0, NULL, 0 } }; @@ -238,10 +242,15 @@ parse_opt (int key, char *arg, struct ar { case 'C': cache_file = arg; + /* Ignore auxiliary cache since we use non-standard cache. */ + opt_ignore_aux_cache = 1; break; case 'f': config_file = arg; break; + case 'i': + opt_ignore_aux_cache = 1; + break; case 'l': opt_manual_link = 1; break; @@ -518,7 +527,7 @@ manual_link (char *library) if (libname) { /* Successfully split names. Check if path is just "/" to avoid - an empty path. */ + an empty path. */ if (libname == path) { libname = library + 1; @@ -572,14 +581,17 @@ manual_link (char *library) free (path); return; } + if (process_file (real_library, library, libname, &flag, &osversion, - &soname, 0)) + &soname, 0, &stat_buf)) { error (0, 0, _("No link created since soname could not be found for %s"), library); free (path); return; } + if (soname == NULL) + soname = implicit_soname (libname, flag); create_links (real_path, path, libname, soname); free (soname); free (path); @@ -625,23 +637,7 @@ struct dlib_entry static void search_dir (const struct dir_entry *entry) { - DIR *dir; - struct dirent64 *direntry; - char *file_name, *dir_name, *real_file_name, *real_name; - int file_name_len, real_file_name_len, len; - char *soname; - struct dlib_entry *dlibs; - struct dlib_entry *dlib_ptr; - struct stat64 lstat_buf, stat_buf; - int is_link, is_dir; uint64_t hwcap = path_hwcap (entry->path); - unsigned int osversion; - - file_name_len = PATH_MAX; - file_name = alloca (file_name_len); - - dlibs = NULL; - if (opt_verbose) { if (hwcap != 0) @@ -650,6 +646,11 @@ search_dir (const struct dir_entry *entr printf ("%s:\n", entry->path); } + char *dir_name; + char *real_file_name; + size_t real_file_name_len; + size_t file_name_len = PATH_MAX; + char *file_name = alloca (file_name_len); if (opt_chroot) { dir_name = chroot_canon (opt_chroot, entry->path); @@ -663,6 +664,7 @@ search_dir (const struct dir_entry *entr real_file_name = file_name; } + DIR *dir; if (dir_name == NULL || (dir = opendir (dir_name)) == NULL) { if (opt_verbose) @@ -672,6 +674,8 @@ search_dir (const struct dir_entry *entr return; } + struct dirent64 *direntry; + struct dlib_entry *dlibs = NULL; while ((direntry = readdir64 (dir)) != NULL) { int flag; @@ -695,7 +699,8 @@ search_dir (const struct dir_entry *entr #endif !is_hwcap_platform (direntry->d_name))) continue; - len = strlen (direntry->d_name); + + size_t len = strlen (direntry->d_name); /* Skip temporary files created by the prelink program. Files with names like these are never really DSOs we want to look at. */ if (len >= sizeof (".#prelink#") - 1) @@ -727,7 +732,10 @@ search_dir (const struct dir_entry *entr } sprintf (real_file_name, "%s/%s", dir_name, direntry->d_name); } + + struct stat64 lstat_buf; #ifdef _DIRENT_HAVE_D_TYPE + /* We optimize and try to do the lstat call only if needed. */ if (direntry->d_type != DT_UNKNOWN) lstat_buf.st_mode = DTTOIF (direntry->d_type); else @@ -738,9 +746,11 @@ search_dir (const struct dir_entry *entr continue; } - is_link = S_ISLNK (lstat_buf.st_mode); + struct stat64 stat_buf; + int is_dir; + int is_link = S_ISLNK (lstat_buf.st_mode); if (is_link) - { + { /* In case of symlink, we check if the symlink refers to a directory. */ if (__builtin_expect (stat64 (real_file_name, &stat_buf), 0)) @@ -754,6 +764,12 @@ search_dir (const struct dir_entry *entr continue; } is_dir = S_ISDIR (stat_buf.st_mode); + + /* lstat_buf is later stored, update contents. */ + lstat_buf.st_dev = stat_buf.st_dev; + lstat_buf.st_ino = stat_buf.st_ino; + lstat_buf.st_size = stat_buf.st_size; + lstat_buf.st_ctime = stat_buf.st_ctime; } else is_dir = S_ISDIR (lstat_buf.st_mode); @@ -767,36 +783,28 @@ search_dir (const struct dir_entry *entr new_entry->path = xstrdup (file_name); new_entry->flag = entry->flag; new_entry->next = NULL; - if (is_link) +#ifdef _DIRENT_HAVE_D_TYPE + /* We have filled in lstat only #ifndef + _DIRENT_HAVE_D_TYPE. Fill it in if needed. */ + if (!is_link + && direntry->d_type != DT_UNKNOWN + && __builtin_expect (lstat64 (real_file_name, &lstat_buf), 0)) { - new_entry->ino = stat_buf.st_ino; - new_entry->dev = stat_buf.st_dev; + error (0, errno, _("Cannot lstat %s"), file_name); + free (new_entry->path); + free (new_entry); + continue; } - else - { -#ifdef _DIRENT_HAVE_D_TYPE - /* We have filled in lstat only #ifndef - _DIRENT_HAVE_D_TYPE. Fill it in if needed. */ - if (direntry->d_type != DT_UNKNOWN - && __builtin_expect (lstat64 (real_file_name, &lstat_buf), - 0)) - { - error (0, errno, _("Cannot lstat %s"), file_name); - free (new_entry->path); - free (new_entry); - continue; - } #endif - - new_entry->ino = lstat_buf.st_ino; - new_entry->dev = lstat_buf.st_dev; - } + new_entry->ino = lstat_buf.st_ino; + new_entry->dev = lstat_buf.st_dev; add_single_dir (new_entry, 0); continue; } else if (!S_ISREG (lstat_buf.st_mode) && !is_link) continue; + char *real_name; if (opt_chroot && is_link) { real_name = chroot_canon (opt_chroot, file_name); @@ -810,14 +818,36 @@ search_dir (const struct dir_entry *entr else real_name = real_file_name; - if (process_file (real_name, file_name, direntry->d_name, &flag, - &osversion, &soname, is_link)) +#ifdef _DIRENT_HAVE_D_TYPE + /* Call lstat64 if not done yet. */ + if (!is_link + && direntry->d_type != DT_UNKNOWN + && __builtin_expect (lstat64 (real_file_name, &lstat_buf), 0)) { - if (real_name != real_file_name) - free (real_name); + error (0, errno, _("Cannot lstat %s"), file_name); continue; } +#endif + + /* First search whether the auxiliary cache contains this + library already and it's not changed. */ + char *soname; + unsigned int osversion; + if (!search_aux_cache (&lstat_buf, &flag, &osversion, &soname)) + { + if (process_file (real_name, file_name, direntry->d_name, &flag, + &osversion, &soname, is_link, &lstat_buf)) + { + if (real_name != real_file_name) + free (real_name); + continue; + } + else if (opt_build_cache) + add_to_aux_cache (&lstat_buf, flag, osversion, soname); + } + if (soname == NULL) + soname = implicit_soname (direntry->d_name, flag); /* A link may just point to itself. */ if (is_link) @@ -834,7 +864,7 @@ search_dir (const struct dir_entry *entr || strncmp (real_base_name, soname, len) != 0) is_link = 0; } - } + } if (real_name != real_file_name) free (real_name); @@ -849,6 +879,7 @@ search_dir (const struct dir_entry *entr && (entry->flag == FLAG_ELF_LIBC5 || entry->flag == FLAG_ELF_LIBC6)) flag = entry->flag; + /* Some sanity checks to print warnings. */ if (opt_verbose) { @@ -864,6 +895,7 @@ search_dir (const struct dir_entry *entr } /* Add library to list. */ + struct dlib_entry *dlib_ptr; for (dlib_ptr = dlibs; dlib_ptr != NULL; dlib_ptr = dlib_ptr->next) { /* Is soname already in list? */ @@ -888,12 +920,13 @@ search_dir (const struct dir_entry *entr dlib_ptr->flag = flag; else error (0, 0, _("libraries %s and %s in directory %s have same soname but different type."), - dlib_ptr->name, direntry->d_name, entry->path); + dlib_ptr->name, direntry->d_name, + entry->path); } free (dlib_ptr->name); - dlib_ptr->osversion = osversion; dlib_ptr->name = xstrdup (direntry->d_name); dlib_ptr->is_link = is_link; + dlib_ptr->osversion = osversion; } /* Don't add this library, abort loop. */ /* Also free soname, since it's dynamically allocated. */ @@ -906,10 +939,10 @@ search_dir (const struct dir_entry *entr { dlib_ptr = (struct dlib_entry *)xmalloc (sizeof (struct dlib_entry)); dlib_ptr->name = xstrdup (direntry->d_name); - dlib_ptr->flag = flag; - dlib_ptr->osversion = osversion; dlib_ptr->soname = soname; + dlib_ptr->flag = flag; dlib_ptr->is_link = is_link; + dlib_ptr->osversion = osversion; /* Add at head of list. */ dlib_ptr->next = dlibs; dlibs = dlib_ptr; @@ -920,6 +953,7 @@ search_dir (const struct dir_entry *entr /* Now dlibs contains a list of all libs - add those to the cache and created all symbolic links. */ + struct dlib_entry *dlib_ptr; for (dlib_ptr = dlibs; dlib_ptr != NULL; dlib_ptr = dlib_ptr->next) { /* Don't create links to links. */ @@ -1246,7 +1280,7 @@ main (int argc, char **argv) if (opt_chroot) { /* Canonicalize the directory name of cache_file, not cache_file, - because we'll rename a temporary cache file to it. */ + because we'll rename a temporary cache file to it. */ char *p = strrchr (cache_file, '/'); char *canon = chroot_canon (opt_chroot, p ? (*p = '\0', cache_file) : "/"); @@ -1293,10 +1327,18 @@ main (int argc, char **argv) add_system_dir (LIBDIR); } + if (! opt_ignore_aux_cache) + load_aux_cache (_PATH_LDCONFIG_AUX_CACHE); + else + init_aux_cache (); + search_dirs (); if (opt_build_cache) - save_cache (cache_file); + { + save_cache (cache_file); + save_aux_cache (_PATH_LDCONFIG_AUX_CACHE); + } return 0; } --- libc/elf/readlib.c.jj 2007-07-16 09:58:46.000000000 +0200 +++ libc/elf/readlib.c 2007-08-01 18:54:26.000000000 +0200 @@ -1,4 +1,4 @@ -/* Copyright (C) 1999-2003, 2005 Free Software Foundation, Inc. +/* Copyright (C) 1999-2003, 2005, 2007 Free Software Foundation, Inc. This file is part of the GNU C Library. Contributed by Andreas Jaeger , 1999 and Jakub Jelinek , 1999. @@ -69,7 +69,7 @@ static struct known_names known_libs[] = int process_file (const char *real_file_name, const char *file_name, const char *lib, int *flag, unsigned int *osversion, - char **soname, int is_link) + char **soname, int is_link, struct stat64 *stat_buf) { FILE *file; struct stat64 statbuf; @@ -135,7 +135,7 @@ process_file (const char *real_file_name ) { /* Aout files don't have a soname, just return the name - including the major number. */ + including the major number. */ char *copy, *major, *dot; copy = xstrdup (lib); major = strstr (copy, ".so."); @@ -175,8 +175,31 @@ process_file (const char *real_file_name munmap (file_contents, statbuf.st_size); fclose (file); + *stat_buf = statbuf; return ret; } +/* Returns made up soname if lib doesn't have explicit DT_SONAME. */ + +char * +implicit_soname (const char *lib, int flag) +{ + char *soname = xstrdup (lib); + + if ((flag & FLAG_TYPE_MASK) != FLAG_LIBC4) + return soname; + + /* Aout files don't have a soname, just return the name + including the major number. */ + char *major = strstr (soname, ".so."); + if (major) + { + char *dot = strstr (major + 4, "."); + if (dot) + *dot = '\0'; + } + return soname; +} + /* Get architecture specific version of process_elf_file. */ #include --- libc/elf/readelflib.c.jj 2007-07-03 12:36:59.000000000 +0200 +++ libc/elf/readelflib.c 2007-08-01 18:54:26.000000000 +0200 @@ -231,11 +231,5 @@ process_elf_file (const char *file_name, } } - /* We reach this point only if the file doesn't contain a DT_SONAME - or if we can't classify the library. If it doesn't have a - soname, return the name of the library. */ - if (*soname == NULL) - *soname = xstrdup (lib); - return 0; } --- libc/elf/cache.c.jj 2007-07-16 09:58:46.000000000 +0200 +++ libc/elf/cache.c 2007-08-01 19:00:30.000000000 +0200 @@ -1,4 +1,4 @@ -/* Copyright (C) 1999-2003,2005,2006 Free Software Foundation, Inc. +/* Copyright (C) 1999-2003,2005,2006,2007 Free Software Foundation, Inc. This file is part of the GNU C Library. Contributed by Andreas Jaeger , 1999. @@ -20,6 +20,7 @@ #include #include #include +#include #include #include #include @@ -80,16 +81,16 @@ print_entry (const char *lib, int flag, fputs (",x86-64", stdout); break; case FLAG_S390_LIB64: - fputs(",64bit", stdout); + fputs (",64bit", stdout); break; case FLAG_POWERPC_LIB64: - fputs(",64bit", stdout); + fputs (",64bit", stdout); break; case FLAG_MIPS64_LIBN32: - fputs(",N32", stdout); + fputs (",N32", stdout); break; case FLAG_MIPS64_LIBN64: - fputs(",64bit", stdout); + fputs (",64bit", stdout); case 0: break; default: @@ -128,19 +129,11 @@ print_entry (const char *lib, int flag, void print_cache (const char *cache_name) { - size_t cache_size; - struct stat64 st; - int fd; - unsigned int i; - struct cache_file *cache; - struct cache_file_new *cache_new = NULL; - const char *cache_data; - int format = 0; - - fd = open (cache_name, O_RDONLY); + int fd = open (cache_name, O_RDONLY); if (fd < 0) error (EXIT_FAILURE, errno, _("Can't open cache file %s\n"), cache_name); + struct stat64 st; if (fstat64 (fd, &st) < 0 /* No need to map the file if it is empty. */ || st.st_size == 0) @@ -149,14 +142,19 @@ print_cache (const char *cache_name) return; } - cache = mmap (0, st.st_size, PROT_READ, MAP_SHARED, fd, 0); + struct cache_file *cache + = mmap (NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0); if (cache == MAP_FAILED) error (EXIT_FAILURE, errno, _("mmap of cache file failed.\n")); - cache_size = st.st_size; + size_t cache_size = st.st_size; if (cache_size < sizeof (struct cache_file)) error (EXIT_FAILURE, 0, _("File is not a cache file.\n")); + struct cache_file_new *cache_new = NULL; + const char *cache_data; + int format = 0; + if (memcmp (cache->magic, CACHEMAGIC, sizeof CACHEMAGIC - 1)) { /* This can only be the new format without the old one. */ @@ -201,7 +199,7 @@ print_cache (const char *cache_name) printf (_("%d libs found in cache `%s'\n"), cache->nlibs, cache_name); /* Print everything. */ - for (i = 0; i < cache->nlibs; i++) + for (unsigned int i = 0; i < cache->nlibs; i++) print_entry (cache_data + cache->libs[i].key, cache->libs[i].flags, 0, 0, cache_data + cache->libs[i].value); @@ -212,7 +210,7 @@ print_cache (const char *cache_name) cache_new->nlibs, cache_name); /* Print everything. */ - for (i = 0; i < cache_new->nlibs; i++) + for (unsigned int i = 0; i < cache_new->nlibs; i++) print_entry (cache_data + cache_new->libs[i].key, cache_new->libs[i].flags, cache_new->libs[i].osversion, @@ -231,15 +229,11 @@ init_cache (void) entries = NULL; } - - -static -int compare (const struct cache_entry *e1, const struct cache_entry *e2) +static int +compare (const struct cache_entry *e1, const struct cache_entry *e2) { - int res; - /* We need to swap entries here to get the correct sort order. */ - res = _dl_cache_libcmp (e2->lib, e1->lib); + int res = _dl_cache_libcmp (e2->lib, e1->lib); if (res == 0) { if (e1->flags < e2->flags) @@ -267,29 +261,19 @@ int compare (const struct cache_entry *e void save_cache (const char *cache_name) { - struct cache_entry *entry; - int fd, idx_old, idx_new; - size_t total_strlen, len; - char *strings, *str, *temp_name; - struct cache_file *file_entries = NULL; - struct cache_file_new *file_entries_new = NULL; - size_t file_entries_size = 0; - size_t file_entries_new_size = 0; - unsigned int str_offset; - /* Number of cache entries. */ - int cache_entry_count = 0; - /* Number of normal cache entries. */ - int cache_entry_old_count = 0; - /* Pad for alignment of cache_file_new. */ - size_t pad; - /* The cache entries are sorted already, save them in this order. */ /* Count the length of all strings. */ /* The old format doesn't contain hwcap entries and doesn't contain libraries in subdirectories with hwcaps entries. Count therefore also all entries with hwcap == 0. */ - total_strlen = 0; + size_t total_strlen = 0; + struct cache_entry *entry; + /* Number of cache entries. */ + int cache_entry_count = 0; + /* Number of normal cache entries. */ + int cache_entry_old_count = 0; + for (entry = entries; entry != NULL; entry = entry->next) { /* Account the final NULs. */ @@ -300,8 +284,8 @@ save_cache (const char *cache_name) } /* Create the on disk cache structure. */ - /* First an array for all strings. */ - strings = (char *)xmalloc (total_strlen); + struct cache_file *file_entries = NULL; + size_t file_entries_size = 0; if (opt_format != 2) { @@ -315,25 +299,27 @@ save_cache (const char *cache_name) /* And the list of all entries in the old format. */ file_entries_size = sizeof (struct cache_file) + cache_entry_old_count * sizeof (struct file_entry); - file_entries = (struct cache_file *) xmalloc (file_entries_size); + file_entries = xmalloc (file_entries_size); /* Fill in the header. */ - memset (file_entries, 0, sizeof (struct cache_file)); + memset (file_entries, '\0', sizeof (struct cache_file)); memcpy (file_entries->magic, CACHEMAGIC, sizeof CACHEMAGIC - 1); file_entries->nlibs = cache_entry_old_count; } + struct cache_file_new *file_entries_new = NULL; + size_t file_entries_new_size = 0; + if (opt_format != 0) { /* And the list of all entries in the new format. */ file_entries_new_size = sizeof (struct cache_file_new) + cache_entry_count * sizeof (struct file_entry_new); - file_entries_new = - (struct cache_file_new *) xmalloc (file_entries_new_size); + file_entries_new = xmalloc (file_entries_new_size); /* Fill in the header. */ - memset (file_entries_new, 0, sizeof (struct cache_file_new)); + memset (file_entries_new, '\0', sizeof (struct cache_file_new)); memcpy (file_entries_new->magic, CACHEMAGIC_NEW, sizeof CACHEMAGIC_NEW - 1); memcpy (file_entries_new->version, CACHE_VERSION, @@ -343,17 +329,24 @@ save_cache (const char *cache_name) file_entries_new->len_strings = total_strlen; } - pad = ALIGN_CACHE (file_entries_size) - file_entries_size; + /* Pad for alignment of cache_file_new. */ + size_t pad = ALIGN_CACHE (file_entries_size) - file_entries_size; /* If we have both formats, we hide the new format in the strings table, we have to adjust all string indices for this so that old libc5/glibc 2 dynamic linkers just ignore them. */ + unsigned int str_offset; if (opt_format != 0) str_offset = file_entries_new_size; else str_offset = 0; - str = strings; + /* An array for all strings. */ + char *strings = xmalloc (total_strlen); + char *str = strings; + int idx_old; + int idx_new; + for (idx_old = 0, idx_new = 0, entry = entries; entry != NULL; entry = entry->next, ++idx_new) { @@ -375,21 +368,18 @@ save_cache (const char *cache_name) file_entries_new->libs[idx_new].hwcap = entry->hwcap; file_entries_new->libs[idx_new].key = str_offset; } - len = strlen (entry->lib); - str = stpcpy (str, entry->lib); - /* Account the final NUL. */ - ++str; - str_offset += len + 1; + + size_t len = strlen (entry->lib) + 1; + str = mempcpy (str, entry->lib, len); + str_offset += len; /* Then the path. */ if (opt_format != 2 && entry->hwcap == 0) file_entries->libs[idx_old].value = str_offset + pad; if (opt_format != 0) file_entries_new->libs[idx_new].value = str_offset; - len = strlen (entry->path); - str = stpcpy (str, entry->path); - /* Account the final NUL. */ - ++str; - str_offset += len + 1; + len = strlen (entry->path) + 1; + str = mempcpy (str, entry->path, len); + str_offset += len; /* Ignore entries with hwcap for old format. */ if (entry->hwcap == 0) ++idx_old; @@ -403,16 +393,12 @@ save_cache (const char *cache_name) /* Write out the cache. */ /* Write cache first to a temporary file and rename it later. */ - temp_name = xmalloc (strlen (cache_name) + 2); + char *temp_name = xmalloc (strlen (cache_name) + 2); sprintf (temp_name, "%s~", cache_name); - /* First remove an old copy if it exists. */ - if (unlink (temp_name) && errno != ENOENT) - error (EXIT_FAILURE, errno, _("Can't remove old temporary cache file %s"), - temp_name); /* Create file. */ - fd = open (temp_name, O_CREAT|O_WRONLY|O_TRUNC|O_NOFOLLOW, - S_IROTH|S_IRGRP|S_IRUSR|S_IWUSR); + int fd = open (temp_name, O_CREAT|O_WRONLY|O_TRUNC|O_NOFOLLOW, + S_IRUSR|S_IWUSR); if (fd < 0) error (EXIT_FAILURE, errno, _("Can't create temporary cache file %s"), temp_name); @@ -439,11 +425,10 @@ save_cache (const char *cache_name) error (EXIT_FAILURE, errno, _("Writing of cache data failed")); } - if (write (fd, strings, total_strlen) != (ssize_t) total_strlen) + if (write (fd, strings, total_strlen) != (ssize_t) total_strlen + || close (fd)) error (EXIT_FAILURE, errno, _("Writing of cache data failed")); - close (fd); - /* Make sure user can always read cache file */ if (chmod (temp_name, S_IROTH|S_IRGRP|S_IRUSR|S_IWUSR)) error (EXIT_FAILURE, errno, @@ -463,8 +448,6 @@ save_cache (const char *cache_name) while (entries) { entry = entries; - free (entry->path); - free (entry->lib); entries = entries->next; free (entry); } @@ -476,33 +459,29 @@ void add_to_cache (const char *path, const char *lib, int flags, unsigned int osversion, uint64_t hwcap) { - struct cache_entry *new_entry, *ptr, *prev; - char *full_path; - size_t len, i; - - new_entry = (struct cache_entry *) xmalloc (sizeof (struct cache_entry)); - - len = strlen (lib) + strlen (path) + 2; - - full_path = (char *) xmalloc (len); - snprintf (full_path, len, "%s/%s", path, lib); - - new_entry->lib = xstrdup (lib); - new_entry->path = full_path; + size_t liblen = strlen (lib) + 1; + size_t len = liblen + strlen (path) + 1; + struct cache_entry *new_entry + = xmalloc (sizeof (struct cache_entry) + liblen + len); + + new_entry->lib = memcpy ((char *) (new_entry + 1), lib, liblen); + new_entry->path = new_entry->lib + liblen; + snprintf (new_entry->path, len, "%s/%s", path, lib); new_entry->flags = flags; new_entry->osversion = osversion; new_entry->hwcap = hwcap; new_entry->bits_hwcap = 0; /* Count the number of bits set in the masked value. */ - for (i = 0; (~((1ULL << i) - 1) & hwcap) != 0 && i < 8 * sizeof (hwcap); ++i) + for (size_t i = 0; + (~((1ULL << i) - 1) & hwcap) != 0 && i < 8 * sizeof (hwcap); ++i) if ((hwcap & (1ULL << i)) != 0) ++new_entry->bits_hwcap; /* Keep the list sorted - search for right place to insert. */ - ptr = entries; - prev = entries; + struct cache_entry *ptr = entries; + struct cache_entry *prev = entries; while (ptr != NULL) { if (compare (ptr, new_entry) > 0) @@ -522,3 +501,304 @@ add_to_cache (const char *path, const ch prev->next = new_entry; } } + + +/* Auxiliary cache. */ + +struct aux_cache_entry_id +{ + uint64_t ino; + uint64_t ctime; + uint64_t size; + uint64_t dev; +}; + +struct aux_cache_entry +{ + struct aux_cache_entry_id id; + int flags; + unsigned int osversion; + int used; + char *soname; + struct aux_cache_entry *next; +}; + +#define AUX_CACHEMAGIC "glibc-ld.so.auxcache-1.0" + +struct aux_cache_file_entry +{ + struct aux_cache_entry_id id; /* Unique id of entry. */ + int32_t flags; /* This is 1 for an ELF library. */ + uint32_t soname; /* String table indice. */ + uint32_t osversion; /* Required OS version. */ + int32_t pad; +}; + +/* ldconfig maintains an auxiliary cache file that allows + only reading those libraries that have changed since the last iteration. + For this for each library some information is cached in the auxiliary + cache. */ +struct aux_cache_file +{ + char magic[sizeof AUX_CACHEMAGIC - 1]; + uint32_t nlibs; /* Number of entries. */ + uint32_t len_strings; /* Size of string table. */ + struct aux_cache_file_entry libs[0]; /* Entries describing libraries. */ + /* After this the string table of size len_strings is found. */ +}; + +static unsigned int primes[] = +{ + 1021, 2039, 4093, 8191, 16381, 32749, 65521, 131071, 262139, + 524287, 1048573, 2097143, 4194301, 8388593, 16777213, 33554393, + 67108859, 134217689, 268435399, 536870909, 1073741789, 2147483647 +}; + +static size_t aux_hash_size; +static struct aux_cache_entry **aux_hash; + +/* Simplistic hash function for aux_cache_entry_id. */ +static unsigned int +aux_cache_entry_id_hash (struct aux_cache_entry_id *id) +{ + uint64_t ret = ((id->ino * 11 + id->ctime) * 11 + id->size) * 11 + id->dev; + return ret ^ (ret >> 32); +} + +static size_t nextprime (size_t x) +{ + for (unsigned int i = 0; i < sizeof (primes) / sizeof (primes[0]); ++i) + if (primes[i] >= x) + return primes[i]; + return x; +} + +void +init_aux_cache (void) +{ + aux_hash_size = primes[3]; + aux_hash = xcalloc (aux_hash_size, sizeof (struct aux_cache_entry *)); +} + +int +search_aux_cache (struct stat64 *stat_buf, int *flags, + unsigned int *osversion, char **soname) +{ + struct aux_cache_entry_id id; + id.ino = (uint64_t) stat_buf->st_ino; + id.ctime = (uint64_t) stat_buf->st_ctime; + id.size = (uint64_t) stat_buf->st_size; + id.dev = (uint64_t) stat_buf->st_dev; + + unsigned int hash = aux_cache_entry_id_hash (&id); + struct aux_cache_entry *entry; + for (entry = aux_hash[hash % aux_hash_size]; entry; entry = entry->next) + if (id.ino == entry->id.ino + && id.ctime == entry->id.ctime + && id.size == entry->id.size + && id.dev == entry->id.dev) + { + *flags = entry->flags; + *osversion = entry->osversion; + if (entry->soname != NULL) + *soname = xstrdup (entry->soname); + else + *soname = NULL; + entry->used = 1; + return 1; + } + + return 0; +} + +static void +insert_to_aux_cache (struct aux_cache_entry_id *id, int flags, + unsigned int osversion, const char *soname, int used) +{ + size_t hash = aux_cache_entry_id_hash (id) % aux_hash_size; + struct aux_cache_entry *entry; + for (entry = aux_hash[hash]; entry; entry = entry->next) + if (id->ino == entry->id.ino + && id->ctime == entry->id.ctime + && id->size == entry->id.size + && id->dev == entry->id.dev) + abort (); + + size_t len = soname ? strlen (soname) + 1 : 0; + entry = xmalloc (sizeof (struct aux_cache_entry) + len); + entry->id = *id; + entry->flags = flags; + entry->osversion = osversion; + entry->used = used; + if (soname != NULL) + entry->soname = memcpy ((char *) (entry + 1), soname, len); + else + entry->soname = NULL; + entry->next = aux_hash[hash]; + aux_hash[hash] = entry; +} + +void +add_to_aux_cache (struct stat64 *stat_buf, int flags, + unsigned int osversion, const char *soname) +{ + struct aux_cache_entry_id id; + id.ino = (uint64_t) stat_buf->st_ino; + id.ctime = (uint64_t) stat_buf->st_ctime; + id.size = (uint64_t) stat_buf->st_size; + id.dev = (uint64_t) stat_buf->st_dev; + insert_to_aux_cache (&id, flags, osversion, soname, 1); +} + +/* Load auxiliary cache to search for unchanged entries. */ +void +load_aux_cache (const char *aux_cache_name) +{ + int fd = open (aux_cache_name, O_RDONLY); + if (fd < 0) + { + init_aux_cache (); + return; + } + + struct stat64 st; + if (fstat64 (fd, &st) < 0 || st.st_size < sizeof (struct aux_cache_file)) + { + close (fd); + init_aux_cache (); + return; + } + + size_t aux_cache_size = st.st_size; + struct aux_cache_file *aux_cache + = mmap (NULL, aux_cache_size, PROT_READ, MAP_PRIVATE, fd, 0); + if (aux_cache == MAP_FAILED + || aux_cache_size < sizeof (struct aux_cache_file) + || memcmp (aux_cache->magic, AUX_CACHEMAGIC, sizeof AUX_CACHEMAGIC - 1) + || aux_cache->nlibs < 0 + || aux_cache->nlibs >= aux_cache_size) + { + close (fd); + init_aux_cache (); + return; + } + + aux_hash_size = nextprime (aux_cache->nlibs); + aux_hash = xcalloc (aux_hash_size, sizeof (struct aux_cache_entry *)); + + const char *aux_cache_data + = (const char *) &aux_cache->libs[aux_cache->nlibs]; + for (unsigned int i = 0; i < aux_cache->nlibs; ++i) + insert_to_aux_cache (&aux_cache->libs[i].id, + aux_cache->libs[i].flags, + aux_cache->libs[i].osversion, + aux_cache->libs[i].soname == 0 + ? NULL : aux_cache_data + aux_cache->libs[i].soname, + 0); + + munmap (aux_cache, aux_cache_size); + close (fd); +} + +/* Save the contents of the auxiliary cache. */ +void +save_aux_cache (const char *aux_cache_name) +{ + /* Count the length of all sonames. We start with empty string. */ + size_t total_strlen = 1; + /* Number of cache entries. */ + int cache_entry_count = 0; + + for (size_t i = 0; i < aux_hash_size; ++i) + for (struct aux_cache_entry *entry = aux_hash[i]; + entry != NULL; entry = entry->next) + if (entry->used) + { + ++cache_entry_count; + if (entry->soname != NULL) + total_strlen += strlen (entry->soname) + 1; + } + + /* Auxiliary cache. */ + size_t file_entries_size + = sizeof (struct aux_cache_file) + + cache_entry_count * sizeof (struct aux_cache_file_entry); + struct aux_cache_file *file_entries + = xmalloc (file_entries_size + total_strlen); + + /* Fill in the header of the auxiliary cache. */ + memset (file_entries, '\0', sizeof (struct aux_cache_file)); + memcpy (file_entries->magic, AUX_CACHEMAGIC, sizeof AUX_CACHEMAGIC - 1); + + file_entries->nlibs = cache_entry_count; + file_entries->len_strings = total_strlen; + + /* Initial String offset for auxiliary cache is always after the + special empty string. */ + unsigned int str_offset = 1; + + /* An array for all strings. */ + char *str = (char *) file_entries + file_entries_size; + *str++ = '\0'; + + size_t idx = 0; + for (size_t i = 0; i < aux_hash_size; ++i) + for (struct aux_cache_entry *entry = aux_hash[i]; + entry != NULL; entry = entry->next) + if (entry->used) + { + file_entries->libs[idx].id = entry->id; + file_entries->libs[idx].flags = entry->flags; + if (entry->soname == NULL) + file_entries->libs[idx].soname = 0; + else + { + file_entries->libs[idx].soname = str_offset; + + size_t len = strlen (entry->soname) + 1; + str = mempcpy (str, entry->soname, len); + str_offset += len; + } + file_entries->libs[idx].osversion = entry->osversion; + file_entries->libs[idx++].pad = 0; + } + + /* Write out auxiliary cache file. */ + /* Write auxiliary cache first to a temporary file and rename it later. */ + + char *temp_name = xmalloc (strlen (aux_cache_name) + 2); + sprintf (temp_name, "%s~", aux_cache_name); + + /* Check that directory exists and create if needed. */ + char *dir = strdupa (aux_cache_name); + dir = dirname (dir); + + struct stat64 st; + if (stat64 (dir, &st) < 0) + { + if (mkdir (dir, 0700) < 0) + goto out_fail; + } + + /* Create file. */ + int fd = open (temp_name, O_CREAT|O_WRONLY|O_TRUNC|O_NOFOLLOW, + S_IRUSR|S_IWUSR); + if (fd < 0) + goto out_fail; + + if (write (fd, file_entries, file_entries_size + total_strlen) + != (ssize_t) (file_entries_size + total_strlen) + || close (fd)) + { + unlink (temp_name); + goto out_fail; + } + + /* Move temporary to its final location. */ + if (rename (temp_name, aux_cache_name)) + unlink (temp_name); + +out_fail: + /* Free allocated memory. */ + free (file_entries); +} --- libc/sysdeps/generic/ldconfig.h.jj 2003-03-14 06:32:49.000000000 +0100 +++ libc/sysdeps/generic/ldconfig.h 2007-08-01 18:54:26.000000000 +0200 @@ -1,4 +1,4 @@ -/* Copyright (C) 1999, 2000, 2002, 2003 Free Software Foundation, Inc. +/* Copyright (C) 1999, 2000, 2002, 2003, 2007 Free Software Foundation, Inc. This file is part of the GNU C Library. Contributed by Andreas Jaeger , 1999. @@ -35,6 +35,9 @@ #define FLAG_MIPS64_LIBN32 0x0600 #define FLAG_MIPS64_LIBN64 0x0700 +/* Name of auxiliary cache. */ +#define _PATH_LDCONFIG_AUX_CACHE "/var/cache/ldconfig/aux-cache" + /* Declared in cache.c. */ extern void print_cache (const char *cache_name); @@ -45,10 +48,24 @@ extern void save_cache (const char *cach extern void add_to_cache (const char *path, const char *lib, int flags, unsigned int osversion, uint64_t hwcap); +extern void init_aux_cache (void); + +extern void load_aux_cache (const char *aux_cache_name); + +extern int search_aux_cache (struct stat64 *stat_buf, int *flags, + unsigned int *osversion, char **soname); + +extern void add_to_aux_cache (struct stat64 *stat_buf, int flags, + unsigned int osversion, const char *soname); + +extern void save_aux_cache (const char *aux_cache_name); + /* Declared in readlib.c. */ extern int process_file (const char *real_file_name, const char *file_name, const char *lib, int *flag, unsigned int *osversion, - char **soname, int is_link); + char **soname, int is_link, struct stat64 *stat_buf); + +extern char *implicit_soname (const char *lib, int flag); /* Declared in readelflib.c. */ extern int process_elf_file (const char *file_name, const char *lib, int *flag, Jakub