From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by sourceware.org (Postfix) with ESMTP id 0FAC63851C04 for ; Fri, 14 May 2021 20:42:21 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 0FAC63851C04 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-336-27aih1rNNu-18Pl65hx5vQ-1; Fri, 14 May 2021 16:42:19 -0400 X-MC-Unique: 27aih1rNNu-18Pl65hx5vQ-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 48E1D1854E24; Fri, 14 May 2021 20:42:18 +0000 (UTC) Received: from [10.10.119.226] (ovpn-119-226.rdu2.redhat.com [10.10.119.226]) by smtp.corp.redhat.com (Postfix) with ESMTP id EECB761156; Fri, 14 May 2021 20:42:17 +0000 (UTC) Subject: Re: tls access fails after runtime fix symbol lookups change From: Stan Cox To: Sultan Alsawaf Cc: systemtap@sourceware.org References: <09ad40e1-86b6-f8f9-3626-ab31f0f41e7a@redhat.com> Message-ID: Date: Fri, 14 May 2021 16:42:17 -0400 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.8.1 MIME-Version: 1.0 In-Reply-To: <09ad40e1-86b6-f8f9-3626-ab31f0f41e7a@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-5.6 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, NICE_REPLY_A, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H4, RCVD_IN_MSPIKE_WL, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: systemtap@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Systemtap mailing list List-Unsubscribe: , List-Archive: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 May 2021 20:42:22 -0000 I poked at this by adding additional debug statements: dbug_task_vma(1, ... if (res == -ESRCH /*|| vm_start + offset == addr*/) res = stap_add_vma_map_info(tsk->group_leader, addr, addr + length, offset, path, module); else if (res == 0 && vm_end + 1 == addr) res = stap_extend_vma_map_info(tsk->group_leader, vm_start, addr + length); stap -DDEBUG_TASK_FINDER=2 -DDEBUG_TASK_FINDER_VMA=2 --disable-cache --runtime=kernel -g -c tls1.x -e ' probe process.function("main") { printf("%s\n", @var("_rtld_global","/usr/lib64/ld-linux-x86-64.so.2")->_dl_ns[0]->_ns_loaded$); printf("%#lx\n",&@var("_rtld_global","/usr/lib64/ld-linux-x86-64.so.2")->_dl_ns[0]->_ns_loaded); }' and notice: The module map that tls accesses is at 0x7fb941037000 This is with vm_start+offset==addr commented out Immediately before condition in question: _stp_vma_mmap_cb:188: At vm check res 0/-3 vm_start+offset==addr 1 vm_end+1==addr 0 vm_start 0x7fb94100b000 vm_end 0x7fb94100c000 offset 0x22000 addr 0x7fb94102d000 length 0x9000 addr+length 0x7fb941036000 So both conditions are false so no stap_add_vma_map_info or stap_extend_vma_map_info and a bit later: _stp_vma_mmap_cb:158: mmap_cb: tsk 88624:88624 path /usr/lib64/ld-2.32.so, addr 0x7fb941036000, length 0x00003000, offset 0x2a000, flags 0x810087 _stp_vma_mmap_cb:188: At vm check res 0/-3 vm_start+offset==addr 0 vm_end+1==addr 0 vm_start 0x7fb94100b000 vm_end 0x7fb94100c000 offset 0x2a000 addr 0x7fb941036000 length 0x3000 addr+length 0x7fb941039000 again both conditions are false, so neither stap_add_vma_map_info nor stap_extend_vma_map_info, (so how is it stap knows about 0x7fb941037000?) works as expected {.l_addr=0, .l_name="", .l_ld=0x403dd8, .l_next=0x7fb941038750, .l_prev=0x0, .l_real=0x7fb9410381a0, .l_ns=0, .l_libname=0x7fb941038728, .l_info=[...], .l_phdr=0x400040, .l_entry=4198496, .l_phnum=12, .l_ldnum=0, .l_searchlist={...}, .l_symbolic_searchlist={...}, .l_loader=0x0, .l_versions=0x7fb940fe5530, .l_nversions=4, .l_nbuckets=1, .l_gnu_bitmask_idxbits=0, .l_gnu_shift=0, .l_gnu_bitmask=0x400350, ={...}, ={...}, .l_direct_opencount=1, .l_type=0, .l_relocated=1, .l_init_called=1, .l_globa 0x7fb941037000 Now if vm_start+offset==addr is active where the module map is at 0x7fe9fa65e000 _stp_vma_mmap_cb:188: At vm check res 0/-3 vm_start+offset==addr 1 vm_end+1==addr 0 vm_start 0x7fe9fa608000 vm_end 0x7fe9fa629000 offset 0x2a000 addr 0x7fe9fa632000 length 0x3000 addr+length 0x7fe9fa635000 In this case stap_add_vma_map_info does get called stap_add_vma_map_info:223: stap_add_vma_map_info adding '/usr/lib64/ld-2.32.so' for 83155 [7fe9fa632000-7fe9fa635000 2a000] ERROR 0x7fe9fa65e000 That seems to be the primary difference.