From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp-out-so.shaw.ca (smtp-out-so.shaw.ca [64.59.136.137]) by sourceware.org (Postfix) with ESMTPS id A03A6386189C for ; Sun, 21 Feb 2021 18:05:25 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org A03A6386189C Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=SystematicSw.ab.ca Authentication-Results: sourceware.org; spf=none smtp.mailfrom=brian.inglis@systematicsw.ab.ca Received: from [192.168.1.104] ([68.147.0.90]) by shaw.ca with ESMTP id Dt6hlPpNknRGtDt6ilGMea; Sun, 21 Feb 2021 11:05:24 -0700 X-Authority-Analysis: v=2.4 cv=cagXElPM c=1 sm=1 tr=0 ts=6032a0e4 a=T+ovY1NZ+FAi/xYICV7Bgg==:117 a=T+ovY1NZ+FAi/xYICV7Bgg==:17 a=IkcTkHD0fZMA:10 a=uYT-Tk0qkVT609LjNaIA:9 a=QEXdDO2ut3YA:10 Reply-To: cygwin@cygwin.com To: cygwin@cygwin.com References: <003401d70864$cd3b3400$67b19c00$@gmail.com> From: Brian Inglis Organization: Systematic Software Subject: Re: CRITICAL ls MEMORY LEAK Message-ID: <2ce762f7-b136-279b-b355-f0ff00cb99e2@SystematicSw.ab.ca> Date: Sun, 21 Feb 2021 11:05:23 -0700 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.7.1 MIME-Version: 1.0 In-Reply-To: <003401d70864$cd3b3400$67b19c00$@gmail.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-CA Content-Transfer-Encoding: 8bit X-CMAE-Envelope: MS4xfJxtKJOgz/2ZfuN9QOyFfguEqHCsQjGG9R0jmtjD2prcFE2o/IJ1nuzLgLl71zPm2d3WWbNzxRB+fhv1al3GtYk9Axa8IN7idhDWBKP5h8KrJ+5WWaUZ i7tKRzGYiJW4AEKdE/hU3fOzWo5MRQ45+Dho+6N+P9NgtasDa2/2nY2R695FeddJtXn9Dphii1vv6F5dR4jCTtyxWvgDrvJb+ck= X-Spam-Status: No, score=1.0 required=5.0 tests=BAYES_00, KAM_DMARC_STATUS, KAM_LAZY_DOMAIN_SECURITY, NICE_REPLY_A, RCVD_IN_BARRACUDACENTRAL, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, SPF_HELO_NONE, SPF_NONE, TXREP autolearn=no autolearn_force=no version=3.4.2 X-Spam-Level: * X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: cygwin@cygwin.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: General Cygwin discussions and problem reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Feb 2021 18:05:27 -0000 On 2021-02-21 08:18, Satalink via Cygwin wrote: > I deal with a lot of very large files on a regular basis. I've noticed that > when I delve into these directories using in mintty and issue the command ls > -l (or ls -color=auto), a very large junk of memory is consumed. The > memory leak seems to be proportionate to the number and size of files within > the containing folder. > > To reproduce: > > generate or use a folder containing 50 (or more) 2G+ files. > > // In this demonstration, I a ran the command on a directory containing 143 > files ranging in size from 2GB to 5GB. > $> free > total used free shared buff/cache available > Mem: 50276004 16465148 33810856 0 0 33810856 > Swap: 12058624 186468 11872156 > $> ls -l -color=auto > . (contents displayed after some delay) > $> free > total used free shared buff/cache available > Mem: 50276004 19844660 30431344 0 0 30431344 > Swap: 12058624 186460 11872164 > // After 10 consecutive executions of the 'ls -al --color=auto' command in > this directory, ls has consumed 86% of my system's real memory. > $> free > total used free shared buff/cache available > Mem: 50276004 43587560 6688444 0 0 6688444 > Swap: 12058624 301068 11757556 > // If I continue (usually unknowingly) my system will completely be depleted > of resources to the point my mouse will barely respond to movement. That number is just the amount of unused physical memory on the system, and will go down as you use the system, because unused memory is wasted meory. Better to use Windows utilities like Task Manager/Performance/Memory, Resource Monitor/Memory, or MS/SysInternals rammap which give system relevant details. You will probably find that a lot of your memory is in Standby which means it is being used to memory map or cache files, and it should be released when needed. Unfortunately Windows often can't release the memory as fast as programs want to use it. Just accessing files can cause AV/Defender to look at what you are doing, and have AV and Search take a look in the files, which uses and ties up a bunch of resources for a while. You need to look a bit further for longer to decide if there are real issues, and if so, where they are. -- Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada This email may be disturbing to some readers as it contains too much technical detail. Reader discretion is advised. [Data in binary units and prefixes, physical quantities in SI.]