* CRITICAL ls MEMORY LEAK
@ 2021-02-21 15:18 neal.garrett
2021-02-21 18:05 ` Brian Inglis
2021-02-22 20:12 ` Doug Henderson
0 siblings, 2 replies; 6+ messages in thread
From: neal.garrett @ 2021-02-21 15:18 UTC (permalink / raw)
To: cygwin
I deal with a lot of very large files on a regular basis. I've noticed that
when I delve into these directories using in mintty and issue the command ls
-l (or ls -color=auto), a very large junk of memory is consumed. The
memory leak seems to be proportionate to the number and size of files within
the containing folder.
To reproduce:
generate or use a folder containing 50 (or more) 2G+ files.
// In this demonstration, I a ran the command on a directory containing 143
files ranging in size from 2GB to 5GB.
$> free
total used free shared buff/cache
available
Mem: 50276004 16465148 33810856 0 0
33810856
Swap: 12058624 186468 11872156
$> ls -l -color=auto
. (contents displayed after some delay)
$> free
total used free shared buff/cache
available
Mem: 50276004 19844660 30431344 0 0
30431344
Swap: 12058624 186460 11872164
// After 10 consecutive executions of the 'ls -al --color=auto' command in
this directory, ls has consumed 86% of my system's real memory.
$> free
total used free shared buff/cache
available
Mem: 50276004 43587560 6688444 0 0
6688444
Swap: 12058624 301068 11757556
// If I continue (usually unknowingly) my system will completely be depleted
of resources to the point my mouse will barely respond to movement.
--
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: CRITICAL ls MEMORY LEAK
2021-02-21 15:18 CRITICAL ls MEMORY LEAK neal.garrett
@ 2021-02-21 18:05 ` Brian Inglis
2021-02-22 20:12 ` Doug Henderson
1 sibling, 0 replies; 6+ messages in thread
From: Brian Inglis @ 2021-02-21 18:05 UTC (permalink / raw)
To: cygwin
On 2021-02-21 08:18, Satalink via Cygwin wrote:
> I deal with a lot of very large files on a regular basis. I've noticed that
> when I delve into these directories using in mintty and issue the command ls
> -l (or ls -color=auto), a very large junk of memory is consumed. The
> memory leak seems to be proportionate to the number and size of files within
> the containing folder.
>
> To reproduce:
>
> generate or use a folder containing 50 (or more) 2G+ files.
>
> // In this demonstration, I a ran the command on a directory containing 143
> files ranging in size from 2GB to 5GB.
> $> free
> total used free shared buff/cache available
> Mem: 50276004 16465148 33810856 0 0 33810856
> Swap: 12058624 186468 11872156
> $> ls -l -color=auto
> . (contents displayed after some delay)
> $> free
> total used free shared buff/cache available
> Mem: 50276004 19844660 30431344 0 0 30431344
> Swap: 12058624 186460 11872164
> // After 10 consecutive executions of the 'ls -al --color=auto' command in
> this directory, ls has consumed 86% of my system's real memory.
> $> free
> total used free shared buff/cache available
> Mem: 50276004 43587560 6688444 0 0 6688444
> Swap: 12058624 301068 11757556
> // If I continue (usually unknowingly) my system will completely be depleted
> of resources to the point my mouse will barely respond to movement.
That number is just the amount of unused physical memory on the system, and will
go down as you use the system, because unused memory is wasted meory.
Better to use Windows utilities like Task Manager/Performance/Memory, Resource
Monitor/Memory, or MS/SysInternals rammap which give system relevant details.
You will probably find that a lot of your memory is in Standby which means it is
being used to memory map or cache files, and it should be released when needed.
Unfortunately Windows often can't release the memory as fast as programs want to
use it.
Just accessing files can cause AV/Defender to look at what you are doing, and
have AV and Search take a look in the files, which uses and ties up a bunch of
resources for a while.
You need to look a bit further for longer to decide if there are real issues,
and if so, where they are.
--
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada
This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: CRITICAL ls MEMORY LEAK
2021-02-21 15:18 CRITICAL ls MEMORY LEAK neal.garrett
2021-02-21 18:05 ` Brian Inglis
@ 2021-02-22 20:12 ` Doug Henderson
2021-02-22 20:30 ` Brian Inglis
1 sibling, 1 reply; 6+ messages in thread
From: Doug Henderson @ 2021-02-22 20:12 UTC (permalink / raw)
To: cygwin
On Sun, 21 Feb 2021 at 08:21, Satalink via Cygwin <cygwin@cygwin.com> wrote:
>
> I deal with a lot of very large files on a regular basis. I've noticed that
> when I delve into these directories using in mintty and issue the command ls
> -l (or ls -color=auto), a very large junk of memory is consumed. The
> memory leak seems to be proportionate to the number and size of files within
> the containing folder.
This is likely due to your virus scanner. If those files contain
non-executable content, it is probably safe to disable virus scans for
those files.
Something that ls does is triggering the scan. That scan causes the
virus scanner to read the entire file. You should see extraordinary
GPU and disk activity for some time after the ls completes. There
might be processes or at least threads for each file being scanned.
Hopefully you will be able to identify a common folder in the path to
those files where it is safe to disable scanning for that folder and
all folders and files within that folder.
HTH
Doug
--
Doug Henderson, Calgary, Alberta, Canada - from gmail.com
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: CRITICAL ls MEMORY LEAK
2021-02-22 20:12 ` Doug Henderson
@ 2021-02-22 20:30 ` Brian Inglis
2021-02-22 21:50 ` Hans-Bernhard Bröker
0 siblings, 1 reply; 6+ messages in thread
From: Brian Inglis @ 2021-02-22 20:30 UTC (permalink / raw)
To: cygwin
On 2021-02-22 13:12, Doug Henderson wrote:
> On Sun, 21 Feb 2021 at 08:21, Satalink wrote:
>> I deal with a lot of very large files on a regular basis. I've noticed that
>> when I delve into these directories using in mintty and issue the command ls
>> -l (or ls -color=auto), a very large junk of memory is consumed. The
>> memory leak seems to be proportionate to the number and size of files within
>> the containing folder.
> This is likely due to your virus scanner. If those files contain
> non-executable content, it is probably safe to disable virus scans for
> those files.
>
> Something that ls does is triggering the scan. That scan causes the
> virus scanner to read the entire file. You should see extraordinary
> GPU and disk activity for some time after the ls completes. There
> might be processes or at least threads for each file being scanned.
>
> Hopefully you will be able to identify a common folder in the path to
> those files where it is safe to disable scanning for that folder and
> all folders and files within that folder.
I've often wondered if the heavy activity is due to Windows' defaults to writing
files with F+RX perms which triggers executable virus scans?
You can't have separate directory and file default perms and unlike Unix,
Windows appears dumb about applying the X bit to files, probably because that
would render downloaded executables non-X, and there is no easy way for users to
change that, whereas Unix requires tools and users explicitly grant X perms.
--
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada
This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: CRITICAL ls MEMORY LEAK
2021-02-22 20:30 ` Brian Inglis
@ 2021-02-22 21:50 ` Hans-Bernhard Bröker
2021-02-22 23:47 ` Brian Inglis
0 siblings, 1 reply; 6+ messages in thread
From: Hans-Bernhard Bröker @ 2021-02-22 21:50 UTC (permalink / raw)
To: cygwin
Am 22.02.2021 um 21:30 schrieb Brian Inglis:
> I've often wondered if the heavy activity is due to Windows' defaults to
> writing files with F+RX perms which triggers executable virus scans?
That could only be the case if Windows actually had an 'x' permission bit.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: CRITICAL ls MEMORY LEAK
2021-02-22 21:50 ` Hans-Bernhard Bröker
@ 2021-02-22 23:47 ` Brian Inglis
0 siblings, 0 replies; 6+ messages in thread
From: Brian Inglis @ 2021-02-22 23:47 UTC (permalink / raw)
To: cygwin
On 2021-02-22 14:50, Hans-Bernhard Bröker wrote:
> Am 22.02.2021 um 21:30 schrieb Brian Inglis:
>
>> I've often wondered if the heavy activity is due to Windows' defaults to
>> writing files with F+RX perms which triggers executable virus scans?
>
> That could only be the case if Windows actually had an 'x' permission bit.
Strictly speaking, I am not sure if *Unix* has an 'x' permission bit, but most
filesystems do provide such a facility (V/FAT e.g. some /boot/ do not), as do
POSIX/Solaris ACLs, and Windows NTFS, whose ACLs have execute permissions:
$ icacls /? | egrep exec\|X
RX - read and execute access
GE - generic execute
X - execute/traverse
GE grants FILE_EXECUTE and STANDARD_RIGHTS_EXECUTE access rights.
Access Mask Format:
|31|30|29|28|27|26|25|24|23|22|21|20|19|18|17|16|15|14|13|12|11|10|9|8|7|6|5|4|3|2|1|0|
|GR|GW|GE|GA|27|26|25|AS|23|22|21|20|19|18|17|16|15|14|13|12|11|10|9|8|7|6|5|4|3|2|1|0|
|..generic..|reserved|..|.......standard........|...........object.specific...........|
GR GENERIC_READ
GW GENERIC_WRITE
GE GENERIC_EXECUTE
GA GENERIC_ALL
AS ACCESS_SYSTEM_SECURITY access Security ACL in object security descriptor
Windows also supports ACLs on system objects, and there may be similar features
on SELinux.
--
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada
This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2021-02-22 23:47 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-02-21 15:18 CRITICAL ls MEMORY LEAK neal.garrett
2021-02-21 18:05 ` Brian Inglis
2021-02-22 20:12 ` Doug Henderson
2021-02-22 20:30 ` Brian Inglis
2021-02-22 21:50 ` Hans-Bernhard Bröker
2021-02-22 23:47 ` Brian Inglis
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).