Re: [Cygwin] package-grep on Sourceware

[ASSI <Stromeko@nexgo.de>]

Hi Frank,

On Monday, November 25, 2019 1:37:01 AM CET Frank Ch. Eigler wrote:
> > you've made changes to the package-grep CGI script on sourceware
> > that pretty much remove its originally intended functionality.  The
> > original was searching the index of the package _contents_ as it
> > would get installed on the disk, while the JSON file you search now
> > only has the metadata for the packages.  That's not even
> > superficially the same thing.  [...]
> 
> OK.  As you may be aware, this simplification was done because the
> previous code's workload was too high (grepping through 600MB+ of text
> per cgi call), and we at overseers@ weren't told otherwise.

I wasn't questioning your motives, that part was pretty clear from the change 
you did.  I would like to understand what characteristic of the workload you 
were trying to avoid as to me it looks like avoiding IO comes at a significant 
cost in either memory or CPU consumption (or both).  I understand that the 
server is a fairly well equipped machine, but not in enough detail to make 
that call.

> > I've looked into what could be done to do that in a more
> > server-friendly way, but it turns out that there either isn't
> > anything that's ready-made for this or I can't find it. [...]
> 
> Thanks for looking into it.  A few other possibilities to
> ameliorate the impact of restoring the previous version:
> 
> - using xargs grep to use our extra cores, to reduce latency

Then I'd suggest to just use ripgrep (is that available?), which automatically 
spreads the work across the available cores (and can be limited to less than 
that if so desired).  Is that available on the machine?  Would it be possible 
to cache the list files in a tmpfs?  This doesn't improve performance on my 
machine, but I have an SSD and no other IO.

> - to impose a mod_qos concurrency limit on package-grep.cgi

I can't really tell what that would mean, but I guess that queries would get 
delayed if too many of them get stacked onto each other?  That would help with 
the load of course.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Waldorf MIDI Implementation & additional documentation:
http://Synth.Stromeko.net/Downloads.html#WaldorfDocs