kawa and apache tomcat

kawa and apache tomcat

[Damien MATTEI]

hi,

i 'm developping and testing a servlet application in tomcat 8 using some kawa code byte-compiled for java, so there is pure java code  and kawa code hosted on the apache tomcat web server,
all works fine but i noticed this when stopping the application (named Sidonie) in tomcat manager, tomcat complains about this in the log files:

15-Jan-2016 11:08:49.115 SEVERE [http-nio-8080-exec-12] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [Sidonie] created a ThreadLocal with key of type [gnu.mapping.Environment.InheritedLocal] (value [gnu.mapping.Environment$InheritedLocal@3dc806b8]) and a value of type [gnu.mapping.InheritingEnvironment] (value [#<environment http-nio-8080-exec-19>]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
15-Jan-2016 11:08:49.116 SEVERE [http-nio-8080-exec-12] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [Sidonie] created a ThreadLocal with key of type [java.lang.InheritableThreadLocal] (value [java.lang.InheritableThreadLocal@7927031f]) and a value of type [kawa.standard.Scheme] (value [kawa.standard.Scheme@3070d094]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
15-Jan-2016 11:08:49.116 SEVERE [http-nio-8080-exec-12] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [Sidonie] created a ThreadLocal with key of type [gnu.mapping.ThreadLocation.ThreadLocalWithDefault] (value [gnu.mapping.ThreadLocation$ThreadLocalWithDefault@599f5a94]) and a value of type [gnu.math.IntNum] (value [10]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
15-Jan-2016 11:08:49.116 SEVERE [http-nio-8080-exec-12] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [Sidonie] created a ThreadLocal with key of type [gnu.mapping.ThreadLocation.ThreadLocalWithDefault] (value [gnu.mapping.ThreadLocation$ThreadLocalWithDefault@6b51425b]) and a value of type [gnu.kawa.io.BinaryOutPort] (value [gnu.kawa.io.BinaryOutPort@4cf02060]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

this message appear the same number of times as the kawa code is called ,for example if i reload the web page which call again the kawa code i will  have two instance of this message in log ...

for info my kawa code are called after a Scheme.registerEnvironment();

tomcat seems cannot be able to free some kawa ressource, perheaps i should free them by myself in  the code but scheme and java have their own garbage collector and normally deallocate ressource when no more in use, i use files with diffrent method too, when i do an 
open ,i close the file at end but not always when using some method such as gnu.list.blob...

any ideas?

regards,
damien 
-- 
Damien.Mattei@unice.fr, Damien.Mattei@oca.eu, UNS / OCA / CNRS

Re: kawa and apache tomcat

[Matthieu Vachon]

Hi Damien,

We've face a similar issue in the past and here a start of
explanation, anyone can correct me if they see any mistake.

You're running inside a servlet container which is responsible of
executing request reaching it. To do so, the container held a thread
pool so it can serve request.

When a request reach the container, it picks a thread from the thread
pool which is then used to execute the request. In this thread, at
some point, you are reaching your Kawa code. The kawa code stores a
few `ThreadLocal` variables which are saved in the thread given by the
servlet container. However, Kawa does not release these thread locals
when code exits his scope of action.

The request completes and the container take back the thread given to
you. I guess then that the container now sees that some thread local
were added to thread without being discarded before reclaiming the
thread and warn you about it.

Depending on what is stored in the thread local storage and if the
threads in the thread pool are renewed or not, memory leak can be
occur because of this. For example, the `gnu.mapping.Environment` set
some stuff but does not remove what it adds (as least I did not find
the call).

This is not a problem in a standard JVM application because once the
application closes, nothing needs to be done. But in a container
environment, that's another story since thread a kinda re-used over
multiple requests. On high load, this can create huge memory leaks
depending on the what is actually stored.

In our case, we are in a rather old version of Kawa. We patched our
version so that all thread locals are stored in a global structure. In
a servlet context listener, we then call a clean up method just before
finishing the request.

We also add to work around a shutdown hook register which was causing
some leaking problems in a servlet container application because it
was registered but never removed
(https://github.com/maoueh/kawa-fork/blob/b7e574c4edeaedaf657960801791534f27e6216c/gnu/kawa/io/WriterManager.java#L86).

Since you are on the latest version of Kawa, I guess fixing Kawa could
be feasible. The inverse of register scheme environment should
probably added so that it can be called once the actual request as
been processed to clean up any resources still held by Kawa.

Overall, maybe it there could be a single register scheme once the
container is loaded and again a single unregister once the container
is shutdown. I don't enough of the Kawa bootstrap process to known if
this would be easy and even feasible.

Hope this helped you a bit.

Regards,
Matt

On Fri, Jan 15, 2016 at 5:45 AM, Damien MATTEI <Damien.Mattei@unice.fr> wrote:
>
> hi,
>
> i 'm developping and testing a servlet application in tomcat 8 using some kawa code byte-compiled for java, so there is pure java code  and kawa code hosted on the apache tomcat web server,
> all works fine but i noticed this when stopping the application (named Sidonie) in tomcat manager, tomcat complains about this in the log files:
>
> 15-Jan-2016 11:08:49.115 SEVERE [http-nio-8080-exec-12] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [Sidonie] created a ThreadLocal with key of type [gnu.mapping.Environment.InheritedLocal] (value [gnu.mapping.Environment$InheritedLocal@3dc806b8]) and a value of type [gnu.mapping.InheritingEnvironment] (value [#<environment http-nio-8080-exec-19>]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
> 15-Jan-2016 11:08:49.116 SEVERE [http-nio-8080-exec-12] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [Sidonie] created a ThreadLocal with key of type [java.lang.InheritableThreadLocal] (value [java.lang.InheritableThreadLocal@7927031f]) and a value of type [kawa.standard.Scheme] (value [kawa.standard.Scheme@3070d094]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
> 15-Jan-2016 11:08:49.116 SEVERE [http-nio-8080-exec-12] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [Sidonie] created a ThreadLocal with key of type [gnu.mapping.ThreadLocation.ThreadLocalWithDefault] (value [gnu.mapping.ThreadLocation$ThreadLocalWithDefault@599f5a94]) and a value of type [gnu.math.IntNum] (value [10]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
> 15-Jan-2016 11:08:49.116 SEVERE [http-nio-8080-exec-12] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [Sidonie] created a ThreadLocal with key of type [gnu.mapping.ThreadLocation.ThreadLocalWithDefault] (value [gnu.mapping.ThreadLocation$ThreadLocalWithDefault@6b51425b]) and a value of type [gnu.kawa.io.BinaryOutPort] (value [gnu.kawa.io.BinaryOutPort@4cf02060]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
>
> this message appear the same number of times as the kawa code is called ,for example if i reload the web page which call again the kawa code i will  have two instance of this message in log ...
>
> for info my kawa code are called after a Scheme.registerEnvironment();
>
> tomcat seems cannot be able to free some kawa ressource, perheaps i should free them by myself in  the code but scheme and java have their own garbage collector and normally deallocate ressource when no more in use, i use files with diffrent method too, when i do an
> open ,i close the file at end but not always when using some method such as gnu.list.blob...
>
> any ideas?
>
> regards,
> damien
> --
> Damien.Mattei@unice.fr, Damien.Mattei@oca.eu, UNS / OCA / CNRS

Re: kawa and apache tomcat

[Per Bothner]

[-- Attachment #1: Type: text/plain, Size: 1106 bytes --]



On 01/15/2016 10:10 AM, Matthieu Vachon wrote:


> This is not a problem in a standard JVM application because once the
> application closes, nothing needs to be done. But in a container
> environment, that's another story since thread a kinda re-used over
> multiple requests. On high load, this can create huge memory leaks
> depending on the what is actually stored.

As I understand it, the problem is not normal usage, but when a servlet is re-loaded.
That creates a new ClassLoader, with a new set of Classes which causes
some complicated interaction with the GC (which I don't properly understand).

In normal usage,with no ClassLoader churn, there might be some temporary leak,
but the garbage should become collectible when a thread services a new request.

Could you try the attached patch?  I haven't actually tested it (except
to verify it compiles).  Let me know if it makes a difference.
It is possible it might *reduce* the number but not eliminate the
warnings.  If so, we would be on the right track, though with more to do.

-- 
	--Per Bothner
per@bothner.com   http://per.bothner.com/

[-- Attachment #2: servet.patch --]
[-- Type: text/x-patch, Size: 1849 bytes --]

Index: gnu/kawa/servlet/KawaServlet.java
===================================================================
--- gnu/kawa/servlet/KawaServlet.java	(revision 8666)
+++ gnu/kawa/servlet/KawaServlet.java	(working copy)
@@ -41,17 +41,9 @@
     throws ServletException, IOException
   {
     CallContext ctx = CallContext.getInstance();
-    HttpRequestContext hctx = HttpRequestContext.instance.get();
-    Context sctx;
-    if (hctx instanceof Context)
-      sctx = (Context) hctx;
-    else
-      {
-        sctx = new Context();
-        Context.setInstance(sctx);
-      }
-    sctx.init(this, request, response);
+    Context sctx = new Context(this, request, response);
     ctx.consumer = sctx.getConsumer();
+    Context.setInstance(sctx);
     // FIXME should path be relative to context?
     Path.setCurrentPath(Path.valueOf(request.getRequestURL().toString()));
     ctx.values = Values.noArgs;
@@ -100,6 +92,7 @@
       }
     finally
       {
+        Context.setInstance(null);
         sctx.servlet = null;
         sctx.request = null;
         sctx.response = null;
@@ -115,8 +108,8 @@
     ServletContext context;
     Map<String,Object> requestParameters;
 
-    public void init(HttpServlet servlet, HttpServletRequest request,
-                     HttpServletResponse response)
+    public Context(HttpServlet servlet, HttpServletRequest request,
+                   HttpServletResponse response)
     {
       if (response == null)
         throw new Error("init with null response");
@@ -317,7 +310,7 @@
     public List<String> getRequestHeaders (String name)
     {
       java.util.Enumeration<java.lang.String> e = request.getHeaders(name);
-      List<String> list = new ArrayList();
+      List<String> list = new ArrayList<String>();
       while (e.hasMoreElements())
         {
           list.add(e.nextElement());

Re: kawa and apache tomcat

[Damien MATTEI]

Hi Mathieu and Per,

sorry for my late answer but i was outside from office for few days,
thank you for your answers, i tried a patch on  the 2.A sources of kawa,  just have few problem to patch it ( i hadn't use patch command for more than a decade),hope it'es the good way:
 mattei@moita kawa-2.1]$ patch < servet.patch 
(Stripping trailing CRs from patch; use --binary to disable.)
can't find file to patch at input line 5
Perhaps you should have used the -p or --strip option?
The text leading up to this was:
--------------------------
|Index: gnu/kawa/servlet/KawaServlet.java
|===================================================================
|--- gnu/kawa/servlet/KawaServlet.java  (revision 8666)
|+++ gnu/kawa/servlet/KawaServlet.java  (working copy)
--------------------------
File to patch: gnu/kawa/servlet/KawaServlet.java 
patching file gnu/kawa/servlet/KawaServlet.java

and run make to compile kawa-2.1. jar with sources patched and add it in the netbeans project the way i do it all the day to make a .war application ,deployed it on the apache tomacat8 server
but the warnings are still here in the logs when i stop the application in tomcat8:
19-Jan-2016 11:47:28.131 SEVERE [http-nio-8080-exec-11] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [Sidonie] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@1758bda9]) and a value of type [gnu.mapping.CallContext] (value [gnu.mapping.CallContext@7c41808c]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
19-Jan-2016 11:47:28.131 SEVERE [http-nio-8080-exec-11] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [Sidonie] created a ThreadLocal with key of type [gnu.mapping.ThreadLocation.ThreadLocalWithDefault] (value [gnu.mapping.ThreadLocation$ThreadLocalWithDefault@4f731308]) and a value of type [gnu.kawa.io.BinaryOutPort] (value [gnu.kawa.io.BinaryOutPort@3dcceb4c]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
19-Jan-2016 11:47:28.132 SEVERE [http-nio-8080-exec-11] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [Sidonie] created a ThreadLocal with key of type [gnu.mapping.Environment.InheritedLocal] (value [gnu.mapping.Environment$InheritedLocal@6f2bbca9]) and a value of type [gnu.mapping.InheritingEnvironment] (value [#<environment http-nio-8080-exec-19>]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
19-Jan-2016 11:47:28.132 SEVERE [http-nio-8080-exec-11] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [Sidonie] created a ThreadLocal with key of type [gnu.kawa.io.Path$1] (value [gnu.kawa.io.Path$1@4822bf88]) and a value of type [gnu.kawa.io.FilePath] (value [/root]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
19-Jan-2016 11:47:28.132 SEVERE [http-nio-8080-exec-11] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [Sidonie] created a ThreadLocal with key of type [java.lang.InheritableThreadLocal] (value [java.lang.InheritableThreadLocal@2012835b]) and a value of type [kawa.standard.Scheme] (value [kawa.standard.Scheme@4f423c7d]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
19-Jan-2016 11:47:28.133 SEVERE [http-nio-8080-exec-11] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [Sidonie] created a ThreadLocal with key of type [gnu.mapping.ThreadLocation.ThreadLocalWithDefault] (value [gnu.mapping.ThreadLocation$ThreadLocalWithDefault@12c104e2]) and a value of type [gnu.math.IntNum] (value [10]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

unless you have others ideas i will continue because i do not think it is critical for now, if under some server load it was a problem i will notice it , anyway i'm melting and compiling java pure source with scheme code in java byte-code in kawa but also in bigloo (another scheme for C and that ahave also a java front end) so if there would be a real problem
 i can easily migrate kawa scheme routines in bigloo scheme routines if it was necessary. 

Regards,
Damien

Le Friday 15 January 2016 19:41:00, vous avez écrit :
> 
> On 01/15/2016 10:10 AM, Matthieu Vachon wrote:
> 
> 
> > This is not a problem in a standard JVM application because once the
> > application closes, nothing needs to be done. But in a container
> > environment, that's another story since thread a kinda re-used over
> > multiple requests. On high load, this can create huge memory leaks
> > depending on the what is actually stored.
> 
> As I understand it, the problem is not normal usage, but when a servlet is re-loaded.
> That creates a new ClassLoader, with a new set of Classes which causes
> some complicated interaction with the GC (which I don't properly understand).
> 
> In normal usage,with no ClassLoader churn, there might be some temporary leak,
> but the garbage should become collectible when a thread services a new request.
> 
> Could you try the attached patch?  I haven't actually tested it (except
> to verify it compiles).  Let me know if it makes a difference.
> It is possible it might *reduce* the number but not eliminate the
> warnings.  If so, we would be on the right track, though with more to do.
> 



-- 
Damien.Mattei@unice.fr, Damien.Mattei@oca.eu, UNS / OCA / CNRS