Thread: JBoss-related question

Håkan Jacobsson <hakan.jacobsson 'at' relevanttraffic.com> writes:

> javax.ejb.EJBException: java.lang.RuntimeException: java.lang.OutOfMemoryError:
> PermGen space
>
> We start JBoss with -Xms2048m -Xmx2048m arguments, which , as I understand it,
> is the maximum heap size we can use

that's not related to the mx. you have to increase the permanent
generation (mnore information in java gc documentation).

--
Guillaume Cottenceau, MNC Mobile News Channel SA, an Alcatel-Lucent Company
Av. de la Gare 10, 1003 Lausanne, Switzerland - direct +41 21 317 50 36

On 1/8/08 7:49 AM, "Guillaume Cottenceau" <gc@mnc.ch> wrote:

> Håkan Jacobsson <hakan.jacobsson 'at' relevanttraffic.com> writes:
>
>> javax.ejb.EJBException: java.lang.RuntimeException:
>> java.lang.OutOfMemoryError:
>> PermGen space
>>
>> We start JBoss with -Xms2048m -Xmx2048m arguments, which , as I understand
>> it,
>> is the maximum heap size we can use
>
> that's not related to the mx. you have to increase the permanent
> generation (mnore information in java gc documentation).

Unfortunately, in my experience, increasing the PermGen space only delays
the problem.  The underlying issue seems to be related to
classloading/unloading when "hot deploying" applications (tomcat, glassfish,
jetty, etc.).  When a new version of the class is deployed, the old class is
unloaded, but apparently doesn't release its PermGen space.

Mike

Hi all,

Thanx for your comments.
Actually, yes, I've deployed without restarting JBoss a couple of times. It might be the cause.
I've restarted the server now and doing some general testing.

This might not solve the problem entirely though, since I tested stuff yesterday and all was fine.
But today when doing some more testing - the exception suddenly was thrown. Without any 'hot re-deployments' (without
anydeployment actually) since I tested yesterday. 
Very strange.


Håkan Jacobsson - System Developer
----------------------------------------------------------------

RELEVANT TRAFFIC EUROPE AB, Riddarg 17D, SE-114 57 Sthlm, Sweden

Mobile (+46) 736 56 97 58
Direct   (+46) 8 56 24 98 05
Phone to office (+46) 8 678 97 50 || Fax (+46) 8 661 19 22


-----Ursprungligt meddelande-----
Från: Michael Goldner [mailto:mgoldner@agmednet.com]
Skickat: den 8 januari 2008 14:15
Till: Guillaume Cottenceau; Håkan Jacobsson
Kopia: pgsql-jdbc@postgresql.org
Ämne: Re: [JDBC] JBoss-related question

On 1/8/08 7:49 AM, "Guillaume Cottenceau" <gc@mnc.ch> wrote:

> Håkan Jacobsson <hakan.jacobsson 'at' relevanttraffic.com> writes:
>
>> javax.ejb.EJBException: java.lang.RuntimeException:
>> java.lang.OutOfMemoryError:
>> PermGen space
>>
>> We start JBoss with -Xms2048m -Xmx2048m arguments, which , as I understand
>> it,
>> is the maximum heap size we can use
>
> that's not related to the mx. you have to increase the permanent
> generation (mnore information in java gc documentation).

Unfortunately, in my experience, increasing the PermGen space only delays
the problem.  The underlying issue seems to be related to
classloading/unloading when "hot deploying" applications (tomcat, glassfish,
jetty, etc.).  When a new version of the class is deployed, the old class is
unloaded, but apparently doesn't release its PermGen space.

Mike

Håkan Jacobsson <hakan.jacobsson 'at' relevanttraffic.com> writes:

> Hi all,
>
> Thanx for your comments.
> Actually, yes, I've deployed without restarting JBoss a couple of times. It might be the cause.
> I've restarted the server now and doing some general testing.
>
> This might not solve the problem entirely though, since I tested stuff yesterday and all was fine.
> But today when doing some more testing - the exception suddenly
> was thrown. Without any 'hot re-deployments' (without any
> deployment actually) since I tested yesterday.

Notice that you can easily monitor the size of the permanent
generation with jmap[1]. It will be then very easy to confirm or
not that the deployment process is the rootcause of the out of
memory in the permanent generation space (whether by design -
normally IIRC a class is never unloaded from the permanent
generation - or because of a bug).

In our experience using Tomcat, for which we do *not*
undeploy/deploy applications but rather restart java altogether,
we had to raise the permanent generation (and it's been a stable
fix - it does not delay the problem). Seems that having a lot of
web applications contexts, each with its own jars (hence with
classes loaded separately), and using quite a few struts and jsp
classes, brings a significant amount of data in the permanent
generation.

Notice that even Sun's documentation specifically talks about the
potential need to increase the permanent generation when using
some JSP implementations[2].

Could you please return to this list about the results of your
investigations, whether there is a bug or an issue with
undeploying/deploying in jboss? Don't forget to tell the versions
of java and jboss you're using.

This is a bit offtopic but still quite interesting, I think.

Ref:
[1] http://zarb.org/~gc/html/doc-misc.html#2007-06-06

[2] http://java.sun.com/docs/hotspot/gc5.0/gc_tuning_5.html#1.1.Other%20Considerations%7Coutline

--
Guillaume Cottenceau, MNC Mobile News Channel SA, an Alcatel-Lucent Company
Av. de la Gare 10, 1003 Lausanne, Switzerland - direct +41 21 317 50 36

Hi all,

I tried jmap on the Jboss process id and below is the result.

---------------------------------------------

Attaching to process ID 11217, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 1.6.0_03-b05

using thread-local object allocation.
Parallel GC with 2 thread(s)

Heap Configuration:
   MinHeapFreeRatio = 40
   MaxHeapFreeRatio = 70
   MaxHeapSize      = 2147483648 (2048.0MB)
   NewSize          = 2686976 (2.5625MB)
   MaxNewSize       = -65536 (-0.0625MB)
   OldSize          = 5439488 (5.1875MB)
   NewRatio         = 2
   SurvivorRatio    = 8
   PermSize         = 21757952 (20.75MB)
   MaxPermSize      = 88080384 (84.0MB)

Heap Usage:
PS Young Generation
Eden Space:
   capacity = 517079040 (493.125MB)
   used     = 86750448 (82.73167419433594MB)
   free     = 430328592 (410.39332580566406MB)
   16.777018848027566% used
From Space:
   capacity = 70189056 (66.9375MB)
   used     = 1130536 (1.0781631469726562MB)
   free     = 69058520 (65.85933685302734MB)
   1.6107012466444912% used
To Space:
   capacity = 89260032 (85.125MB)
   used     = 0 (0.0MB)
   free     = 89260032 (85.125MB)
   0.0% used
PS Old Generation
   capacity = 1431699456 (1365.375MB)
   used     = 122750176 (117.06369018554688MB)
   free     = 1308949280 (1248.3113098144531MB)
   8.573739096259041% used
PS Perm Generation
   capacity = 86114304 (82.125MB)
   used     = 86077472 (82.08987426757812MB)
   free     = 36832 (0.035125732421875MB)
   99.9572289407344% used

-------------------------------------------------

I'm not 100% on how to interpret the result - but looking at the last entry 'PS Perm Generation': 99.9%
used, it doesn't look all that good.
I did restart JBoss and then I tested some stuff, before running jmap. So, this hasn't got to do with re-deployment
withoutrestarting Jboss. 


Håkan Jacobsson - System Developer
----------------------------------------------------------------

RELEVANT TRAFFIC EUROPE AB, Riddarg 17D, SE-114 57 Sthlm, Sweden

Mobile (+46) 736 56 97 58
Direct   (+46) 8 56 24 98 05
Phone to office (+46) 8 678 97 50 || Fax (+46) 8 661 19 22


-----Ursprungligt meddelande-----
Från: Guillaume Cottenceau [mailto:gc@mnc.ch]
Skickat: den 8 januari 2008 14:43
Till: Håkan Jacobsson
Kopia: Michael Goldner; pgsql-jdbc@postgresql.org
Ämne: Re: [JDBC] JBoss-related question

Håkan Jacobsson <hakan.jacobsson 'at' relevanttraffic.com> writes:

> Hi all,
>
> Thanx for your comments.
> Actually, yes, I've deployed without restarting JBoss a couple of times. It might be the cause.
> I've restarted the server now and doing some general testing.
>
> This might not solve the problem entirely though, since I tested stuff yesterday and all was fine.
> But today when doing some more testing - the exception suddenly
> was thrown. Without any 'hot re-deployments' (without any
> deployment actually) since I tested yesterday.

Notice that you can easily monitor the size of the permanent
generation with jmap[1]. It will be then very easy to confirm or
not that the deployment process is the rootcause of the out of
memory in the permanent generation space (whether by design -
normally IIRC a class is never unloaded from the permanent
generation - or because of a bug).

In our experience using Tomcat, for which we do *not*
undeploy/deploy applications but rather restart java altogether,
we had to raise the permanent generation (and it's been a stable
fix - it does not delay the problem). Seems that having a lot of
web applications contexts, each with its own jars (hence with
classes loaded separately), and using quite a few struts and jsp
classes, brings a significant amount of data in the permanent
generation.

Notice that even Sun's documentation specifically talks about the
potential need to increase the permanent generation when using
some JSP implementations[2].

Could you please return to this list about the results of your
investigations, whether there is a bug or an issue with
undeploying/deploying in jboss? Don't forget to tell the versions
of java and jboss you're using.

This is a bit offtopic but still quite interesting, I think.

Ref:
[1] http://zarb.org/~gc/html/doc-misc.html#2007-06-06

[2] http://java.sun.com/docs/hotspot/gc5.0/gc_tuning_5.html#1.1.Other%20Considerations%7Coutline

--
Guillaume Cottenceau, MNC Mobile News Channel SA, an Alcatel-Lucent Company
Av. de la Gare 10, 1003 Lausanne, Switzerland - direct +41 21 317 50 36

Håkan Jacobsson <hakan.jacobsson 'at' relevanttraffic.com> writes:

> Hi all,
>
> I tried jmap on the Jboss process id and below is the result.

[...]

>    MaxPermSize      = 88080384 (84.0MB)

[...]

> PS Perm Generation
>    capacity = 86114304 (82.125MB)
>    used     = 86077472 (82.08987426757812MB)
>    free     = 36832 (0.035125732421875MB)
>    99.9572289407344% used
>
> -------------------------------------------------
>
> I'm not 100% on how to interpret the result - but looking at the last entry 'PS Perm Generation': 99.9%
> used, it doesn't look all that good.

It's counted compared to the currently maximum capacity, not tp
the maximum allowed capacity. You're really at 97.7% use, which
surely does not look good. You should at least increase the
MaxPermSize to 128MB, and then you should monitor that figure to
see how it behaves on your server, to make sure you don't run out
of space.

> I did restart JBoss and then I tested some stuff, before
> running jmap. So, this hasn't got to do with re-deployment
> without restarting Jboss.

It may have to do with just having a large number of classes in
your application container, as it's the case for us. It's not bad
by itself, it just needs a little more memory.

--
Guillaume Cottenceau, MNC Mobile News Channel SA, an Alcatel-Lucent Company
Av. de la Gare 10, 1003 Lausanne, Switzerland - direct +41 21 317 50 36

Guillaume,

Thanx a lot - I'll try that!

Håkan Jacobsson - System Developer
----------------------------------------------------------------

RELEVANT TRAFFIC EUROPE AB, Riddarg 17D, SE-114 57 Sthlm, Sweden

Mobile (+46) 736 56 97 58
Direct   (+46) 8 56 24 98 05
Phone to office (+46) 8 678 97 50 || Fax (+46) 8 661 19 22


-----Ursprungligt meddelande-----
Från: Guillaume Cottenceau [mailto:gc@mnc.ch]
Skickat: den 8 januari 2008 17:43
Till: Håkan Jacobsson
Kopia: Michael Goldner; pgsql-jdbc@postgresql.org
Ämne: Re: SV: [JDBC] JBoss-related question

Håkan Jacobsson <hakan.jacobsson 'at' relevanttraffic.com> writes:

> Hi all,
>
> I tried jmap on the Jboss process id and below is the result.

[...]

>    MaxPermSize      = 88080384 (84.0MB)

[...]

> PS Perm Generation
>    capacity = 86114304 (82.125MB)
>    used     = 86077472 (82.08987426757812MB)
>    free     = 36832 (0.035125732421875MB)
>    99.9572289407344% used
>
> -------------------------------------------------
>
> I'm not 100% on how to interpret the result - but looking at the last entry 'PS Perm Generation': 99.9%
> used, it doesn't look all that good.

It's counted compared to the currently maximum capacity, not tp
the maximum allowed capacity. You're really at 97.7% use, which
surely does not look good. You should at least increase the
MaxPermSize to 128MB, and then you should monitor that figure to
see how it behaves on your server, to make sure you don't run out
of space.

> I did restart JBoss and then I tested some stuff, before
> running jmap. So, this hasn't got to do with re-deployment
> without restarting Jboss.

It may have to do with just having a large number of classes in
your application container, as it's the case for us. It's not bad
by itself, it just needs a little more memory.

--
Guillaume Cottenceau, MNC Mobile News Channel SA, an Alcatel-Lucent Company
Av. de la Gare 10, 1003 Lausanne, Switzerland - direct +41 21 317 50 36

Hi all,

Ok, seems this issue is resolved with the raised MaxPermSize.
Thanx so much for helping out - Guillaume and Michael especially!

I've read that the OutOufMemoryError - PermGen Space also might come from memory leaks in the actual code. But the most
commoncause seems to be (as already suggested in this thread) deployment without restarting the server. 

In our case, I think (and hope=)) the Error occurs because of the large amount of classes used (like Guillaume
suggested).

Someone asked me to supply the Java- and JBoss versions we use, so:
Java version: 1.5.0_06
JBoss version: 4.2.2.GA

Thanx again, guys!

Håkan Jacobsson - System Developer
----------------------------------------------------------------

RELEVANT TRAFFIC EUROPE AB, Riddarg 17D, SE-114 57 Sthlm, Sweden

Mobile (+46) 736 56 97 58
Direct   (+46) 8 56 24 98 05
Phone to office (+46) 8 678 97 50 || Fax (+46) 8 661 19 22


-----Ursprungligt meddelande-----
Från: Guillaume Cottenceau [mailto:gc@mnc.ch]
Skickat: den 8 januari 2008 17:43
Till: Håkan Jacobsson
Kopia: Michael Goldner; pgsql-jdbc@postgresql.org
Ämne: Re: SV: [JDBC] JBoss-related question

Håkan Jacobsson <hakan.jacobsson 'at' relevanttraffic.com> writes:

> Hi all,
>
> I tried jmap on the Jboss process id and below is the result.

[...]

>    MaxPermSize      = 88080384 (84.0MB)

[...]

> PS Perm Generation
>    capacity = 86114304 (82.125MB)
>    used     = 86077472 (82.08987426757812MB)
>    free     = 36832 (0.035125732421875MB)
>    99.9572289407344% used
>
> -------------------------------------------------
>
> I'm not 100% on how to interpret the result - but looking at the last entry 'PS Perm Generation': 99.9%
> used, it doesn't look all that good.

It's counted compared to the currently maximum capacity, not tp
the maximum allowed capacity. You're really at 97.7% use, which
surely does not look good. You should at least increase the
MaxPermSize to 128MB, and then you should monitor that figure to
see how it behaves on your server, to make sure you don't run out
of space.

> I did restart JBoss and then I tested some stuff, before
> running jmap. So, this hasn't got to do with re-deployment
> without restarting Jboss.

It may have to do with just having a large number of classes in
your application container, as it's the case for us. It's not bad
by itself, it just needs a little more memory.

--
Guillaume Cottenceau, MNC Mobile News Channel SA, an Alcatel-Lucent Company
Av. de la Gare 10, 1003 Lausanne, Switzerland - direct +41 21 317 50 36