Comments on: Java: Analyze Performance Issue

By: J.T. Wenting

J.T. Wenting — Tue, 24 Jun 2008 06:16:39 +0000

That’s not actually strange at all.
Somewhere in the code there’s a place where you’re loosing some object references in such a way that they become inaccessible but don’t become elligible for garbage collection.
It takes time for that pool of objects to eat up enough memory to become a real problem.
At 2 million transactions a day, if you’re loosing 100 bytes per transaction (not a lot), you’re still loosing 200 million bytes a day (or close to 200 megabytes), which is a lot.
Those 100 bytes per transaction could be a single reference left somewhere in a static Collection that never gets released or cleaned out.

By: admin

admin — Tue, 24 Jun 2008 02:00:30 +0000

Actually I do agree with you. When this problem happened, I went through the code (which is actually written 8 years back and recently revamped partially), check the OS patches, Java patches, collect performance data, etc and did modify certain part of the code which I suspect is the cause of the problem. Unfortunately we still cannot resolve it.
Reading through the Sun documentation on GC suggests that we can use the concurrent GC for best responsiveness, and RMI is doing a full distributed GC every minute by default. By tuning those parameters, I at least resolve the issue though I may not address the real problem which “maybe” is in the code.

The strange thing is that this problem only happened over time.

Our system is processing at 20 to 30 transaction per seconds, which means approx. 2 millions transactions in a day, and the problem happens only after a day…

By: William Louth

William Louth — Mon, 23 Jun 2008 18:51:09 +0000

I think you have the classification wrong here. GC is rarely the actual problem but the symptom of another problem and changing to other collectors and more exotic vm arguments is the majority of the time a masking of an issue with a short term temporary band aid.

Kind regards,

William