Tomcat stops responding due to too many requests - using Jackrabbit

[rokham]

Hi,

I've written a web application using Jackrabbit. I have Tomcat as my web server and I'm having a strange problem when a large number of requests are sent to Tomcat simultaneously.

Here's the scenario:

When I have about 7 people accessing the server and sending about 10-15 requests at a time, the server starts off very strong and keeps going for about 5 minutes (Each of the 7 people sends 3-4 batches of requests over those 5 minutes. This means in 5 minutes we have 7 people sending about 50 requests each. Therefore a total of 350 requests). After the 5 minutes the sever slows down and about 2-3 minutes after doing the same routine, Tomcat stops responding.

I was monitoring the system and the CPU was calm and the memory was running at about 100 MB. But it's really strange that Tomcat stops responding even though the server is not doing anything. And it's not like the request has timed out. No error msg or anything is sent back from Tomcat.

I'm using Jackrabbit as my DB and I'm having a really hard time spotting the problem. I don't know if I need to tweak tomcat's configs, or something in Jackrabbit or is it something with Tomcat and Jackrabbit talking to each other, that I need to know about.

I came across the post below and the line saying It is caused by an in accessible data base.  made me wonder if there's a glitch between tomcat and jackrabbit.
http://www.mail-archive.com/[e-mail]/msg158745.html

Thank you in advance. Any help is much appreciated, since I need to have the system ready in a week.

[rokham]

Note that all these requests are  WRITING  to Jackrabbit. I also just came across this article, claiming that "Jackrabbit isn’t actually able to handle concurrent queries well".

Does this mean I should move away from Jackrabbit?

rokham wrote:

Hi,

I've written a web application using Jackrabbit. I have Tomcat as my web server and I'm having a strange problem when a large number of requests are sent to Tomcat simultaneously.

Here's the scenario:

When I have about 7 people accessing the server and sending about 10-15 requests at a time, the server starts off very strong and keeps going for about 5 minutes (Each of the 7 people sends 3-4 batches of requests over those 5 minutes. This means in 5 minutes we have 7 people sending about 50 requests each. Therefore a total of 350 requests). After the 5 minutes the sever slows down and about 2-3 minutes after doing the same routine, Tomcat stops responding.

I was monitoring the system and the CPU was calm and the memory was running at about 100 MB. But it's really strange that Tomcat stops responding even though the server is not doing anything. And it's not like the request has timed out. No error msg or anything is sent back from Tomcat.

I'm using Jackrabbit as my DB and I'm having a really hard time spotting the problem. I don't know if I need to tweak tomcat's configs, or something in Jackrabbit or is it something with Tomcat and Jackrabbit talking to each other, that I need to know about.

I came across the post below and the line saying It is caused by an in accessible data base.  made me wonder if there's a glitch between tomcat and jackrabbit.
http://www.mail-archive.com/[e-mail]/msg158745.html

Thank you in advance. Any help is much appreciated, since I need to have the system ready in a week.

[Marcel Reutegger]

Hi,

please try to get a thread dump from the jvm when tomcat does not
respond. using suns jdk 6 you can use the following tools:

jps (will give you the ids of your java processes)
jstack <pid> (gives you the thread dump)

that will help to analyze what causes the slow down.

regards
 marcel

On Thu, Nov 12, 2009 at 02:40, rokham <[hidden email]> wrote:

>
> Hi,
>
> I've written a web application using Jackrabbit. I have Tomcat as my web
> server and I'm having a strange problem when a large number of requests are
> sent to Tomcat simultaneously.
>
> Here's the scenario:
>
> When I have about 7 people accessing the server and sending about 10-15
> requests at a time, the server starts off very strong and keeps going for
> about 5 minutes (Each of the 7 people sends 3-4 batches of requests over
> those 5 minutes. This means in 5 minutes we have 7 people sending about 50
> requests each. Therefore a total of 350 requests). After the 5 minutes the
> sever slows down and about 2-3 minutes after doing the same routine, Tomcat
> stops responding.
>
> I was monitoring the system and the CPU was calm and the memory was running
> at about 100 MB. But it's really strange that Tomcat stops responding even
> though the server is not doing anything. And it's not like the request has
> timed out. No error msg or anything is sent back from Tomcat.
>
> I'm using Jackrabbit as my DB and I'm having a really hard time spotting the
> problem. I don't know if I need to tweak tomcat's configs, or something in
> Jackrabbit or is it something with Tomcat and Jackrabbit talking to each
> other, that I need to know about.
>
> I came across the post below and the line saying It is caused by an in
> accessible data base.  made me wonder if there's a glitch between tomcat and
> jackrabbit.
> http://www.mail-archive.com/tomcat-user@.../msg158745.html
>
> Thank you in advance. Any help is much appreciated, since I need to have the
> system ready in a week.
>
>
>
> --
> View this message in context: http://n4.nabble.com/Tomcat-stops-responding-due-to-too-many-requests-using-Jackrabbit-tp605488p605488.html
> Sent from the Jackrabbit - Dev mailing list archive at Nabble.com.
>

[Marcel Reutegger]

Hi,

On Thu, Nov 12, 2009 at 04:27, rokham <[hidden email]> wrote:
> Note that all these requests are  WRITING  to Jackrabbit. I also just came
> across this article, claiming that "Jackrabbit isn’t actually able to handle
> concurrent queries well".

do you mean JCR queries or SQL queries to the underlying database?

jackrabbit is able to handle and execute concurrent JCR queries.

however, writes to the underlying database are currently serialized.

regards
 marcel

[rokham]

Hi,

Thanks a bunch for your reply. I learned yet another new thing :)

I think my problem lied in my program's usage of sessions accessing Jackrabbit. I had a single session processing all the requests, but after reading a lot online, special thanks to this page:

http://n4.nabble.com/Threads-in-jackrabbit-application-td520034.html#a520034

I learned that each thread should have its own session. I made the changes and the performance shot up very significantly.

I used to be able to add/update/query Jackrabbit with about 300-500 requests (over 5 minutes) and it would come to a halt. Now I was able to perform 7,200 operations (over about 40 minutes) and it was still up and running (no glitches felt). I still need to run longer and harsher tests on the application but I feel a lot more confident in its performance.

Thanks again for your help,

Rokham

Marcel Reutegger wrote:

Hi,

On Thu, Nov 12, 2009 at 04:27, rokham <[e-mail]> wrote:
> Note that all these requests are  WRITING  to Jackrabbit. I also just came
> across this article, claiming that "Jackrabbit isn’t actually able to handle
> concurrent queries well".

do you mean JCR queries or SQL queries to the underlying database?

jackrabbit is able to handle and execute concurrent JCR queries.

however, writes to the underlying database are currently serialized.

regards
 marcel

[Stefan Guggisberg]

please don't cross-post! your question was perfectly appropriate
on the users list, no need to post it on the dev list as well...

cheers
stefan

On Fri, Nov 13, 2009 at 5:09 AM, rokham <[hidden email]> wrote:

>
> Hi,
>
> Thanks a bunch for your reply. I learned yet another new thing :)
>
> I think my problem lied in my program's usage of sessions accessing
> Jackrabbit. I had a single session processing all the requests, but after
> reading a lot online, special thanks to this page:
>
> http://n4.nabble.com/Threads-in-jackrabbit-application-td520034.html#a520034
>
> I learned that each thread should have its own session. I made the changes
> and the performance shot up very significantly.
>
> I used to be able to add/update/query Jackrabbit with about 300-500 requests
> (over 5 minutes) and it would come to a halt. Now I was able to perform
> 7,200 operations (over about 40 minutes) and it was still up and running (no
> glitches felt). I still need to run longer and harsher tests on the
> application but I feel a lot more confident in its performance.
>
> Thanks again for your help,
>
> Rokham
>
>
> Marcel Reutegger wrote:
>>
>> Hi,
>>
>> On Thu, Nov 12, 2009 at 04:27, rokham <[hidden email]> wrote:
>>> Note that all these requests are  WRITING  to Jackrabbit. I also just
>>> came
>>> across this article, claiming that "Jackrabbit isn’t actually able to
>>> handle
>>> concurrent queries well".
>>
>> do you mean JCR queries or SQL queries to the underlying database?
>>
>> jackrabbit is able to handle and execute concurrent JCR queries.
>>
>> however, writes to the underlying database are currently serialized.
>>
>> regards
>>  marcel
>>
>>
>
> --
> View this message in context: http://n4.nabble.com/Tomcat-stops-responding-due-to-too-many-requests-using-Jackrabbit-tp605488p620747.html
> Sent from the Jackrabbit - Dev mailing list archive at Nabble.com.
>

[rokham]

Hi Stefan,

Thanks for the note. Will keep that in mind in the future :)

On Fri, Nov 13, 2009 at 3:32 AM, Stefan Guggisberg [via Jackrabbit] <[hidden email]> wrote:

please don't cross-post! your question was perfectly appropriate
on the users list, no need to post it on the dev list as well...

cheers
stefan

On Fri, Nov 13, 2009 at 5:09 AM, rokham <[hidden email]> wrote:

>
> Hi,
>
> Thanks a bunch for your reply. I learned yet another new thing :)
>
> I think my problem lied in my program's usage of sessions accessing
> Jackrabbit. I had a single session processing all the requests, but after
> reading a lot online, special thanks to this page:
>
> http://n4.nabble.com/Threads-in-jackrabbit-application-td520034.html#a520034
>
> I learned that each thread should have its own session. I made the changes
> and the performance shot up very significantly.
>
> I used to be able to add/update/query Jackrabbit with about 300-500 requests
> (over 5 minutes) and it would come to a halt. Now I was able to perform
> 7,200 operations (over about 40 minutes) and it was still up and running (no
> glitches felt). I still need to run longer and harsher tests on the
> application but I feel a lot more confident in its performance.
>
> Thanks again for your help,
>
> Rokham

>

>
> Marcel Reutegger wrote:
>>
>> Hi,
>>
>> On Thu, Nov 12, 2009 at 04:27, rokham <[hidden email]> wrote:
>>> Note that all these requests are  WRITING  to Jackrabbit. I also just
>>> came
>>> across this article, claiming that "Jackrabbit isn’t actually able to
>>> handle
>>> concurrent queries well".
>>
>> do you mean JCR queries or SQL queries to the underlying database?
>>
>> jackrabbit is able to handle and execute concurrent JCR queries.
>>
>> however, writes to the underlying database are currently serialized.
>>
>> regards
>>  marcel
>>
>>
>
> --

> View this message in context: http://n4.nabble.com/Tomcat-stops-responding-due-to-too-many-requests-using-Jackrabbit-tp605488p620747.html

> Sent from the Jackrabbit - Dev mailing list archive at Nabble.com.
>