Unexpected ItemNotFoundException (cafebabe-cafe-babe-cafe-babecafebabe)

6 messages Options
Embed this post
Permalink
aasoj j

Unexpected ItemNotFoundException (cafebabe-cafe-babe-cafe-babecafebabe)

Reply Threaded More More options
Print post
Permalink
Hi all,

We have started seeing an unexpected error in our application.

javax.jcr.ItemNotFoundException: failed to build path of
b91e0f2a-844e-4c7e-97d9-1efa03878956: cafebabe-cafe-babe-cafe-babecafebabe
has no child entry for b91e0f2a-844e-4c7e-97d9-1efa03878956
        at
org.apache.jackrabbit.core.HierarchyManagerImpl.buildPath(HierarchyManagerImpl.java:289)
        at
org.apache.jackrabbit.core.CachingHierarchyManager.buildPath(CachingHierarchyManager.java:195)
        at
org.apache.jackrabbit.core.HierarchyManagerImpl.getPath(HierarchyManagerImpl.java:393)
        at
org.apache.jackrabbit.core.CachingHierarchyManager.getPath(CachingHierarchyManager.java:229)
        at
org.apache.jackrabbit.core.ItemImpl.getPrimaryPath(ItemImpl.java:213)
        at
org.apache.jackrabbit.core.NodeImpl.getPrimaryPath(NodeImpl.java:3240)

JR version 1.5.5 (cluster deployment, 2 nodes)
MySQL persistence
java version 1.6.0_06 (64 bit)

We tried restarting the application, deleting indexes. But the exception is
seen every time. Sessions are not shared, i.e. a new session is created for
every read or write. The tree has very few nodes, and we were adding 4 nodes
sequentially in the same session.

We are not able to reproduce this issue on a fresh setup.

Please see if you could answer the following questions:
1. How do we recover from this state?
2. How do we prevent this situation in future?


Regards
aasoj
aasoj j

Re: Unexpected ItemNotFoundException (cafebabe-cafe-babe-cafe-babecafebabe)

Reply Threaded More More options
Print post
Permalink
Hi,

Could this "(JCR-2171) Deadlock in SharedItemStateManager on session.move
and node.save" be causing this issue below?

"Jukka Zitting updated JCR-2171:
-------------------------------

    Attachment: JCR-2171.patch

Yes, I can see the issue you're encountering.

The cause of the problem is the fact that instead of simply evicting
modified entries when receiving node state updates, the
CachingHierarchyManager tries to read the modified states and update the
cache accordingly. It seems that this behaviour was added (and is needed?)
for the shareable node feature."



The primary question is, how is the repository recovered?


Thanks and Regards
aasoj


On Wed, Jul 8, 2009 at 7:28 PM, aasoj j <[hidden email]> wrote:

> Hi all,
>
> We have started seeing an unexpected error in our application.
>
> javax.jcr.ItemNotFoundException: failed to build path of
> b91e0f2a-844e-4c7e-97d9-1efa03878956: cafebabe-cafe-babe-cafe-babecafebabe
> has no child entry for b91e0f2a-844e-4c7e-97d9-1efa03878956
>         at
> org.apache.jackrabbit.core.HierarchyManagerImpl.buildPath(HierarchyManagerImpl.java:289)
>         at
> org.apache.jackrabbit.core.CachingHierarchyManager.buildPath(CachingHierarchyManager.java:195)
>         at
> org.apache.jackrabbit.core.HierarchyManagerImpl.getPath(HierarchyManagerImpl.java:393)
>         at
> org.apache.jackrabbit.core.CachingHierarchyManager.getPath(CachingHierarchyManager.java:229)
>         at
> org.apache.jackrabbit.core.ItemImpl.getPrimaryPath(ItemImpl.java:213)
>         at
> org.apache.jackrabbit.core.NodeImpl.getPrimaryPath(NodeImpl.java:3240)
>
> JR version 1.5.5 (cluster deployment, 2 nodes)
> MySQL persistence
> java version 1.6.0_06 (64 bit)
>
> We tried restarting the application, deleting indexes. But the exception is
> seen every time. Sessions are not shared, i.e. a new session is created for
> every read or write. The tree has very few nodes, and we were adding 4 nodes
> sequentially in the same session.
>
> We are not able to reproduce this issue on a fresh setup.
>
> Please see if you could answer the following questions:
> 1. How do we recover from this state?
> 2. How do we prevent this situation in future?
>
>
> Regards
> aasoj
>
>
Sébastien Launay

Re: Unexpected ItemNotFoundException (cafebabe-cafe-babe-cafe-babecafebabe)

Reply Threaded More More options
Print post
Permalink
Hi,

I don't think this is JCR-2171 issue as the deadlock hangs the
process and does not throw any exceptions.

The UUID not found (cafebabe-cafe-babe-cafe-babecafebabe) is the
root node (jcr:root) which is mandatory and is the entry point for
reconstructing node hierarchy.

Is this error occurs only for new nodes on the cluster ?
If so, this can be the following issue:
https://issues.apache.org/jira/browse/JCR-1558

Otherwise, i think your repository is inconsistent, maybe the root node
has been lost or corrupted or the local configuration of a node is not
valid.
In this case, you may want to try restarting the node from scratch after
deleting the home repository folder (data is shared in MySQL),
this can take times because all changelog needs to be replayed for creating
the index.

aasoj j a écrit :

> Hi,
>
> Could this "(JCR-2171) Deadlock in SharedItemStateManager on session.move
> and node.save" be causing this issue below?
>
> "Jukka Zitting updated JCR-2171:
> -------------------------------
>
>     Attachment: JCR-2171.patch
>
> Yes, I can see the issue you're encountering.
>
> The cause of the problem is the fact that instead of simply evicting
> modified entries when receiving node state updates, the
> CachingHierarchyManager tries to read the modified states and update the
> cache accordingly. It seems that this behaviour was added (and is needed?)
> for the shareable node feature."
>
>
>
> The primary question is, how is the repository recovered?
>
>
> Thanks and Regards
> aasoj
>
>
> On Wed, Jul 8, 2009 at 7:28 PM, aasoj j <[hidden email]> wrote:
>
>  
>> Hi all,
>>
>> We have started seeing an unexpected error in our application.
>>
>> javax.jcr.ItemNotFoundException: failed to build path of
>> b91e0f2a-844e-4c7e-97d9-1efa03878956: cafebabe-cafe-babe-cafe-babecafebabe
>> has no child entry for b91e0f2a-844e-4c7e-97d9-1efa03878956
>>         at
>> org.apache.jackrabbit.core.HierarchyManagerImpl.buildPath(HierarchyManagerImpl.java:289)
>>         at
>> org.apache.jackrabbit.core.CachingHierarchyManager.buildPath(CachingHierarchyManager.java:195)
>>         at
>> org.apache.jackrabbit.core.HierarchyManagerImpl.getPath(HierarchyManagerImpl.java:393)
>>         at
>> org.apache.jackrabbit.core.CachingHierarchyManager.getPath(CachingHierarchyManager.java:229)
>>         at
>> org.apache.jackrabbit.core.ItemImpl.getPrimaryPath(ItemImpl.java:213)
>>         at
>> org.apache.jackrabbit.core.NodeImpl.getPrimaryPath(NodeImpl.java:3240)
>>
>> JR version 1.5.5 (cluster deployment, 2 nodes)
>> MySQL persistence
>> java version 1.6.0_06 (64 bit)
>>
>> We tried restarting the application, deleting indexes. But the exception is
>> seen every time. Sessions are not shared, i.e. a new session is created for
>> every read or write. The tree has very few nodes, and we were adding 4 nodes
>> sequentially in the same session.
>>
>> We are not able to reproduce this issue on a fresh setup.
>>
>> Please see if you could answer the following questions:
>> 1. How do we recover from this state?
>> 2. How do we prevent this situation in future?
>>
>>
>> Regards
>> aasoj
>>
>>
>>    
>
>  


--
Sébastien Launay
Alexander Klimetschek

Re: Unexpected ItemNotFoundException (cafebabe-cafe-babe-cafe-babecafebabe)

Reply Threaded More More options
Print post
Permalink
On Thu, Jul 9, 2009 at 10:11 AM, Sébastien
Launay<[hidden email]> wrote:
> The UUID not found (cafebabe-cafe-babe-cafe-babecafebabe) is the
> root node (jcr:root) which is mandatory and is the entry point for
> reconstructing node hierarchy.

No, it's not the root node that is not found, the subject of the
thread is incorrect. The exception says:

javax.jcr.ItemNotFoundException: failed to build path of
b91e0f2a-844e-4c7e-97d9-1efa03878956: cafebabe-cafe-babe-cafe-babecafebabe
has no child entry for b91e0f2a-844e-4c7e-97d9-1efa03878956

Which means that the root node doesn't have b91* as a child, whereas
b91* seems to have the root node as parent, ie. it is an orphaned
node.

But I don't know where the issue could be. Maybe you try with the
latest Jackrabbit 1.5.6 and see if the issue persists.

Regards,
Alex

--
Alexander Klimetschek
[hidden email]
aasoj j

Re: Unexpected ItemNotFoundException (cafebabe-cafe-babe-cafe-babecafebabe)

Reply Threaded More More options
Print post
Permalink
In reply to this post by Sébastien Launay
Hi,

Thanks much for your help. I think my mails subject was misleading.
Repository initialization was fine. I got this message for one of the
children nodes of root node. The error log goes like this:

javax.jcr.ItemNotFoundException: failed to build path of
b91e0f2a-844e-4c7e-97d9-1efa03878956: cafebabe-cafe-babe-cafe-babecafebabe
has no child entry for b91e0f2a-844e-4c7e-97d9-1efa03878956

If I understand it correctly, I root's child node entry was missing.

How can I recover the repository from this state without recreating the
whole tree?

Thanks again
Regards
aasoj

On Thu, Jul 9, 2009 at 1:41 PM, Sébastien Launay <
[hidden email]> wrote:

> Hi,
>
> I don't think this is JCR-2171 issue as the deadlock hangs the
> process and does not throw any exceptions.
>
> The UUID not found (cafebabe-cafe-babe-cafe-babecafebabe) is the
> root node (jcr:root) which is mandatory and is the entry point for
> reconstructing node hierarchy.
>
> Is this error occurs only for new nodes on the cluster ?
> If so, this can be the following issue:
> https://issues.apache.org/jira/browse/JCR-1558
>
> Otherwise, i think your repository is inconsistent, maybe the root node
> has been lost or corrupted or the local configuration of a node is not
> valid.
> In this case, you may want to try restarting the node from scratch after
> deleting the home repository folder (data is shared in MySQL),
> this can take times because all changelog needs to be replayed for creating
> the index.
>
> aasoj j a écrit :
> > Hi,
> >
> > Could this "(JCR-2171) Deadlock in SharedItemStateManager on session.move
> > and node.save" be causing this issue below?
> >
> > "Jukka Zitting updated JCR-2171:
> > -------------------------------
> >
> >     Attachment: JCR-2171.patch
> >
> > Yes, I can see the issue you're encountering.
> >
> > The cause of the problem is the fact that instead of simply evicting
> > modified entries when receiving node state updates, the
> > CachingHierarchyManager tries to read the modified states and update the
> > cache accordingly. It seems that this behaviour was added (and is
> needed?)
> > for the shareable node feature."
> >
> >
> >
> > The primary question is, how is the repository recovered?
> >
> >
> > Thanks and Regards
> > aasoj
> >
> >
> > On Wed, Jul 8, 2009 at 7:28 PM, aasoj j <[hidden email]> wrote:
> >
> >
> >> Hi all,
> >>
> >> We have started seeing an unexpected error in our application.
> >>
> >> javax.jcr.ItemNotFoundException: failed to build path of
> >> b91e0f2a-844e-4c7e-97d9-1efa03878956:
> cafebabe-cafe-babe-cafe-babecafebabe
> >> has no child entry for b91e0f2a-844e-4c7e-97d9-1efa03878956
> >>         at
> >>
> org.apache.jackrabbit.core.HierarchyManagerImpl.buildPath(HierarchyManagerImpl.java:289)
> >>         at
> >>
> org.apache.jackrabbit.core.CachingHierarchyManager.buildPath(CachingHierarchyManager.java:195)
> >>         at
> >>
> org.apache.jackrabbit.core.HierarchyManagerImpl.getPath(HierarchyManagerImpl.java:393)
> >>         at
> >>
> org.apache.jackrabbit.core.CachingHierarchyManager.getPath(CachingHierarchyManager.java:229)
> >>         at
> >> org.apache.jackrabbit.core.ItemImpl.getPrimaryPath(ItemImpl.java:213)
> >>         at
> >> org.apache.jackrabbit.core.NodeImpl.getPrimaryPath(NodeImpl.java:3240)
> >>
> >> JR version 1.5.5 (cluster deployment, 2 nodes)
> >> MySQL persistence
> >> java version 1.6.0_06 (64 bit)
> >>
> >> We tried restarting the application, deleting indexes. But the exception
> is
> >> seen every time. Sessions are not shared, i.e. a new session is created
> for
> >> every read or write. The tree has very few nodes, and we were adding 4
> nodes
> >> sequentially in the same session.
> >>
> >> We are not able to reproduce this issue on a fresh setup.
> >>
> >> Please see if you could answer the following questions:
> >> 1. How do we recover from this state?
> >> 2. How do we prevent this situation in future?
> >>
> >>
> >> Regards
> >> aasoj
> >>
> >>
> >>
> >
> >
>
>
> --
> Sébastien Launay
>
aasoj j

Re: Unexpected ItemNotFoundException (cafebabe-cafe-babe-cafe-babecafebabe)

Reply Threaded More More options
Print post
Permalink
In reply to this post by Alexander Klimetschek
Hi Alex,

Apologies for the confusing subject. You have identified the problem
correctly.

i also noticed the following logs ...

2009-07-07 21:12:58 ERROR
org.apache.jackrabbit.core.persistence.bundle.BundleDbPersistenceManager:1395
rollback failed
2009-07-07 21:12:58 ERROR
org.apache.jackrabbit.core.persistence.bundle.BundleDbPersistenceManager:1397
Reason: Connection.close() has already been called. Invalid operation in
this state.
2009-07-07 21:12:58 ERROR
org.apache.jackrabbit.core.persistence.bundle.BundleDbPersistenceManager:1398
State/Code: 08003/0
2009-07-07 21:12:58 ERROR
org.apache.jackrabbit.core.persistence.bundle.util.ConnectionRecoveryManager:453
failed to close connection, reason: Connection.close() has already been
called. Invalid operation in this state., state/code: 08003/0
2009-07-07 21:12:59 INFO
org.apache.jackrabbit.core.persistence.bundle.util.ConnectionRecoveryManager:345
Database: MySQL / 5.1.30-Yahoo-SMP-log
2009-07-07 21:12:59 INFO
org.apache.jackrabbit.core.persistence.bundle.util.ConnectionRecoveryManager:346
Driver: MySQL-AB JDBC Driver / mysql-connector-java-5.1.6 ( Revision:
${svn.Revision} )

When can this happen?

For me, even more important is to get fix the repository once this happens?


Thanks much

Regards
aasoj



On Thu, Jul 9, 2009 at 2:08 PM, Alexander Klimetschek <[hidden email]>wrote:

> On Thu, Jul 9, 2009 at 10:11 AM, Sébastien
> Launay<[hidden email]> wrote:
> > The UUID not found (cafebabe-cafe-babe-cafe-babecafebabe) is the
> > root node (jcr:root) which is mandatory and is the entry point for
> > reconstructing node hierarchy.
>
> No, it's not the root node that is not found, the subject of the
> thread is incorrect. The exception says:
>
> javax.jcr.ItemNotFoundException: failed to build path of
> b91e0f2a-844e-4c7e-97d9-1efa03878956: cafebabe-cafe-babe-cafe-babecafebabe
> has no child entry for b91e0f2a-844e-4c7e-97d9-1efa03878956
>
> Which means that the root node doesn't have b91* as a child, whereas
> b91* seems to have the root node as parent, ie. it is an orphaned
> node.
>
> But I don't know where the issue could be. Maybe you try with the
> latest Jackrabbit 1.5.6 and see if the issue persists.
>
> Regards,
> Alex
>
> --
> Alexander Klimetschek
> [hidden email]
>