Skip to main content

[jsr338-experts] Re: new sub-thread about handing "deadlock detected" error from the database...

  • From: Scott Marlow <smarlow@...>
  • To: "Ireland, Evan" <evan.ireland@...>
  • Cc: "jsr338-experts@..." <jsr338-experts@...>
  • Subject: [jsr338-experts] Re: new sub-thread about handing "deadlock detected" error from the database...
  • Date: Wed, 20 Mar 2013 08:53:27 -0400

Evan,

From my point of view, there are multiple aspects to this discussion. Some of the aspects could be handled by non-JPA technologies that may not exist yet. I think that the problem of avoiding concurrent EntityManager invocations has been around for a long time but just didn't become a priority until now.

To meet the new JPA 2.1 requirements, the thread-unsafe EntityManager needs to be protected from concurrent access, however that can be accomplished by the container/provider respectfully.

I think that some of the related aspects are:

1. Informing the application that the tx timed out so that the *state of the world* may be collected (this is more of a management aspect than JPA IMO). Currently, this is already possible (in vendor specific ways) and I'm not sure if we would want to standardize the notification of this event occurring somewhere (in some EE.future version). In a previous (pre EE) Java application server that I'm familiar with, I liked how application code could receive notifications of events like this (e.g. applications could be notified if a database connection went bad so that the application could help notify someone to restart the db server).

2. Having a consistent way for the application thread to learn that a tx timeout occurred. I think that this was the point of your use case below but I don't believe the application will always see an IllegalStateException, although it could see an ISE. The application will eventually see that the transaction ended but that might not happen until the CMT method returns (or BMT logic tries to end the TX that no longer exists). Previous to the new 2.1 requirement for handling concurrency, applications could of seen various IllegalState like exceptions (e.g. NullPointerException). Can we depend on the application *eventually* seeing that the tx disappeared when the application tries to end the TX? Do we need more than that (in some EE.future version?)

Scott

On 03/14/2013 04:09 PM, Ireland, Evan wrote:
Scott,

I did not mean to suggest that DBMS-issued rollback would cause a concurrency 
problem with the EntityManager. But I shared that as an example of something 
that might affect the state of the current transaction unbeknownst to the 
application thread. We saw it with CMP entity beans when working with the 
SPECjAppServer2004 performance benchmark a few years back. I don't have any 
current test cases to share.

If we come back to JPA, I am concerned not of a concurrency issue that two threads are using the 
EntityManager at the same time, but a scenario which I try to emphasize with examples (T1 is 
"application" thread, T2 is "background" thread).

   T1: EM.doSomething(); // suceeeds
   T1: EM.close(); // succeeds
   T1: EM.doSomething(); // throws IllegalStateException, because application 
thread has violated the contract of using the EM after close

Now we fix the app code:

   T1: EM.doSomething(); // suceeeds
   T1: EM.doSomething(); // succeeds
   T1: EM.close(); // succeeds

All good. Many (most) applications will not be coded to catch 
IllegalStateException on EM calls as those applications are not 
(particularly) buggy, and therefore don't need to cope with handling with 
their own bugs :-)

Now one day, due to TX timeout, we see a new pattern:

   T1: EM.doSomething(); // suceeeds
   T2: EM.close(); // succeeds (note this is done by background thread)
   T1: EM.doSomething(); // throws IllegalStateException, because background thread 
"unexpectedly" messed with the EM

What I am saying is that if we allow T2 to 'close' the EM while T1 may be using it (not necessarily 
"concurrently", but even in a sequential manner), then the final call should throw a PersistenceException 
("TX timed out"), not an IllegalStateException ("Bad API usage"). Thread T1 has not "done 
anything wrong", it is just unlucky, and unluckiness deserves to be distinguished. Another way to look at this 
is that the fact that the TX timeout just introduced another reason for PersistenceExceptions to be thrown (not much 
different from when an engineer trips over the power cord for the DBMS server, and the persistence provider cannot 
talk to it - that shouldn't result in IllegalStateException either).

Perhaps we need a different kind of "close" (e.g. "abort") on the EM that the 
background thread can call, in order to get this right. EM.isOpen() should still return true, but 
perhaps EM.isDead() or somesuch could return true.

-----Original Message-----
From: Scott Marlow [mailto:smarlow@...]
Sent: Friday, 15 March 2013 4:48 a.m.
To: jsr338-experts@...
Cc: Ireland, Evan
Subject: Re: new sub-thread about handing "deadlock detected" error from the 
database...


On 03/13/2013 09:42 PM, Ireland, Evan wrote:
Responsibility should be on the agent that calls EntityManager.close.

If the TX timed out, presumably that is the container.

If the DBMS forces rollback (e.g. deadlock), then presumably the persistence 
provider would detect that case.

When the DBMS forces rollback, there is no concurrency problem today
that I know of.  I'm not sure how clear I was before, probably not
enough.  The concurrency issue is about a background thread calling
EntityManager.close() while the application thread is in the middle of
another EntityManager.close().

We all know that EnitityManager is not required to be thread safe,
although some implementations offer thread safety as an option at the
persistence unit definition level (which could slow application
performance if enabled from the javadoc comments :).

My "stop the presses" cry about this situation, is to bring awareness
about this potential problem.  I didn't know if others were aware or
not.  Now that we are all more aware of the issue, I think (everyone's)
customers will expect us to fix the concurrency issue in our
implementations whether we come to an agreement or not, to how we can
avoid concurrency errors.

Having said that, I'm very interested in hearing about the pain that
your users are seeing and how we might be able to help when the DBMS
returns a "deadlock detected error".

If I understand your proposal, the fix would be to throw a specific
error instead of the IllegalStateException.  Do you have a link pointing
to a case that is accessible to us, that shows an example of an
application getting a "deadlock detected" error and the exception call
stack that goes with it?

Scott


-----Original Message-----
From: Linda DeMichiel [mailto:linda.demichiel@...]
Sent: Thursday, 14 March 2013 2:39 p.m.
To: jsr338-experts@...
Subject: [jsr338-experts] Re: transaction-scoped persistence context being 
closed at JTA transaction completion time from non-application thread ...



On 3/13/2013 6:35 PM, Ireland, Evan wrote:
Can we require that if the reason that EntityManager.close was called is due 
to some background (or database server) activity, then PersistenceException 
rather than IllegalStateException should be thrown by the EntityManager?


At the point at which close is called, or other?  I.e., by the container or 
the persistence provider?

-----Original Message-----
From: Linda DeMichiel [mailto:linda.demichiel@...]
Sent: Thursday, 14 March 2013 2:33 p.m.
To: jsr338-experts@...
Subject: [jsr338-experts] Re: transaction-scoped persistence context being 
closed at JTA transaction completion time from non-application thread ...



On 3/13/2013 6:26 PM, Ireland, Evan wrote:
Linda,

If containers were that easy to build we wouldn't have jobs.

We need to think of the developer who is using the EntityManager!


Yes.  Interestingly enough, however, this particularly issue hasn't been 
raised
here in the 7 years since the release of JPA 1.0 :-)

Given that the Java EE 7 release is set to close very shortly, I don't think 
we
can realistically impose this requirement on containers at this point.


-----Original Message-----
From: Linda DeMichiel [mailto:linda.demichiel@...]
Sent: Thursday, 14 March 2013 2:22 p.m.
To: jsr338-experts@...
Subject: [jsr338-experts] Re: transaction-scoped persistence context being 
closed at JTA transaction completion time from non-application thread ...



On 3/13/2013 6:16 PM, Ireland, Evan wrote:
Linda,

It would seem reasonable for EntityManager.close to be called at whatever 
time it would have been called if the transaction had not timed out (or 
deadlocked, etc).


Well, FWIW that was my initial thought.  However, this imposes more 
complexity on the container implementation, so
I'm not sure that it should be made a requirement.


-----Original Message-----
From: Linda DeMichiel [mailto:linda.demichiel@...]
Sent: Thursday, 14 March 2013 1:57 p.m.
To: jsr338-experts@...; Ireland, Evan
Subject: Re: [jsr338-experts] Re: transaction-scoped persistence context 
being closed at JTA transaction completion time from non-application thread 
...

Hi Evan,

On 3/13/2013 2:14 PM, Ireland, Evan wrote:
Folks,

I do have a concern about the case where EntityManager.close is called by a 
background thread (e.g. transaction timeout). And who knows when I should 
have brought this up, but I just thought of it now and it reminded me of some 
old related issues from entity beans with CMP.

My concern is that the application thread that is using the EntityManager may 
start receiving IllegalStateExceptions. Conceivably everywhere they use an 
EntityManager they would have to add try/catch blocks (or alter existing 
ones, if they currently code only for PersistenceException).

IllegalStateException should generally be thrown only due to an API usage error by the 
"main" application thread, i.e. the one that "owns" the EntityManager, not 
because of the action of a background thread. It makes the API very fragile if 
IllegalStateException can just start cropping up any old time, at the whim of the container.

Much preferable would be a subclass of PersistenceException that is thrown when an 
EntityManager call cannot be completed because the transaction has been marked for 
rollback (or already rolled back by either the DBMS or by the container), but any 
"close" call on the EntityManager should be application-issued, not issued by 
a background thread.

For a bit of background, we encountered a somewhat similar and very difficult case a 
few years back that never really had a wonderful solution, except for "marking the 
container's transaction for rollback" in the event of an unexpected SQLException 
(e.g. other than for duplicate key). In that case, it was observed when using entity 
beans with CMP. Let me explain...

Suppose you have a container-managed transaction, and the container uses explicit begin 
transaction / commit transaction pairs to delimit transactions. Now with some 
databases, in the case of deadlock, the DBMS may decide to rollback the transaction 
immediately, without waiting for the container to issue a commit or rollback statement. 
Now the JDBC connection is in a very interesting state: depending on the DBMS, it might 
have started a new transaction, or it might now be in auto-commit mode. Whereas the 
container thinks the DBMS transaction is still active, although it might have noticed 
that the deadlocked statement threw a SQLException. Depending on the container 
implementation, this problem can be compounded by session beans that catch exceptions 
and "eat" them (i.e. not rethrowing or wrapping them).

Now back to JPA, my take on this is:

If the DBMS unilaterally rolls back a transaction, or the container does, 
then mark the container's transaction for rollback, and ensure that 
subsequent EntityManager calls that cannot be sensibly (or safely) completed 
due to the rollback that already happened, will result in a 
PersistenceException (or some new subclass of it). Critically, leave the 
EntityManager itself alone, don't call 'close' on it, and don't throw 
IllegalStateException unexpectedly.


I understand your point about throwing the PersistenceException.  However, 
when do you expect the container
to call EntityManager.close()?

-----Original Message-----
From: Scott Marlow [mailto:smarlow@...]
Sent: Thursday, 14 March 2013 8:10 a.m.
To: jsr338-experts@...
Cc: Linda DeMichiel
Subject: [jsr338-experts] Re: transaction-scoped persistence context being 
closed at JTA transaction completion time from non-application thread ...

On 03/13/2013 02:15 PM, Linda DeMichiel wrote:
Hi Scott,

On 3/13/2013 11:04 AM, Scott Marlow wrote:
Are others responding privately perhaps? :)


No.  I wish they were responding in *any* manner!


At a minimum, I would like to state that the JTA transaction could be
rolled back from an external thread in the
following sections:


As you point out, the JTA spec already allows this, so if that is all we
were to do, I'm not sure I
see the point.   In case I am being dense though, can you tell me what
words you would like to see added to
the spec.

For 7.9.1, how about something like:

"
        After the JTA transaction has completed (either by transaction commit
or rollback), the container closes the entity manager by calling
EntityManager.close.  The JTA transaction may rollback in a background
thread (e.g. transaction timeout), in which case, the container should
arrange for the entity manager to be closed but the
EntityManager.close() should not be concurrently called while the
application is in an EntityManager invocation.
"

The 7.9.2 wording can be similar I think:
"
When the JTA transaction rolls back, the provider must detach all
managed entities if the persistence context is of type
SynchronizationType.SYNCHRONIZED or has otherwise been joined to the
transaction.  The JTA transaction may rollback in a background thread
(e.g. transaction timeout), in which case, the provider should arrange
for the managed entities to be detached from the persistence context but
not concurrently while the application is in an EntityManager invocation.
"


To me, the real issue seems to be whether we can/should provide any
guidance as to how to handle such
situations.   I'd like to get the benefit of hearing from the vendors
here as to what their implementations
do.

thanks,

-Linda


The current wording is:

"
7.9.1 Container Responsibilities
...
* After the JTA transaction has completed (either by transaction
commit or rollback), the container closes the entity
manager calling EntityManager.close.
...

7.9.2 Provider Responsibilities
...
* When the JTA transaction rolls back, the provider must detach all
managed entities if the persistence context is of
type SynchronizationType.SYNCHRONIZED or has otherwise been joined to
the transaction.
...
"


On 03/08/2013 08:31 PM, Linda DeMichiel wrote:


On 3/7/2013 4:48 PM, Scott Marlow wrote:
On 03/07/2013 05:41 PM, Linda DeMichiel wrote:


On 3/6/2013 2:16 PM, Scott Marlow wrote:
[1] requires that at Transaction completion, the container closes (or
returns to cache), the transaction-scoped
persistence context. What is supposed to happen when the JTA
transaction completes in a different thread than the
application thread? For example, if a background thread calls the
Synchronization.afterCompletion() because the tx
timeout period has been exceeded (as some Transaction Managers may
do), its not exactly thread-safe to call
EntityManager.close() (see [2]). Specifically, the application could
be in the middle of a persist or some other
EntityManager method, when EntityManager.close() is called.


The team here tells me that this should not be happening, and that the
transaction managers they are
familiar with will just mark the transaction for rollback rather than
rolling it back at the point
of timeout.

Currently, we roll the transaction back from the background timer
thread. The JTA spec [4] does allow different threads
to start/end the transaction.


Yes, I realize this is permitted.


Nevertheless, if the container were working with a TM where a timeout
did result in immediate
rollback and invocation of afterCompletion, the container should note
this, and at the point at
which the transaction would normally be completed then do the actual
close as it normally would.

Should we include a form of the above text in the JPA 2.1 spec
(section 7.9.1 [1])?


Unfortunately, I don't think this may always work, because the
container may be relying on synchronization notifications at the
normally expected tx end to know when it should be calling close
(i.e., it may not know when the tx was started). If EJB CMT were
used, the container would know when a tx was started and could use a
business method boundary as the interpositioning point. If a
container wrapped UserTransaction, I suppose it could use that point
as well, but it is not obvious to me how this would be handled
otherwise.

How does your implementation handle this?

I'd also like to hear from the other implementations here as to
what they do and how their transaction manager implementations
handle timeout.


How would we word what the provider side has to do when detaching
entities after rollback [3]? I'm not sure that the
persistence provider will have the same chance to make a note for the
container side to take action on (if there is an
EE container involved). There is also the expectation that any JPA
provider will work, with any EE container to consider.


What do your transaction manager and container do?

Related to the above, if a JTA transaction rollback occurs in a
background thread [3], how are the managed entities
expected to be detached without violating the EntityManager
thread-safety [2]?

There may be vendor specific solutions but shouldn't we (JPA spec eg)
account for the interaction of thread-unsafe
persistence contexts and the JTA Synchronization.afterCompletion that
may be invoked in non-application (background)
threads?

Scott

[1] 7.9.1 Container Responsibilities - After the JTA transaction has
completed (either by transaction commit or
rollback), the container closes the entity manager calling
EntityManager.close.

[2] 7.2 Obtaining an EntityManager - An entity manager must not be
shared among multiple concurrently executing threads,
as the entity manager and persistence context are not required to be
threadsafe. Entity managers must only be accessed
in a single-threaded manner.

[3] 7.9.2 Provider Responsibilities - When the JTA transaction rolls
back, the provider must detach all managed entities
if the persistence context is of type
SynchronizationType.SYNCHRONIZED
or has otherwise been joined to the transaction.

[4] JTA 1.1 spec 3.4.3 Thread of Control:

"
The X/Open XA interface specifies that the transaction association
related xa calls must be invoked from the same thread
context. This thread-of-control requirement is not applicable to the
object-oriented component-based application
run-time environment, in which application threads are dispatched
dynamically at method invocation time. Different Java
threads may be using the same connection resource to access the
resource manager if the connection spans multiple method
invocation. Depending on the implementation of the application server,
different Java threads may be involved with the
same XAResource object. The resource context and the transaction
context may be operated independent of thread context.
This means, for example, that it's possible for different threads to
be invoking the XAResource.start and XAResource.end
methods.

If the application server allows multiple threads to use a single
XAResource object and the associated connection to the
resource manager, it is the responsibility of the application server
to ensure that there is only one transaction
context associated with the resource at any point of time.

Thus the XAResource interface specified in this document requires that
the resource managers be able to support the
two-phase commit protocol from any thread context.
"

Scott










[jsr338-experts] Re: transaction-scoped persistence context being closed at JTA transaction completion time from non-application thread ...

(continued)

[jsr338-experts] Re: transaction-scoped persistence context being closed at JTA transaction completion time from non-application thread ...

Linda DeMichiel 03/14/2013

[jsr338-experts] Re: transaction-scoped persistence context being closed at JTA transaction completion time from non-application thread ...

Ireland, Evan 03/14/2013

[jsr338-experts] Re: transaction-scoped persistence context being closed at JTA transaction completion time from non-application thread ...

Linda DeMichiel 03/14/2013

[jsr338-experts] Re: transaction-scoped persistence context being closed at JTA transaction completion time from non-application thread ...

Ireland, Evan 03/14/2013

[jsr338-experts] Re: transaction-scoped persistence context being closed at JTA transaction completion time from non-application thread ...

Linda DeMichiel 03/14/2013

[jsr338-experts] Re: transaction-scoped persistence context being closed at JTA transaction completion time from non-application thread ...

Ireland, Evan 03/14/2013

[jsr338-experts] Re: transaction-scoped persistence context being closed at JTA transaction completion time from non-application thread ...

Linda DeMichiel 03/14/2013

[jsr338-experts] Re: transaction-scoped persistence context being closed at JTA transaction completion time from non-application thread ...

Ireland, Evan 03/14/2013

[jsr338-experts] Re: new sub-thread about handing "deadlock detected" error from the database...

Scott Marlow 03/14/2013

[jsr338-experts] Re: new sub-thread about handing "deadlock detected" error from the database...

Ireland, Evan 03/14/2013

[jsr338-experts] Re: new sub-thread about handing "deadlock detected" error from the database...

Scott Marlow 03/20/2013

[jsr338-experts] Re: transaction-scoped persistence context being closed at JTA transaction completion time from non-application thread ...

Scott Marlow 03/14/2013

[jsr338-experts] Re: transaction-scoped persistence context being closed at JTA transaction completion time from non-application thread ...

Scott Marlow 03/14/2013

[jsr338-experts] Re: transaction-scoped persistence context being closed at JTA transaction completion time from non-application thread ...

Linda DeMichiel 03/13/2013

[jsr338-experts] Re: transaction-scoped persistence context being closed at JTA transaction completion time from non-application thread ...

Ireland, Evan 03/14/2013

[jsr338-experts] Re: transaction-scoped persistence context being closed at JTA transaction completion time from non-application thread ...

gordon yorke 03/14/2013

[jsr338-experts] Re: transaction-scoped persistence context being closed at JTA transaction completion time from non-application thread ...

gordon yorke 03/14/2013

[jsr338-experts] Re: transaction-scoped persistence context being closed at JTA transaction completion time from non-application thread ...

Linda DeMichiel 03/14/2013

[jsr338-experts] Re: transaction-scoped persistence context being closed at JTA transaction completion time from non-application thread ...

Steve Ebersole 03/13/2013
 
 
Close
loading
Please Confirm
Close