mq
  1. mq
  2. MQ-317

After untidy shutdown broker fails to start with "java.lang.IllegalStateException: File not synced. You must call Iterator to play back log file."

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 4.5.2
    • Fix Version/s: 5.0
    • Component/s: broker-core
    • Labels:
      None
    • Environment:

      Java 1.7.0_11
      Glassfish 3.1.2.2 (build 5)
      OpenMQ 4.5.2 Patch 1 (Build 3-d)

      Description

      The appserver/broker stopped processing messages and some of our queues built up a backlog of under a thousand messages. The
      appserver (and LOCAL broker) were shutdown. After restarting Glassfish and LOCAL broker, the following messages were logged by the
      broker:

      {{
      [21/Jun/2013:02:20:11 UTC] [B1278]: Update state of interest [consumer:0, type=NONE] for message 19578-10.240.101.150(f2:55:6b:e4:d1:59)-34284-1371708784242
      [21/Jun/2013:02:20:11 UTC] [B1277]: Reconstruct message 7793-10.240.100.117(d4:52:84:1:52:d0)-42626-1371708784996
      [21/Jun/2013:02:20:11 UTC] [B1278]: Update state of interest [consumer:0, type=NONE] for message 7793-10.240.100.117(d4:52:84:1:52:d0)-42626-1371708784996
      [21/Jun/2013:02:20:11 UTC] [B1277]: Reconstruct message 30506-10.240.100.141(d3:83:5c:e4:aa:cd)-55422-1371708818692
      [21/Jun/2013:02:20:11 UTC] [B1277]: Reconstruct message 7822-10.240.100.117(cd:7d:54:3b:a9:77)-42619-1371708818782
      [21/Jun/2013:02:20:11 UTC] [B1278]: Update state of interest [consumer:0, type=NONE] for message 30506-10.240.100.141(d3:83:5c:e4:aa:cd)-55422-1371708818692
      [21/Jun/2013:02:20:11 UTC] [B1278]: Update state of interest [consumer:0, type=NONE] for message 7822-10.240.100.117(cd:7d:54:3b:a9:77)-42619-1371708818782
      [21/Jun/2013:02:20:11 UTC] [B1277]: Reconstruct message 7850-10.240.100.117(db:b:2e:46:d0:cc)-43994-1371708822331
      [21/Jun/2013:02:20:11 UTC] [B1278]: Update state of interest [consumer:0, type=NONE] for message 7850-10.240.100.117(db:b:2e:46:d0:cc)-43994-1371708822331
      [21/Jun/2013:02:20:11 UTC] ERROR Mismatch in number of logged and stored consumers for 8200-10.240.100.117(ea:c8:7b:72:ca:c5)-44139-1371709438648 logged=0 stored=1
      [21/Jun/2013:02:20:11 UTC] ERROR logged interests does not contain stored interst. sysid= 8200-10.240.100.117(ea:c8:7b:72:ca:c5)-44139-1371709438648 consumerid=[consumer:0, type=NONE]
      [21/Jun/2013:02:20:11 UTC] [B1276]: Replace existing message 8200-10.240.100.117(ea:c8:7b:72:ca:c5)-44139-1371709438648
      [21/Jun/2013:02:20:11 UTC] [B1277]: Reconstruct message 8200-10.240.100.117(ea:c8:7b:72:ca:c5)-44139-1371709438648
      [21/Jun/2013:02:20:11 UTC] [B2096]: Unable to load transaction information.:
      com.sun.messaging.jmq.jmsserver.util.BrokerException: [B4212]: Message 8200-10.240.100.117(ea:c8:7b:72:ca:c5)-44139-1371709438648 already exists in destination centralEventReceiver [Queue]
      at com.sun.messaging.jmq.jmsserver.core.Destination.queueMessage(Destination.java:3171)
      at com.sun.messaging.jmq.jmsserver.core.Destination.queueMessage(Destination.java:3095)
      at com.sun.messaging.jmq.jmsserver.persist.file.TransactionLogReplayer.rerouteMessage(TransactionLogReplayer.java:194)
      at com.sun.messaging.jmq.jmsserver.persist.file.TransactionLogReplayer.replaySentMessage(TransactionLogReplayer.java:163)
      at com.sun.messaging.jmq.jmsserver.persist.file.TransactionLogReplayer.replaySentMessages(TransactionLogReplayer.java:107)
      at com.sun.messaging.jmq.jmsserver.persist.file.TransactionLogReplayer.replayTransactionWork(TransactionLogReplayer.java:92)
      at com.sun.messaging.jmq.jmsserver.persist.file.ClusterTransactionManager.replayTransactionEvent(ClusterTransactionManager.java:276)
      at com.sun.messaging.jmq.jmsserver.persist.file.TransactionLogManager.replayTransactionLogOnStartup(TransactionLogManager.java:625)
      at com.sun.messaging.jmq.jmsserver.persist.file.TransactionLogManager.startup(TransactionLogManager.java:256)
      at com.sun.messaging.jmq.jmsserver.persist.file.FileStore.init(FileStore.java:477)
      at com.sun.messaging.jmq.jmsserver.data.TransactionList.<init>(TransactionList.java:192)
      at com.sun.messaging.jmq.jmsserver.Broker._start(Broker.java:1170)
      at com.sun.messaging.jmq.jmsserver.Broker.start(Broker.java:456)
      at com.sun.messaging.jmq.jmsserver.Broker.main(Broker.java:2151)
      Caused by: java.lang.IllegalStateException: Message exist in the store
      at com.sun.messaging.jmq.util.lists.SimpleNFLHashMap.put(SimpleNFLHashMap.java:376)
      at com.sun.messaging.jmq.util.lists.SimpleNFLHashMap.put(SimpleNFLHashMap.java:291)
      at com.sun.messaging.jmq.jmsserver.core.Destination.putMessage(Destination.java:3279)
      at com.sun.messaging.jmq.jmsserver.core.Destination.queueMessage(Destination.java:3145)
      ... 13 more
      [21/Jun/2013:02:20:11 UTC] [B1239]: Using platform MBean server
      [21/Jun/2013:02:20:11 UTC] RMI Registry started on port 27776
      [21/Jun/2013:02:20:11 UTC] JESMF classes not present - JESMF support will not be enabled.
      [21/Jun/2013:02:20:11 UTC] JMX Connector Server jmxrmi started successfully with url service:jmx:rmi://appdepota.depot.test.corp.erdmg.com/jndi/rmi://10.240.100.117:27776/10.240.100.117/27676/jmxrmi
      [21/Jun/2013:02:20:11 UTC] [B1004]: Starting the admin service using tcp(host = *, port=0, mode=dedicated) with min threads 4 and max threads of 10
      [21/Jun/2013:02:20:11 UTC] [B1227]: Using file user repository for admin service connection authentication
      [21/Jun/2013:02:20:11 UTC] [B1004]: Starting the jms service using tcp(host = *, port=0, mode=dedicated) with min threads 10 and max threads of 1000
      [21/Jun/2013:02:20:11 UTC] [B1227]: Using file user repository for jms service connection authentication
      [21/Jun/2013:02:20:11 UTC] [B1069]: Running as master broker for the cluster.
      [21/Jun/2013:02:20:11 UTC] [B1383]: Received cluster configuration change records (0, 1371781211538) from mq://10.240.100.117:27676/?instName=depotclusterdepotserver1&brokerSessionUID=450992613703103488
      [21/Jun/2013:02:20:11 UTC] [B1263]: Resume full JMS service
      [21/Jun/2013:02:20:11 UTC] [B1075]: Received persistent state change records from the master broker. Ready to accept client connections.
      [21/Jun/2013:02:20:11 UTC] [B1004]: Starting the cluster service using tcp [ 0.0.0.0/0.0.0.0:36349 ] with min threads 1 and max threads of 1
      [21/Jun/2013:02:20:11 UTC] [B1228]: Cluster ping interval is 60 seconds
      [21/Jun/2013:02:20:11 UTC] [B1039]: Broker "depotclusterdepotserver1@10.240.100.117:27676" ready.
      [21/Jun/2013:02:20:11 UTC] [B1179]: Activated broker
      Address = mq://10.240.101.226:27676/?instName=depotclusterdepotserver2&brokerSessionUID=1923951166812821760
      StartTime = 1371781141732
      ProtocolVersion = 410
      [21/Jun/2013:02:20:11 UTC] [B1071]: Established cluster connection to broker mq://10.240.101.226:27676/?instName=depotclusterdepotserver2&brokerSessionUID=1923951166812821760[appdepotb/10.240.101.226:43120]
      [21/Jun/2013:02:20:11 UTC] [B1132]: Auto-creating destination creditCardRefund [Queue]
      [21/Jun/2013:02:20:11 UTC] [B1065]: Accepting: admin@127.0.0.1:43811->admin:38970. Count: service=1 broker=1
      [21/Jun/2013:02:20:11 UTC] [B1065]: Accepting: guest@127.0.0.1:33312->jms:47205. Count: service=1 broker=2
      [21/Jun/2013:02:20:11 UTC] [B1066]: Closing: guest@127.0.0.1:33312->jms:47205 because "[B0059]: Client closed the connection". Count: service=0 broker=1
      [21/Jun/2013:02:20:11 UTC] [B1065]: Accepting: guest@127.0.0.1:33314->jms:47205. Count: service=1 broker=2
      [21/Jun/2013:02:20:22 UTC] [B1065]: Accepting: guest@127.0.0.1:33316->jms:47205. Count: service=2 broker=3
      [21/Jun/2013:02:20:44 UTC] [B1065]: Accepting: guest@127.0.0.1:33338->jms:47205. Count: service=3 broker=4
      [21/Jun/2013:02:20:44 UTC] [B1132]: Auto-creating destination mobileDeviceNotificationDestination [Queue]
      [21/Jun/2013:02:20:44 UTC] [B1065]: Accepting: guest@127.0.0.1:33340->jms:47205. Count: service=4 broker=5
      [21/Jun/2013:02:21:02 UTC] [B1065]: Accepting: guest@127.0.0.1:33358->jms:47205. Count: service=5 broker=6
      [21/Jun/2013:02:21:24 UTC] [B1065]: Accepting: guest@127.0.0.1:33376->jms:47205. Count: service=6 broker=7
      [21/Jun/2013:02:21:42 UTC] [B1065]: Accepting: guest@127.0.0.1:33396->jms:47205. Count: service=7 broker=8
      [21/Jun/2013:02:21:42 UTC] [B1132]: Auto-creating destination reportRunner [Queue]
      [21/Jun/2013:02:21:45 UTC] [B1065]: Accepting: guest@127.0.0.1:33398->jms:47205. Count: service=8 broker=9
      [21/Jun/2013:02:21:45 UTC] ERROR [B3007]: Message 8200-10.240.100.117(ea:c8:7b:72:ca:c5)-44139-1371709438648 on destination Q:centralEventReceiver could not be found in the store
      [21/Jun/2013:02:21:45 UTC] WARNING Message 8200-10.240.100.117(ea:c8:7b:72:ca:c5)-44139-1371709438648 on destination Q:centralEventReceiver could not be found in the store:
      com.sun.messaging.jmq.jmsserver.util.BrokerException: Message 8200-10.240.100.117(ea:c8:7b:72:ca:c5)-44139-1371709438648 on destination Q:centralEventReceiver could not be found in the store
      at com.sun.messaging.jmq.jmsserver.persist.file.DstMsgStore.getMessageInfo(DstMsgStore.java:481)
      at com.sun.messaging.jmq.jmsserver.persist.file.DstMsgStore.updateInterestState(DstMsgStore.java:747)
      at com.sun.messaging.jmq.jmsserver.persist.file.MsgStore.updateInterestState(MsgStore.java:705)
      at com.sun.messaging.jmq.jmsserver.persist.file.FileStore.updateInterestState(FileStore.java:1169)
      at com.sun.messaging.jmq.jmsserver.core.PacketReference.delivered(PacketReference.java:1651)
      at com.sun.messaging.jmq.jmsserver.core.Session.fillNextPacket(Session.java:758)
      at com.sun.messaging.jmq.jmsserver.service.imq.IMQIPConnection.fillNextPacket(IMQIPConnection.java:1914)
      at com.sun.messaging.jmq.jmsserver.service.imq.IMQIPConnection.writeData(IMQIPConnection.java:1559)
      at com.sun.messaging.jmq.jmsserver.service.imq.IMQIPConnection.process(IMQIPConnection.java:535)
      at com.sun.messaging.jmq.jmsserver.service.imq.OperationRunnable.process(OperationRunnable.java:176)
      at com.sun.messaging.jmq.jmsserver.util.pool.BasicRunnable.run(BasicRunnable.java:499)
      at java.lang.Thread.run(Thread.java:722)
      [21/Jun/2013:02:21:45 UTC] [B1065]: Accepting: guest@127.0.0.1:33400->jms:47205. Count: service=9 broker=10
      [21/Jun/2013:02:21:54 UTC] [B1065]: Accepting: guest@10.240.100.117:48451->jms:47205. Count: service=10 broker=11
      [21/Jun/2013:02:21:54 UTC] [B1065]: Accepting: guest@10.240.100.117:48455->jms:47205. Count: service=11 broker=12
      [21/Jun/2013:02:21:54 UTC] [B1065]: Accepting: guest@10.240.100.117:48459->jms:47205. Count: service=12 broker=13
      [21/Jun/2013:02:21:54 UTC] [B1065]: Accepting: guest@10.240.100.117:48463->jms:47205. Count: service=13 broker=14
      [21/Jun/2013:02:21:54 UTC] [B1065]: Accepting: guest@10.240.100.117:48467->jms:47205. Count: service=14 broker=15
      [21/Jun/2013:02:21:54 UTC] [B1065]: Accepting: guest@10.240.100.117:48475->jms:47205. Count: service=15 broker=16
      [21/Jun/2013:02:21:55 UTC] [B1065]: Accepting: guest@10.240.100.117:48483->jms:47205. Count: service=16 broker=17
      [21/Jun/2013:02:21:55 UTC] [B1065]: Accepting: guest@10.240.100.117:48487->jms:47205. Count: service=17 broker=18
      [21/Jun/2013:02:21:55 UTC] [B1065]: Accepting: guest@10.240.100.117:48491->jms:47205. Count: service=18 broker=19
      [21/Jun/2013:02:21:55 UTC] [B1065]: Accepting: guest@10.240.100.117:48495->jms:47205. Count: service=19 broker=20
      :
      :
      [21/Jun/2013:02:21:58 UTC] ERROR [B3100]: Unexpected Broker Internal Error : [Uncaught Exception]:
      com.sun.messaging.jmq.jmsserver.util.BrokerException: [B4117]: Unexpected Broker Exception: [Unexpected Error processing message]
      at com.sun.messaging.jmq.jmsserver.data.PacketRouter.handleMessage(PacketRouter.java:216)
      at com.sun.messaging.jmq.jmsserver.service.imq.IMQIPConnection.readData(IMQIPConnection.java:1362)
      at com.sun.messaging.jmq.jmsserver.service.imq.IMQIPConnection.process(IMQIPConnection.java:548)
      at com.sun.messaging.jmq.jmsserver.service.imq.OperationRunnable.process(OperationRunnable.java:176)
      at com.sun.messaging.jmq.jmsserver.util.pool.BasicRunnable.run(BasicRunnable.java:499)
      at java.lang.Thread.run(Thread.java:722)
      Caused by: java.lang.IllegalStateException: File not synced. You must call Iterator to play back log file.
      at com.sun.messaging.jmq.util.txnlog.file.FileTransactionLogWriter.writeRecord(FileTransactionLogWriter.java:841)
      at com.sun.messaging.jmq.util.txnlog.file.FileTransactionLogWriter.write(FileTransactionLogWriter.java:824)
      at com.sun.messaging.jmq.jmsserver.persist.file.TransactionLogManager.writeTransactionEvent(TransactionLogManager.java:794)
      at com.sun.messaging.jmq.jmsserver.persist.file.TransactionLogManager.logTxn(TransactionLogManager.java:754)
      at com.sun.messaging.jmq.jmsserver.persist.file.FileStore.logTxn(FileStore.java:2480)
      at com.sun.messaging.jmq.jmsserver.data.TransactionList.logClusterTransaction(TransactionList.java:1989)
      at com.sun.messaging.jmq.jmsserver.data.handlers.TransactionHandler.doRemotePrepare(TransactionHandler.java:2284)
      at com.sun.messaging.jmq.jmsserver.data.handlers.TransactionHandler.doPrepare(TransactionHandler.java:2207)
      at com.sun.messaging.jmq.jmsserver.data.handlers.TransactionHandler.doRemoteCommit(TransactionHandler.java:2490)
      at com.sun.messaging.jmq.jmsserver.data.handlers.TransactionHandler.doCommit(TransactionHandler.java:1061)
      at com.sun.messaging.jmq.jmsserver.data.handlers.TransactionHandler.handle(TransactionHandler.java:730)
      at com.sun.messaging.jmq.jmsserver.data.PacketRouter.handleMessage(PacketRouter.java:188)
      ... 5 more
      [21/Jun/2013:02:21:58 UTC] [B1065]: Accepting: guest@127.0.0.1:33473->jms:47205. Count: service=12 broker=13
      [21/Jun/2013:02:21:58 UTC] ERROR [B3100]: Unexpected Broker Internal Error : [Uncaught Exception]:
      com.sun.messaging.jmq.jmsserver.util.BrokerException: [B4117]: Unexpected Broker Exception: [Unexpected Error processing message]
      at com.sun.messaging.jmq.jmsserver.data.PacketRouter.handleMessage(PacketRouter.java:216)
      at com.sun.messaging.jmq.jmsserver.service.imq.IMQIPConnection.readData(IMQIPConnection.java:1362)
      at com.sun.messaging.jmq.jmsserver.service.imq.IMQIPConnection.process(IMQIPConnection.java:548)
      at com.sun.messaging.jmq.jmsserver.service.imq.OperationRunnable.process(OperationRunnable.java:176)
      at com.sun.messaging.jmq.jmsserver.util.pool.BasicRunnable.run(BasicRunnable.java:499)
      at java.lang.Thread.run(Thread.java:722)
      Caused by: java.lang.IllegalStateException: File not synced. You must call Iterator to play back log file.
      at com.sun.messaging.jmq.util.txnlog.file.FileTransactionLogWriter.writeRecord(FileTransactionLogWriter.java:841)
      at com.sun.messaging.jmq.util.txnlog.file.FileTransactionLogWriter.write(FileTransactionLogWriter.java:824)
      at com.sun.messaging.jmq.jmsserver.persist.file.TransactionLogManager.writeTransactionEvent(TransactionLogManager.java:794)
      at com.sun.messaging.jmq.jmsserver.persist.file.TransactionLogManager.logTxn(TransactionLogManager.java:754)
      at com.sun.messaging.jmq.jmsserver.persist.file.FileStore.logTxn(FileStore.java:2480)
      at com.sun.messaging.jmq.jmsserver.data.TransactionList.logClusterTransaction(TransactionList.java:1989)
      at com.sun.messaging.jmq.jmsserver.data.handlers.TransactionHandler.doRemotePrepare(TransactionHandler.java:2284)
      at com.sun.messaging.jmq.jmsserver.data.handlers.TransactionHandler.doPrepare(TransactionHandler.java:2207)
      at com.sun.messaging.jmq.jmsserver.data.handlers.TransactionHandler.doRemoteCommit(TransactionHandler.java:2490)
      at com.sun.messaging.jmq.jmsserver.data.handlers.TransactionHandler.doCommit(TransactionHandler.java:1061)
      at com.sun.messaging.jmq.jmsserver.data.handlers.TransactionHandler.handle(TransactionHandler.java:730)
      at com.sun.messaging.jmq.jmsserver.data.PacketRouter.handleMessage(PacketRouter.java:188)
      ... 5 more
      [21/Jun/2013:02:22:02 UTC] [B1065]: Accepting: guest@127.0.0.1:33492->jms:47205. Count: service=13 broker=14
      [21/Jun/2013:02:22:02 UTC] ERROR java.lang.IllegalStateException: File not synced. You must call Iterator to play back log file.: TUID=7453245609361745154 Xid=6170706465706F74612E6465706F742E746573742E636F72702E6572646D672E636F6D2C6465706F742D736572766572312C5032333730302C000A000000EFBE89646170706465706F74612E6465706F742E746573742E636F72702E6572646D672E636F6D2C6465706F742D736572766572312C503233373030:
      java.lang.IllegalStateException: File not synced. You must call Iterator to play back log file.
      at com.sun.messaging.jmq.util.txnlog.file.FileTransactionLogWriter.writeRecord(FileTransactionLogWriter.java:841)
      at com.sun.messaging.jmq.util.txnlog.file.FileTransactionLogWriter.write(FileTransactionLogWriter.java:824)
      at com.sun.messaging.jmq.jmsserver.persist.file.TransactionLogManager.writeTransactionEvent(TransactionLogManager.java:794)
      at com.sun.messaging.jmq.jmsserver.persist.file.TransactionLogManager.logTxn(TransactionLogManager.java:754)
      at com.sun.messaging.jmq.jmsserver.persist.file.FileStore.logTxn(FileStore.java:2480)
      at com.sun.messaging.jmq.jmsserver.data.handlers.TransactionHandler.logTxn(TransactionHandler.java:1512)
      at com.sun.messaging.jmq.jmsserver.data.handlers.TransactionHandler.doPrepare(TransactionHandler.java:2222)
      at com.sun.messaging.jmq.jmsserver.data.handlers.TransactionHandler.handle(TransactionHandler.java:621)
      at com.sun.messaging.jmq.jmsserver.data.PacketRouter.handleMessage(PacketRouter.java:188)
      at com.sun.messaging.jmq.jmsserver.service.imq.IMQIPConnection.readData(IMQIPConnection.java:1362)
      at com.sun.messaging.jmq.jmsserver.service.imq.IMQIPConnection.process(IMQIPConnection.java:548)
      at com.sun.messaging.jmq.jmsserver.service.imq.OperationRunnable.process(OperationRunnable.java:176)
      at com.sun.messaging.jmq.jmsserver.util.pool.BasicRunnable.run(BasicRunnable.java:499)
      at java.lang.Thread.run(Thread.java:722)
      [21/Jun/2013:02:22:02 UTC] [B1065]: Accepting: guest@127.0.0.1:33494->jms:47205. Count: service=14 broker=15
      [21/Jun/2013:02:22:02 UTC] [B1065]: Accepting: guest@127.0.0.1:33496->jms:47205. Count: service=15 broker=16
      [21/Jun/2013:02:22:02 UTC] ERROR java.lang.IllegalStateException: File not synced. You must call Iterator to play back log file.: TUID=7453245609361744896 Xid=6170706465706F74612E6465706F742E746573742E636F72702E6572646D672E636F6D2C6465706F742D736572766572312C5032333730302C0006000000EFBE89646170706465706F74612E6465706F742E746573742E636F72702E6572646D672E636F6D2C6465706F742D736572766572312C503233373030:
      java.lang.IllegalStateException: File not synced. You must call Iterator to play back log file.
      at com.sun.messaging.jmq.util.txnlog.file.FileTransactionLogWriter.writeRecord(FileTransactionLogWriter.java:841)
      at com.sun.messaging.jmq.util.txnlog.file.FileTransactionLogWriter.write(FileTransactionLogWriter.java:824)
      at com.sun.messaging.jmq.jmsserver.persist.file.TransactionLogManager.writeTransactionEvent(TransactionLogManager.java:794)
      at com.sun.messaging.jmq.jmsserver.persist.file.TransactionLogManager.logTxn(TransactionLogManager.java:754)
      at com.sun.messaging.jmq.jmsserver.persist.file.FileStore.logTxn(FileStore.java:2480)
      at com.sun.messaging.jmq.jmsserver.data.handlers.TransactionHandler.logTxn(TransactionHandler.java:1512)
      at com.sun.messaging.jmq.jmsserver.data.handlers.TransactionHandler.doPrepare(TransactionHandler.java:2222)
      at com.sun.messaging.jmq.jmsserver.data.handlers.TransactionHandler.handle(TransactionHandler.java:621)
      at com.sun.messaging.jmq.jmsserver.data.PacketRouter.handleMessage(PacketRouter.java:188)
      at com.sun.messaging.jmq.jmsserver.service.imq.IMQIPConnection.readData(IMQIPConnection.java:1362)
      at com.sun.messaging.jmq.jmsserver.service.imq.IMQIPConnection.process(IMQIPConnection.java:548)
      at com.sun.messaging.jmq.jmsserver.service.imq.OperationRunnable.process(OperationRunnable.java:176)
      at com.sun.messaging.jmq.jmsserver.util.pool.BasicRunnable.run(BasicRunnable.java:499)
      at java.lang.Thread.run(Thread.java:722)
      }}

      This was followed by screeds of similar java.lang.IllegalStateException messages similar to the one shown above.

      In the end I had had to remove the fs370 directory and have the broker reinitialise the storage to get the application up and
      running, and in so doing lost all messages that were in flight at the time the problem arose. (As the broker was running in a
      cluster I had to repeat the same process on the other node).

      The only other issue that I could find that relates to this kind of error message was MQ-199 but that is addressed in MQ 5.0. I have
      commented on that issue to ask if the fix will be backported to 4.5.2.

        Activity

        Hide
        amyk added a comment -

        5.0, not 4.5.2x - typically a patch release is not open source

        Show
        amyk added a comment - 5.0, not 4.5.2x - typically a patch release is not open source
        Hide
        hopesea added a comment -

        Is 5.0 production ready? It is described as a developer release on the site. Does that mean those using 4.5.2 open source in production have to live with broker storage corruptions until 5.0 is reaches release?

        Show
        hopesea added a comment - Is 5.0 production ready? It is described as a developer release on the site. Does that mean those using 4.5.2 open source in production have to live with broker storage corruptions until 5.0 is reaches release?
        Hide
        amyk added a comment - - edited

        Firstly, the problem was not a broker storage corruption. A bug (14497787) in the store prevented the txnlog recovery to complete and left the store in a runtime state unable to accept new operations.

        5.0 passed all nightly/milestone regression tests as well as QA tests for all supported new features, and it has been released. For official word, please refer to official announcement/statement. 5.0 is also included with GlassFish 4.0 release
        https://glassfish.java.net/download.html

        Show
        amyk added a comment - - edited Firstly, the problem was not a broker storage corruption. A bug (14497787) in the store prevented the txnlog recovery to complete and left the store in a runtime state unable to accept new operations. 5.0 passed all nightly/milestone regression tests as well as QA tests for all supported new features, and it has been released. For official word, please refer to official announcement/statement. 5.0 is also included with GlassFish 4.0 release https://glassfish.java.net/download.html
        Hide
        hopesea added a comment -

        Thanks for clarifying. If the storage is not corrupted, what are the recovery steps for 4.5.2 to allow the broker to resume normal operations and process messages that have backed up in the queues? We have hundreds of messages passing through queues per second, so can't afford to lose operation for any period of time.

        Show
        hopesea added a comment - Thanks for clarifying. If the storage is not corrupted, what are the recovery steps for 4.5.2 to allow the broker to resume normal operations and process messages that have backed up in the queues? We have hundreds of messages passing through queues per second, so can't afford to lose operation for any period of time.
        Hide
        amyk added a comment -

        The recovery code in the store needs 14497787 fix, which is in 5.0 and 4.5.2p2, to complete the recovery.

        Show
        amyk added a comment - The recovery code in the store needs 14497787 fix, which is in 5.0 and 4.5.2p2, to complete the recovery.

          People

          • Assignee:
            amyk
            Reporter:
            hopesea
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: