This website summarizes a couple of more ways for changing the transaction timeout of jboss. For example, you can specify it in your standalone.xml configuration file as a subsystem like this:
<subsystem xmlns="urn:jboss:domain:transactions:3.0"><core-environment><process-id><uuid/></process-id></core-environment><recovery-environment socket-binding="txn-recovery-environment" status-socket-binding="txn-status-manager"/><coordinator-environment default-timeout="<your timeout value here>"/></subsystem>Alternative adjust the transaction timeout prameter via the jboss admin platform under subsystems->transaction->Time Out
Maybe that will help.