Hi,
We've been running an Oracle Linux Server 6.4 virtual machine for over a month now and about a week ago we started getting issues accessing it.
Basically the machine hangs and we are not able to access it using SSH, only option is reboot, which takes about 15 minutes, and then it stays stable for about a day or two.
Today, is happened several times, we already rebooted twice but the errors keep poping up stating the there are processes hanging for more than 120 seconds (ex. "task oracle:1366 blocked for more than 120 seconds"), and the the filesystem is placed in read-only mode.
Below you can find some of the messages reported in /var/log/messages, I can upload the whole file if necessary.
Thanks in advance for any help.
Regards,
Ricardo
Jun 27 08:36:09 noronescvm1 kernel: INFO: task oracle:1366 blocked for more than 120 seconds.
Jun 27 08:36:09 noronescvm1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 27 08:36:09 noronescvm1 kernel: oracle D 0000000000000000 0 1366 1365 0x00000080
Jun 27 08:36:09 noronescvm1 kernel: ffff88014d309dc8 0000000000000086 ffff88014d309e38 ffff88014b399e80
Jun 27 08:36:09 noronescvm1 kernel: 0000000000000000 ffff88014aaf58d0 0000000000000000 0000000000000000
Jun 27 08:36:09 noronescvm1 kernel: ffff88014dfba638 ffff88014d309fd8 000000000000fb88 ffff88014dfba638
Jun 27 08:36:09 noronescvm1 kernel: Call Trace:
Jun 27 08:36:09 noronescvm1 kernel: [<ffffffff8109708e>] ? prepare_to_wait+0x4e/0x80
Jun 27 08:36:09 noronescvm1 kernel: [<ffffffffa0057ca5>] jbd2_log_wait_commit+0xc5/0x140 [jbd2]
Jun 27 08:36:09 noronescvm1 kernel: [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
Jun 27 08:36:09 noronescvm1 kernel: [<ffffffffa0058038>] jbd2_complete_transaction+0x68/0xb0 [jbd2]
Jun 27 08:36:09 noronescvm1 kernel: [<ffffffffa0073251>] ext4_sync_file+0x121/0x1d0 [ext4]
Jun 27 08:36:09 noronescvm1 kernel: [<ffffffff811b1d51>] vfs_fsync_range+0xa1/0xe0
Jun 27 08:36:09 noronescvm1 kernel: [<ffffffff811b1dfd>] vfs_fsync+0x1d/0x20
Jun 27 08:36:09 noronescvm1 kernel: [<ffffffff811b1e3e>] do_fsync+0x3e/0x60
Jun 27 08:36:09 noronescvm1 kernel: [<ffffffff811b1e90>] sys_fsync+0x10/0x20
Jun 27 08:36:09 noronescvm1 kernel: [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
Jun 27 08:38:14 noronescvm1 kernel: hv_storvsc vmbus_0_13: cmd 0x2a scsi status 0x2 srb status 0x4
Jun 27 08:38:14 noronescvm1 kernel: hv_storvsc vmbus_0_13: cmd 0x2a scsi status 0x2 srb status 0x4
Jun 27 08:38:18 noronescvm1 kernel: hv_storvsc vmbus_0_1: cmd 0x2a scsi status 0x2 srb status 0x4
Jun 27 08:39:48 noronescvm1 kernel: hv_storvsc vmbus_0_1: cmd 0x2a scsi status 0x2 srb status 0x4
Jun 27 08:40:29 noronescvm1 kernel: hv_storvsc vmbus_0_13: cmd 0x2a scsi status 0x2 srb status 0x4
Jun 27 08:40:29 noronescvm1 kernel: hv_storvsc vmbus_0_13: cmd 0x2a scsi status 0x2 srb status 0x4
Jun 27 08:42:04 noronescvm1 kernel: hv_storvsc vmbus_0_1: cmd 0x2a scsi status 0x2 srb status 0x4
Jun 27 08:42:04 noronescvm1 kernel: hv_storvsc vmbus_0_1: cmd 0x2a scsi status 0x2 srb status 0x4
Jun 27 08:42:45 noronescvm1 kernel: hv_storvsc vmbus_0_13: cmd 0x2a scsi status 0x2 srb status 0x4
Jun 27 08:42:45 noronescvm1 kernel: hv_storvsc vmbus_0_13: cmd 0x2a scsi status 0x2 srb status 0x4
Jun 27 08:44:21 noronescvm1 kernel: hv_storvsc vmbus_0_1: cmd 0x2a scsi status 0x2 srb status 0x4
Jun 27 08:44:21 noronescvm1 kernel: hv_storvsc vmbus_0_1: cmd 0x2a scsi status 0x2 srb status 0x4
Jun 27 08:45:01 noronescvm1 kernel: hv_storvsc vmbus_0_13: cmd 0x2a scsi status 0x2 srb status 0x4
Jun 27 08:45:01 noronescvm1 kernel: hv_storvsc vmbus_0_13: cmd 0x2a scsi status 0x2 srb status 0x4
Jun 27 08:45:12 noronescvm1 kernel: sd 5:0:0:0: [sdc] Unhandled error code
Jun 27 08:45:12 noronescvm1 kernel: sd 5:0:0:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
Jun 27 08:45:12 noronescvm1 kernel: sd 5:0:0:0: [sdc] CDB: Write(10): 2a 00 01 04 00 97 00 00 30 00
Jun 27 08:45:12 noronescvm1 kernel: sd 5:0:0:0: [sdc] Unhandled error code
Jun 27 08:45:12 noronescvm1 kernel: sd 5:0:0:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
Jun 27 08:45:12 noronescvm1 kernel: sd 5:0:0:0: [sdc] CDB: Write(10): 2a 00 01 04 00 8f 00 00 08 00
Jun 27 08:45:12 noronescvm1 kernel: Aborting journal on device sdc1-8.
Jun 27 08:45:12 noronescvm1 kernel: EXT4-fs (sdc1): delayed block allocation failed for inode 789493 at logical offset 1613 with max blocks 1 with error -30
Jun 27 08:45:12 noronescvm1 kernel:
Jun 27 08:45:12 noronescvm1 kernel: This should not happen!! Data will be lost
Jun 27 08:45:12 noronescvm1 kernel: EXT4-fs error (device sdc1) in ext4_da_writepages: Journal has aborted
Jun 27 08:46:37 noronescvm1 kernel: hv_storvsc vmbus_0_1: cmd 0x2a scsi status 0x2 srb status 0x4
Jun 27 08:46:37 noronescvm1 kernel: hv_storvsc vmbus_0_1: cmd 0x2a scsi status 0x2 srb status 0x4