#1
  1. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jan 2012
    Posts
    5
    Rep Power
    0

    My server died with very strange error in syslog


    Hi,

    Today my server died with very strange error in syslog, can you somebody tell me what's happening and whether that's is a attack or something like this?


    601694.166251] INFO: task sshd:20442 blocked for more than 120 seconds.
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601694.166277] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601694.166313] sshd D 0000000000000002 0 20442 29235
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601694.166339] ffff810056209a08 0000000000000086 00000000562099c8 0000000000000000
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601694.166381] ffff8105e7c3c080 ffff810250b00080 ffff8105e7c3c308 0000000300000000
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601694.166422] 0000000000000000 0000000100000000 0000000000000001 ffffffff8024ac66
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601694.166450] Call Trace:
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601694.166492] [<ffffffff8024ac66>] getnstimeofday+0x39/0x98
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601694.166517] [<ffffffff80271122>] sync_page+0x0/0x41
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601694.166539] [<ffffffff80429667>] io_schedule+0x5c/0x9e
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601694.166562] [<ffffffff8027115e>] sync_page+0x3c/0x41
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601694.166584] [<ffffffff804297da>] __wait_on_bit_lock+0x36/0x66
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601694.166608] [<ffffffff8027110f>] __lock_page+0x5e/0x64
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601694.166631] [<ffffffff802461ff>] wake_bit_function+0x0/0x23
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601694.166654] [<ffffffff802712d5>] find_lock_page+0x5e/0x8a
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601694.166677] [<ffffffff80273032>] filemap_fault+0x72/0x33c
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601694.166702] [<ffffffff8027e69c>] __do_fault+0x50/0x3e8
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601694.166728] [<ffffffff80281a5f>] handle_mm_fault+0x452/0x8de
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601694.166752] [<ffffffff8027c33f>] vma_prio_tree_insert+0x20/0x39
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601694.166776] [<ffffffff80284c82>] vma_link+0xd6/0xfb
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601694.166801] [<ffffffff80221fbc>] do_page_fault+0x5d8/0x9c8
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601694.166826] [<ffffffff8042ab79>] error_exit+0x0/0x60
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601694.168834] [<ffffffff80320b99>] __clear_user+0x16/0x34
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601694.168858] [<ffffffff802cac55>] padzero+0x1b/0x2b
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601694.168880] [<ffffffff802cb8b8>] load_elf_binary+0x876/0x16a9
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601694.168907] [<ffffffff8029f037>] copy_strings+0x1bc/0x1cd
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601694.168931] [<ffffffff8029e7b5>] search_binary_handler+0xb0/0x1e5
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601694.168955] [<ffffffff8029fef8>] do_execve+0x168/0x215
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601694.168979] [<ffffffff8020a439>] sys_execve+0x35/0x4c
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601694.169002] [<ffffffff8020c31a>] stub_execve+0x6a/0xc0
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601694.169027]
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601696.313653] INFO: task pdflush:20427 blocked for more than 120 seconds.
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601696.313679] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601696.313715] pdflush D 0000000000000002 0 20427 2
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601696.313741] ffff810583e1dbc0 0000000000000046 ffff81063b0d1210 ffff81014163f658
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601696.314312] ffff8103b60f5830 ffff810101066140 ffff8103b60f5ab8 00000007a0030c0d
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601696.314353] 0000000000000086 ffff8106394417d8 ffff810583e1dc30 ffffffff8024ac66
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601696.314382] Call Trace:
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601696.314424] [<ffffffff8024ac66>] getnstimeofday+0x39/0x98
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601696.314448] [<ffffffff80271122>] sync_page+0x0/0x41
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601696.314471] [<ffffffff80429667>] io_schedule+0x5c/0x9e
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601696.314494] [<ffffffff8027115e>] sync_page+0x3c/0x41
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601696.314516] [<ffffffff804297da>] __wait_on_bit_lock+0x36/0x66
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601696.317026] [<ffffffff8027110f>] __lock_page+0x5e/0x64
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601696.317049] [<ffffffff802461ff>] wake_bit_function+0x0/0x23
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601696.317073] [<ffffffff80278e76>] pagevec_lookup_tag+0x1a/0x21
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601696.317096] [<ffffffff8027760a>] write_cache_pages+0x10c/0x2b1
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601696.317120] [<ffffffff802771b1>] __writepage+0x0/0x23
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601696.317144] [<ffffffff8022bdc8>] task_rq_lock+0x4d/0x7f
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601696.317167] [<ffffffff80428cf6>] wait_for_common+0xda/0x13a
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601696.317190] [<ffffffff8022c168>] default_wake_function+0x0/0xe
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601696.317215] [<ffffffff802777f2>] do_writepages+0x27/0x2d
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601696.317239] [<ffffffff802b69f4>] __writeback_single_inode+0x144/0x29d
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601696.317263] [<ffffffff80248440>] enqueue_hrtimer+0xd7/0xe4
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601696.317287] [<ffffffff80248bb6>] hrtimer_start+0x112/0x134
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601696.317311] [<ffffffff802b6eb8>] sync_sb_inodes+0x1b1/0x293
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601696.317335] [<ffffffff802b72e6>] writeback_inodes+0x62/0xb3
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601696.317359] [<ffffffff80278044>] background_writeout+0x87/0xbb
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601696.317384] [<ffffffff80278512>] pdflush+0x0/0x211
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601696.317405] [<ffffffff80278676>] pdflush+0x164/0x211
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601696.317427] [<ffffffff80277fbd>] background_writeout+0x0/0xbb
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601696.317451] [<ffffffff802460ab>] kthread+0x47/0x74
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601696.317472] [<ffffffff802300ed>] schedule_tail+0x27/0x5c
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601696.317496] [<ffffffff8020cf38>] child_rip+0xa/0x12
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601696.317520] [<ffffffff80246064>] kthread+0x0/0x74
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601696.317541] [<ffffffff8020cf2e>] child_rip+0x0/0x12
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601696.317562]
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.527373] INFO: task kjournald:1875 blocked for more than 120 seconds.
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.527399] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.527436] kjournald D ffff8102b994e780 0 1875 2
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.527462] ffff8106381abe10 0000000000000046 ffff810356f7e6e0 0000000000000002
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.527503] ffff810639897060 ffff81046c371770 ffff8106398972e8 0000000500000001
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.527544] 0000000000000282 0000000000000003 ffff8106381abe10 ffffffff8022ad2f
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.527776] Call Trace:
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.527818] [<ffffffff8022ad2f>] __wake_up+0x38/0x4f
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.527845] [<ffffffffa0193ad4>] :jbd:journal_commit_transaction+0xda/0xbf2
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.527882] [<ffffffff802461d1>] autoremove_wake_function+0x0/0x2e
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.527907] [<ffffffff8023cfb7>] try_to_del_timer_sync+0x51/0x5a
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.527935] [<ffffffffa0196d2c>] :jbd:kjournald+0xc1/0x1fb
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.527957] [<ffffffff802461d1>] autoremove_wake_function+0x0/0x2e
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.527983] [<ffffffffa0196c6b>] :jbd:kjournald+0x0/0x1fb
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.528007] [<ffffffff802460ab>] kthread+0x47/0x74
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.528029] [<ffffffff802300ed>] schedule_tail+0x27/0x5c
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.528052] [<ffffffff8020cf38>] child_rip+0xa/0x12
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.528076] [<ffffffff8021a866>] lapic_next_event+0xf/0x13
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.528562] [<ffffffff80246064>] kthread+0x0/0x74
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.528584] [<ffffffff8020cf2e>] child_rip+0x0/0x12
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.528605]
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.528637] INFO: task apache2:20061 blocked for more than 120 seconds.
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.528660] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.528697] apache2 D ffff81063901a000 0 20061 380
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.528722] ffff810140f2b8f8 0000000000000086 00000000ffffffff ffffffff80248440
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.528763] ffff8106391355b0 ffff81063cfcf1a0 ffff810639135838 0000000340f2bc88
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.528804] 0000000000000000 0000000000000000 ffff810588de34a0 ffff810100000002
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.528833] Call Trace:
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.528864] [<ffffffff80248440>] enqueue_hrtimer+0xd7/0xe4
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.528892] [<ffffffffa0192b9b>] :jbd:start_this_handle+0x289/0x304
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.528916] [<ffffffff802461d1>] autoremove_wake_function+0x0/0x2e
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.528943] [<ffffffffa0192cb1>] :jbd:journal_start+0x9b/0xd2
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.528972] [<ffffffffa01a60d1>] :ext3:ext3_ordered_writepage+0x4e/0x16f
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.528999] [<ffffffff8027ac3a>] shrink_page_list+0x32d/0x559
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.529023] [<ffffffff802875ae>] page_check_address+0x127/0x181
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.529049] [<ffffffff80223fb5>] ptep_clear_flush_young+0xf/0x27
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.529073] [<ffffffff802876a2>] page_referenced_one+0x76/0xbd
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.529099] [<ffffffff8027af94>] shrink_inactive_list+0x12e/0x39f
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.529129] [<ffffffff8027b2e8>] shrink_zone+0xe3/0x106
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.529153] [<ffffffff8027b681>] try_to_free_pages+0x21d/0x361
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.529178] [<ffffffff8027a6ff>] isolate_pages_global+0x0/0x2f
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.529203] [<ffffffff80276b05>] __alloc_pages_internal+0x26a/0x3bf
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.529230] [<ffffffff8028a5ad>] read_swap_cache_async+0x48/0xc2
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.529256] [<ffffffff8028a67e>] swapin_readahead+0x57/0x98
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.529280] [<ffffffff80281b17>] handle_mm_fault+0x50a/0x8de
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.529304] [<ffffffff802a1cc1>] generic_permission+0x5a/0xd8
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.529329] [<ffffffff8023b609>] cap_set_effective+0x1d/0x51
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.529353] [<ffffffff80221fbc>] do_page_fault+0x5d8/0x9c8
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.529377] [<ffffffff8042947f>] thread_return+0x6b/0xac
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.529401] [<ffffffff8042ab79>] error_exit+0x0/0x60
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.529426]
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.529461] INFO: task apache2:20354 blocked for more than 120 seconds.
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.529484] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.529520] apache2 D ffff81063901a000 0 20354 380
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.529546] ffff8104157ffbd8 0000000000000082 ffff8104157ffb98 ffffffff80229020
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.529587] ffff8104d4bcf6f0 ffff8101cb084810 ffff8104d4bcf978 000000000103b880
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.529628] 004eeb33e1daa3ba 00000000ffffffff 0000000000000003 ffffffff8022efec
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.529656] Call Trace:
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.529687] [<ffffffff80229020>] hrtick_start_fair+0xfb/0x144
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.529712] [<ffffffff8022efec>] hrtick_set+0x9e/0xf7
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.529737] [<ffffffffa0192b9b>] :jbd:start_this_handle+0x289/0x304
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.529761] [<ffffffff802461d1>] autoremove_wake_function+0x0/0x2e
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.529789] [<ffffffffa0192cb1>] :jbd:journal_start+0x9b/0xd2
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.529816] [<ffffffffa01a6870>] :ext3:ext3_dirty_inode+0x28/0x7b
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.529841] [<ffffffff802b7134>] __mark_inode_dirty+0x29/0x179
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.529866] [<ffffffff802adee3>] touch_atime+0x109/0x112
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.532516] [<ffffffff80272df3>] generic_file_aio_read+0x446/0x4ae
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.532546] [<ffffffff8029b103>] do_sync_read+0xc9/0x10c
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.532571] [<ffffffff802461d1>] autoremove_wake_function+0x0/0x2e
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.532598] [<ffffffff8029b8f4>] vfs_read+0xaa/0x152
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.532621] [<ffffffff8029bcd5>] sys_read+0x45/0x6e
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.532644] [<ffffffff8020beda>] system_call_after_swapgs+0x8a/0x8f
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601728.532669]
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601739.263345] INFO: task apache2:20240 blocked for more than 120 seconds.
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601739.263370] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601739.263407] apache2 D ffff81063901a000 0 20240 380
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601739.263433] ffff81010108dbd8 0000000000000086 ffff81010108db98 ffffffff80229020
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601739.263474] ffff810101066140 ffff81003ee4e300 ffff8101010663c8 0000000101045880
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601739.263515] 004eeb364c4dfc79 00000000ffffffff 0000000000000004 ffffffff8022efec
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601739.263544] Call Trace:
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601739.263582] [<ffffffff80229020>] hrtick_start_fair+0xfb/0x144
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601739.263607] [<ffffffff8022efec>] hrtick_set+0x9e/0xf7
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601739.263634] [<ffffffffa0192b9b>] :jbd:start_this_handle+0x289/0x304
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601739.263659] [<ffffffff802461d1>] autoremove_wake_function+0x0/0x2e
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601739.263687] [<ffffffffa0192cb1>] :jbd:journal_start+0x9b/0xd2
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601739.263715] [<ffffffffa01a6870>] :ext3:ext3_dirty_inode+0x28/0x7b
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601739.263740] [<ffffffff802b7134>] __mark_inode_dirty+0x29/0x179
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601739.263764] [<ffffffff802adee3>] touch_atime+0x109/0x112
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601739.263788] [<ffffffff80272df3>] generic_file_aio_read+0x446/0x4ae
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601739.263817] [<ffffffff8029b103>] do_sync_read+0xc9/0x10c
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601739.263842] [<ffffffff802461d1>] autoremove_wake_function+0x0/0x2e
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601739.263869] [<ffffffff80221fbc>] do_page_fault+0x5d8/0x9c8
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601739.263893] [<ffffffff8029b8f4>] vfs_read+0xaa/0x152
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601739.263915] [<ffffffff8029bcd5>] sys_read+0x45/0x6e
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601739.263938] [<ffffffff8020beda>] system_call_after_swapgs+0x8a/0x8f
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601739.263965]
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601743.560439] INFO: task apache2:20244 blocked for more than 120 seconds.
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601743.560465] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601743.560501] apache2 D ffff81063901a000 0 20244 380
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601743.560527] ffff810101099bd8 0000000000000082 ffff810101099b98 ffffffff80229020
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601743.560568] ffff81010108e180 ffff8102c2d9e0c0 ffff81010108e408 000000080108b880
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601743.560609] 004eeb36ff634968 00000000ffffffff 0000000000000000 ffffffff8022efec
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601743.560638] Call Trace:
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601743.560676] [<ffffffff80229020>] hrtick_start_fair+0xfb/0x144
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601743.560701] [<ffffffff8022efec>] hrtick_set+0x9e/0xf7
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601743.560728] [<ffffffffa0192b9b>] :jbd:start_this_handle+0x289/0x304
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601743.560752] [<ffffffff802461d1>] autoremove_wake_function+0x0/0x2e
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601743.560780] [<ffffffffa0192cb1>] :jbd:journal_start+0x9b/0xd2
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601743.560809] [<ffffffffa01a6870>] :ext3:ext3_dirty_inode+0x28/0x7b
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601743.560834] [<ffffffff802b7134>] __mark_inode_dirty+0x29/0x179
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601743.560859] [<ffffffff802adee3>] touch_atime+0x109/0x112
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601743.560883] [<ffffffff80272df3>] generic_file_aio_read+0x446/0x4ae
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601743.560913] [<ffffffff8029b103>] do_sync_read+0xc9/0x10c
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601743.560937] [<ffffffff802461d1>] autoremove_wake_function+0x0/0x2e
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601743.560964] [<ffffffff80221fbc>] do_page_fault+0x5d8/0x9c8
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601743.560988] [<ffffffff8029b8f4>] vfs_read+0xaa/0x152
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601743.561010] [<ffffffff8029bcd5>] sys_read+0x45/0x6e
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601743.561033] [<ffffffff8020beda>] system_call_after_swapgs+0x8a/0x8f
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601743.561059]
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601745.706021] INFO: task mysqld:20411 blocked for more than 120 seconds.
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601745.706047] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601745.706084] mysqld D ffff81063901a000 0 20411 25232
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601745.706109] ffff81057bb51bd8 0000000000000086 ffff81057bb51b98 ffffffff80229020
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601745.706151] ffff810385148790 ffff8103b60f4480 ffff810385148a18 0000000a0109f880
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601745.706192] 004eeb379dfc7ad0 00000000ffffffff 0000000000000000 ffffffff8022efec
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601745.706220] Call Trace:
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601745.706258] [<ffffffff80229020>] hrtick_start_fair+0xfb/0x144
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601745.706283] [<ffffffff8022efec>] hrtick_set+0x9e/0xf7
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601745.706310] [<ffffffffa0192b9b>] :jbd:start_this_handle+0x289/0x304
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601745.706335] [<ffffffff802461d1>] autoremove_wake_function+0x0/0x2e
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601745.706362] [<ffffffffa0192cb1>] :jbd:journal_start+0x9b/0xd2
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601745.706391] [<ffffffffa01a6870>] :ext3:ext3_dirty_inode+0x28/0x7b
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601745.706416] [<ffffffff802b7134>] __mark_inode_dirty+0x29/0x179
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601745.706440] [<ffffffff802adee3>] touch_atime+0x109/0x112
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601745.706464] [<ffffffff80272df3>] generic_file_aio_read+0x446/0x4ae
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601745.706493] [<ffffffff8029b103>] do_sync_read+0xc9/0x10c
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601745.706518] [<ffffffff802461d1>] autoremove_wake_function+0x0/0x2e
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601745.706546] [<ffffffff8029b8f4>] vfs_read+0xaa/0x152
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601745.706568] [<ffffffff8029bcd5>] sys_read+0x45/0x6e
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601745.706591] [<ffffffff8020beda>] system_call_after_swapgs+0x8a/0x8f
    Feb 1 10:55:17 Debian-50-lenny-64-minimal kernel: [22601745.706617]

    RVICE ALERT: localhost;Total Processes;CRITICAL;SOFT;2;CRITICAL - Plugin timed out after 10 seconds
    Feb 1 11:05:27 Debian-50-lenny-64-minimal nagios3: SERVICE ALERT: localhost;Current Load;CRITICAL;SOFT;1;CRITICAL - load average: 104.30, 149.60, 138.59
    Feb 1 11:06:06 Debian-50-lenny-64-minimal nagios3: SERVICE ALERT: localhost;Total Processes;CRITICAL;SOFT;3;CRITICAL - Plugin timed out after 10 seconds
    Feb 1 11:06:28 Debian-50-lenny-64-minimal nagios3: SERVICE ALERT: localhost;Current Load;CRITICAL;SOFT;2;CRITICAL - load average: 88.56, 137.66, 135.14
    Feb 1 11:07:09 Debian-50-lenny-64-minimal nagios3: SERVICE ALERT: localhost;Total Processes;WARNING;HARD;4;PROCS WARNING: 340 processes
    Feb 1 11:07:09 Debian-50-lenny-64-minimal nagios3: SERVICE NOTIFICATION: root;localhost;Total Processes;WARNING;notify-service-by-email;PROCS WARNING: 340 processes
    Feb 1 11:07:12 Debian-50-lenny-64-minimal nagios3: Caught SIGTERM, shutting down...


    ...............................................................................
    ...............................................................................

    there are a lot of the same data related to apache2 mysql and proftpd

    ...............................................................................
    ...............................................................................

    RVICE ALERT: localhost;Total Processes;CRITICAL;SOFT;2;CRITICAL - Plugin timed out after 10 seconds
    Feb 1 11:05:27 Debian-50-lenny-64-minimal nagios3: SERVICE ALERT: localhost;Current Load;CRITICAL;SOFT;1;CRITICAL - load average: 104.30, 149.60, 138.59
    Feb 1 11:06:06 Debian-50-lenny-64-minimal nagios3: SERVICE ALERT: localhost;Total Processes;CRITICAL;SOFT;3;CRITICAL - Plugin timed out after 10 seconds
    Feb 1 11:06:28 Debian-50-lenny-64-minimal nagios3: SERVICE ALERT: localhost;Current Load;CRITICAL;SOFT;2;CRITICAL - load average: 88.56, 137.66, 135.14
    Feb 1 11:07:09 Debian-50-lenny-64-minimal nagios3: SERVICE ALERT: localhost;Total Processes;WARNING;HARD;4;PROCS WARNING: 340 processes
    Feb 1 11:07:09 Debian-50-lenny-64-minimal nagios3: SERVICE NOTIFICATION: root;localhost;Total Processes;WARNING;notify-service-by-email;PROCS WARNING: 340 processes
    Feb 1 11:07:12 Debian-50-lenny-64-minimal nagios3: Caught SIGTERM, shutting down...


    after reboot server go back and works good again, but i never before saw things like this.

    any ideas? thanks in advance
  2. #2
  3. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jan 2012
    Posts
    5
    Rep Power
    0
    I examined detailed output from syslogd and how did you see from above log file there are only three processes, but in whole content of log file I saw how when apache die, next process is mysql so also die, sshd, exim and so on to the whole server ...
  4. #3
  5. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Feb 2012
    Posts
    2
    Rep Power
    0
    Originally Posted by tripmod
    I examined detailed output from syslogd and how did you see from above log file there are only three processes, but in whole content of log file I saw how when apache die, next process is mysql so also die, sshd, exim and so on to the whole server ...
    Hi, any update in this matter?

    I am experiencing the exact same as you. I guess.. My server randomly crashes, but i am able to ping it but apache and sshd has died.

    I have been looking in my /var/log/message as well and i am having the same as you..

    Hmmm?
  6. #4
  7. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jan 2012
    Posts
    5
    Rep Power
    0
    hi, after last crash on my system, I have no detect the same situation, I really do not know what's happening in this moment, if you find a solution I will be happy if you shared with me, I also looking for a solution
  8. #5
  9. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Feb 2012
    Posts
    2
    Rep Power
    0
    Hmmf okay :/

    My server did the same thing this morning again.. After booting the server i found out that i was out of diskspace (my mysql couldnt start). For me it seems logical that service after service would crash do to no diskspace... but i deleted ~20gb apache hoping that it would solve the problem... but now, ~10 hours lator it happends again..

    And it's the same exact error in my /var/log/message. And its happening SO MANY times (every like 15 seconds):

    Feb 13 18:18:33 ditte kernel: [37443.560831] apache2 D 0000000000000013 0 27571 5457
    Feb 13 18:18:33 ditte kernel: [37443.560831] ffff81000856fde0 0000000000000082 ffff810098221000 ffff81013fbd8884
    Feb 13 18:18:33 ditte kernel: [37443.560831] ffff810062998f20 ffff810046420200 ffff8100629991a8 0000000200000000
    Feb 13 18:18:33 ditte kernel: [37443.560831] ffff81013fbd8880 ffff81000856fe68 00000000ffffffe9 ffff81013fbd8884
    Feb 13 18:18:33 ditte kernel: [37443.560831] Call Trace:
    Feb 13 18:18:33 ditte kernel: [37443.560831] [<ffffffff802a4b42>] do_path_lookup+0x158/0x1cf
    Feb 13 18:18:33 ditte kernel: [37443.560831] [<ffffffff8042996e>] __mutex_lock_slowpath+0x64/0x9b
    Feb 13 18:18:33 ditte kernel: [37443.560831] [<ffffffff804297d3>] mutex_lock+0xa/0xb
    Feb 13 18:18:33 ditte kernel: [37443.560831] [<ffffffff802a5839>] do_filp_open+0x11a/0x7c4
    Feb 13 18:18:33 ditte kernel: [37443.560831] [<ffffffff80246211>] autoremove_wake_function+0x0/0x2e
    Feb 13 18:18:33 ditte kernel: [37443.560831] [<ffffffff80299940>] get_unused_fd_flags+0x71/0x115
    Feb 13 18:18:33 ditte kernel: [37443.560831] [<ffffffff80299a2a>] do_sys_open+0x46/0xc3
    Feb 13 18:18:33 ditte kernel: [37443.560831] [<ffffffff8020beda>] system_call_after_swapgs+0x8a/0x8f
    Feb 13 18:18:33 ditte kernel: [37443.560831]
    Feb 13 18:18:44 ditte kernel: [37458.562303] apache2 D 0000000000000013 0 10904 5457



    PLEASE, anyone - how to fix this prob?

IMN logo majestic logo threadwatch logo seochat tools logo