Home Sign Up Sign In
kosenpai
V2EX  ›  问与答

Digital Ocean 新加坡最近总是 out of memory 崩溃

  •   
  •   kosenpai · May 17, 2016 · 3169 views
    This topic created in 3676 days ago, the information mentioned may be changed or developed.

    自用爬梯，一直稳定运行了一年多，最近 VPS 运行半天就完全死掉， ssh 也登不上去。看 syslog 发现崩溃之前都是 out of memory 。但是看内存记录发现崩溃前内存占用完全正常，不正常的部分是 CPU 占用在崩溃前异常高。下面是崩溃的一段，感觉就是这样被搞死的。请问大家这是怎么了？

    May 17 06:16:01 PIMSserver kernel: [18538.680826] sh invoked oom-killer: gfp_mask=0x84d0, order=0, oom_score_adj=0
May 17 06:16:01 PIMSserver kernel: [18538.680852] sh cpuset=/ mems_allowed=0
May 17 06:16:01 PIMSserver kernel: [18538.680875] CPU: 0 PID: 19134 Comm: sh Tainted: P OX 3.13.0-48-generic #80-Ubuntu
May 17 06:16:01 PIMSserver kernel: [18538.680878] Hardware name: DigitalOcean Droplet, BIOS 20160425 04/25/2016
May 17 06:16:01 PIMSserver kernel: [18538.680882] 0000000000000000 ffff880018913978 ffffffff81721506 ffff880019dc8000
May 17 06:16:01 PIMSserver kernel: [18538.680896] ffff880018913a00 ffffffff8171bdc1 ffffffff81067996 ffff8800189139d8
May 17 06:16:01 PIMSserver kernel: [18538.680900] ffffffff810c75dc 0000000000000001 ffff88001fffae28 0000000000000000
May 17 06:16:01 PIMSserver kernel: [18538.680904] Call Trace:
May 17 06:16:01 PIMSserver kernel: [18538.680979] [<ffffffff81721506>] dump_stack+0x45/0x56
May 17 06:16:01 PIMSserver kernel: [18538.680991] [<ffffffff8171bdc1>] dump_header+0x7f/0x1f1
May 17 06:16:01 PIMSserver kernel: [18538.681025] [<ffffffff81067996>] ? put_online_cpus+0x56/0x80
May 17 06:16:01 PIMSserver kernel: [18538.681045] [<ffffffff810c75dc>] ? rcu_oom_notify+0xcc/0xf0
May 17 06:16:01 PIMSserver kernel: [18538.681067] [<ffffffff811529be>] oom_kill_process+0x1ce/0x330
May 17 06:16:01 PIMSserver kernel: [18538.681097] [<ffffffff812d7225>] ? security_capable_noaudit+0x15/0x20
May 17 06:16:01 PIMSserver kernel: [18538.681102] [<ffffffff811530f4>] out_of_memory+0x414/0x450
May 17 06:16:01 PIMSserver kernel: [18538.681112] [<ffffffff81159460>] __alloc_pages_nodemask+0xa60/0xb80
May 17 06:16:01 PIMSserver kernel: [18538.681135] [<ffffffff81197ad3>] alloc_pages_current+0xa3/0x160
May 17 06:16:01 PIMSserver kernel: [18538.681146] [<ffffffff81179064>] __pmd_alloc+0x24/0x110
May 17 06:16:01 PIMSserver kernel: [18538.681152] [<ffffffff8117a825>] handle_mm_fault+0xd75/0xf10
May 17 06:16:01 PIMSserver kernel: [18538.681183] [<ffffffff811787f3>] ? follow_page_mask+0x73/0x5b0
May 17 06:16:01 PIMSserver kernel: [18538.681190] [<ffffffff811787be>] ? follow_page_mask+0x3e/0x5b0
May 17 06:16:01 PIMSserver kernel: [18538.681195] [<ffffffff8117ab26>] __get_user_pages+0x166/0x5e0
May 17 06:16:01 PIMSserver kernel: [18538.681199] [<ffffffff8117aff2>] get_user_pages+0x52/0x60
May 17 06:16:01 PIMSserver kernel: [18538.681214] [<ffffffff811c4246>] copy_strings.isra.17+0x256/0x2e0
May 17 06:16:01 PIMSserver kernel: [18538.681219] [<ffffffff811c4304>] copy_strings_kernel+0x34/0x40
May 17 06:16:01 PIMSserver kernel: [18538.681224] [<ffffffff811c586c>] do_execve_common.isra.22+0x4fc/0x7e0
May 17 06:16:01 PIMSserver kernel: [18538.681229] [<ffffffff811c5de6>] SyS_execve+0x36/0x50
May 17 06:16:01 PIMSserver kernel: [18538.681239] [<ffffffff81732569>] stub_execve+0x69/0xa0
May 17 06:16:01 PIMSserver kernel: [18538.681242] Mem-Info:
May 17 06:16:01 PIMSserver kernel: [18538.681250] Node 0 DMA per-cpu:
May 17 06:16:01 PIMSserver kernel: [18538.681257] CPU 0: hi: 0, btch: 1 usd: 0
May 17 06:16:01 PIMSserver kernel: [18538.681259] Node 0 DMA32 per-cpu:
May 17 06:16:01 PIMSserver kernel: [18538.681262] CPU 0: hi: 186, btch: 31 usd: 160
May 17 06:16:01 PIMSserver kernel: [18538.681268] active_anon:866 inactive_anon:890 isolated_anon:0
May 17 06:16:01 PIMSserver kernel: [18538.681268] active_file:37 inactive_file:46 isolated_file:0
May 17 06:16:01 PIMSserver kernel: [18538.681268] unevictable:0 dirty:1 writeback:0 unstable:0
May 17 06:16:01 PIMSserver kernel: [18538.681268] free:1471 slab_reclaimable:2333 slab_unreclaimable:4106
May 17 06:16:01 PIMSserver kernel: [18538.681268] mapped:24 shmem:5 pagetables:1581 bounce:0
May 17 06:16:01 PIMSserver kernel: [18538.681268] free_cma:0
May 17 06:16:01 PIMSserver kernel: [18538.681274] Node 0 DMA free:1928kB min:88kB low:108kB high:132kB active_anon:0kB inactive_anon:0kB active_file:16kB inactive_file:12kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:16kB shmem:0kB slab_reclaimable:96kB slab_unreclaimable:404kB kernel_stack:8kB pagetables:144kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:6801 all_unreclaimable? yes
May 17 06:16:01 PIMSserver kernel: [18538.681288] lowmem_reserve[]: 0 471 471 471
May 17 06:16:01 PIMSserver kernel: [18538.681296] Node 0 DMA32 free:3956kB min:2732kB low:3412kB high:4096kB active_anon:3464kB inactive_anon:3560kB active_file:132kB inactive_file:172kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:507896kB managed:485892kB mlocked:0kB dirty:4kB writeback:0kB mapped:80kB shmem:20kB slab_reclaimable:9236kB slab_unreclaimable:16020kB kernel_stack:992kB pagetables:6180kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:11036 all_unreclaimable? yes
May 17 06:16:01 PIMSserver kernel: [18538.681303] lowmem_reserve[]: 0 0 0 0
May 17 06:16:01 PIMSserver kernel: [18538.681307] Node 0 DMA: 20*4kB (U) 29*8kB (UM) 35*16kB (UM) 15*32kB (UMR) 5*64kB (UMR) 0*128kB 1*256kB (R) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1928kB
May 17 06:16:01 PIMSserver kernel: [18538.681334] Node 0 DMA32: 537*4kB (UEM) 208*8kB (UM) 1*16kB (M) 0*32kB 0*64kB 1*128kB (R) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3956kB
May 17 06:16:01 PIMSserver kernel: [18538.681353] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
May 17 06:16:01 PIMSserver kernel: [18538.681355] 554 total pagecache pages
May 17 06:16:01 PIMSserver kernel: [18538.681357] 453 pages in swap cache
May 17 06:16:01 PIMSserver kernel: [18538.681360] Swap cache stats: add 526366, delete 525913, find 169308/273582
May 17 06:16:01 PIMSserver kernel: [18538.681362] Free swap = 4110736kB
May 17 06:16:01 PIMSserver kernel: [18538.681363] Total swap = 4194300kB
May 17 06:16:01 PIMSserver kernel: [18538.681365] 130972 pages RAM
May 17 06:16:01 PIMSserver kernel: [18538.681367] 0 pages HighMem/MovableOnly
  • pimsserver
  • 06:16:01
  • kernel
  • May
    7 replies  •  2016-05-20 09:24:05 +08:00
    meiriyitie
        1
    meiriyitie  
       May 17, 2016
    重装系统大法？ 也许 do 想让你升级到 10 刀哈哈
    tvallday
        2
    tvallday  
       May 17, 2016 via iPad
    Swap space 太小了？整个大的？其实 10 刀的应该就不会出现 memory 不够的情况了，节省很多无谓的时间。
    ksky
        3
    ksky  
       May 17, 2016
    装 Mysql 了吗。内存不够不是 apache 就是 mysql 的问题。显然是有人在弄你的网站。
    kosenpai
        4
    kosenpai  
    OP
       May 18, 2016
    @meiriyitie 现在正在试新建一个，看看是不是还是有问题。
    kosenpai
        5
    kosenpai  
    OP
       May 18, 2016
    @tvallday 应该不是内存的问题，从图里可以看到崩溃的时候 CPU 爆炸，但是内存还是正常的。写了 tickets 问 DO ，但是他们也不给建议……
    kosenpai
        6
    kosenpai  
    OP
       May 18, 2016
    @ksky mysql 和 apache 是刚装的，之前只有 PPTP 和 ss 。崩溃在 mysql 安装之前就开始发生了。现在每天都崩溃一次，弄的我也崩溃了。不过谢谢提醒，也许有人通过 PPTP 来攻击我呢？我去查查 log 。
    kosenpai
        7
    kosenpai  
    OP
       May 20, 2016
    我查了 log 发现每条都和 Supervisor 有关，我都忘了什么时候装了这个。现在卸载掉之后稳定运行一天了。再看一段时间有没有问题。
    About   ·   Help   ·   Advertise   ·   Blog   ·   API   ·   FAQ   ·   Solana   ·   4458 Online   Highest 6679   ·     Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 · 50ms · UTC 10:04 · PVG 18:04 · LAX 03:04 · JFK 06:04
    ♥ Do have faith in what you're doing.