Discussion Forums
Discussion Forums > Category: Compute > Forum: Amazon Elastic Compute Cloud (EC2) >Thread: Page allocation failure. order:5, mode:0x20
Advanced search options
Page allocation failure. order:5, mode:0x20
Posted by: Berend de Boer
Posted on: Mar 16, 2010 11:18 AM
  Click to reply to this thread Reply
This question is not answered. Answer it to earn points.
I'm seeing a fair amount of the following page allocation failures. Can anyone tell me if this is important? What could the cause be? This is a Ubuntu image on a 32-bit high CPU instance.

All pointers appreciated.


Mar 16 07:10:07 server2 kernel: [887576.432964] __ratelimit: 87 callbacks suppressed
Mar 16 07:10:07 server2 kernel: [887576.432968] swapper: page allocation failure. order:5, mode:0x20
Mar 16 07:10:07 server2 kernel: [887576.432972] Pid: 0, comm: swapper Not tainted 2.6.31-302-ec2 #7-Ubuntu
Mar 16 07:10:07 server2 kernel: [887576.432975] Call Trace:
Mar 16 07:10:07 server2 kernel: [887576.432984]  [<c04c4f42>] ? printk+0x18/0x1e
Mar 16 07:10:07 server2 kernel: [887576.432991]  [<c0192e00>] __alloc_pages_slowpath+0x340/0x480
Mar 16 07:10:07 server2 kernel: [887576.432996]  [<c019304f>] __alloc_pages_nodemask+0x10f/0x120
Mar 16 07:10:07 server2 kernel: [887576.433000]  [<c01b911c>] T.653+0x3c/0xc0
Mar 16 07:10:07 server2 kernel: [887576.433004]  [<c01b925e>] T.651+0xbe/0x310
Mar 16 07:10:07 server2 kernel: [887576.433009]  [<c03deb84>] ? notify_remote_via_irq+0x64/0xe0
Mar 16 07:10:07 server2 kernel: [887576.433012]  [<c01b96ae>] cache_alloc_refill+0x1fe/0x240
Mar 16 07:10:07 server2 kernel: [887576.433016]  [<c01ba45e>] __kmalloc+0x10e/0x1a0
Mar 16 07:10:07 server2 kernel: [887576.433021]  [<c042f3c3>] pskb_expand_head+0x53/0x170
Mar 16 07:10:07 server2 kernel: [887576.433025]  [<c043107c>] __pskb_pull_tail+0x4c/0x2d0
Mar 16 07:10:07 server2 kernel: [887576.433031]  [<c0455d71>] ? nf_iterate+0x71/0x90
Mar 16 07:10:07 server2 kernel: [887576.433035]  [<c043abb3>] dev_queue_xmit+0x143/0x500
Mar 16 07:10:07 server2 kernel: [887576.433038]  [<c0455f06>] ? nf_hook_slow+0x96/0xd0
Mar 16 07:10:07 server2 kernel: [887576.433053]  [<c046153f>] ip_finish_output+0x12f/0x2c0
Mar 16 07:10:07 server2 kernel: [887576.433056]  [<c046176f>] ip_output+0x9f/0xb0
Mar 16 07:10:07 server2 kernel: [887576.433064]  [<c045eda0>] ? dst_output+0x0/0x10
Mar 16 07:10:07 server2 kernel: [887576.433067]  [<c045fbb8>] ip_local_out+0x18/0x20
Mar 16 07:10:07 server2 kernel: [887576.433070]  [<c046005c>] ip_queue_xmit+0x13c/0x370
Mar 16 07:10:07 server2 kernel: [887576.433074]  [<c0129269>] ? enqueue_entity+0xc9/0x140
Mar 16 07:10:07 server2 kernel: [887576.433077]  [<c0472c7b>] ? tcp_established_options+0x2b/0xb0
Mar 16 07:10:07 server2 kernel: [887576.433080]  [<c0472d3d>] ? tcp_current_mss+0x3d/0x60
Mar 16 07:10:07 server2 kernel: [887576.433083]  [<c0473459>] tcp_transmit_skb+0x399/0x650
Mar 16 07:10:07 server2 kernel: [887576.433086]  [<c0470fcf>] ? tcp_clean_rtx_queue+0x5af/0x790
Mar 16 07:10:07 server2 kernel: [887576.433089]  [<c04755c7>] tcp_write_xmit+0x167/0x460
Mar 16 07:10:07 server2 kernel: [887576.433092]  [<c0475ca1>] __tcp_push_pending_frames+0x31/0x90
Mar 16 07:10:07 server2 kernel: [887576.433095]  [<c046d6c6>] tcp_data_snd_check+0x26/0xe0
Mar 16 07:10:07 server2 kernel: [887576.433097]  [<c0471abb>] tcp_rcv_established+0x2eb/0x540
Mar 16 07:10:07 server2 kernel: [887576.433101]  [<c04798ad>] tcp_v4_do_rcv+0xbd/0x110
Mar 16 07:10:07 server2 kernel: [887576.433103]  [<c047a70e>] tcp_v4_rcv+0x5be/0x6f0
Mar 16 07:10:07 server2 kernel: [887576.433107]  [<c045c650>] ? ip_local_deliver_finish+0x0/0x1f0
Mar 16 07:10:07 server2 kernel: [887576.433110]  [<c045c6e7>] ip_local_deliver_finish+0x97/0x1f0
Mar 16 07:10:07 server2 kernel: [887576.433112]  [<c045c8c7>] ip_local_deliver+0x87/0x90
Mar 16 07:10:07 server2 kernel: [887576.433115]  [<c045c650>] ? ip_local_deliver_finish+0x0/0x1f0
Mar 16 07:10:07 server2 kernel: [887576.433118]  [<c045c2c3>] ip_rcv_finish+0xe3/0x360
Mar 16 07:10:07 server2 kernel: [887576.433122]  [<c04a9ff4>] ? packet_rcv_spkt+0xf4/0x130
Mar 16 07:10:07 server2 kernel: [887576.433125]  [<c045caca>] ip_rcv+0x1fa/0x2b0
Mar 16 07:10:07 server2 kernel: [887576.433128]  [<c04a9ff4>] ? packet_rcv_spkt+0xf4/0x130
Mar 16 07:10:07 server2 kernel: [887576.433131]  [<c0439f87>] netif_receive_skb+0x387/0x500
Mar 16 07:10:07 server2 kernel: [887576.433135]  [<c03ee2e4>] netif_poll+0x274/0x7f0
Mar 16 07:10:07 server2 kernel: [887576.433138]  [<c043a917>] net_rx_action+0x127/0x280
Mar 16 07:10:07 server2 kernel: [887576.433142]  [<c0135a9c>] __do_softirq+0xac/0x1e0
Mar 16 07:10:07 server2 kernel: [887576.433146]  [<c016f7e7>] ? handle_IRQ_event+0x97/0x180
Mar 16 07:10:07 server2 kernel: [887576.433149]  [<c0135c55>] do_softirq+0x85/0x90
Mar 16 07:10:07 server2 kernel: [887576.433152]  [<c0135d6d>] irq_exit+0x2d/0x40
Mar 16 07:10:07 server2 kernel: [887576.433155]  [<c03de4c5>] evtchn_do_upcall+0x1c5/0x1f0
Mar 16 07:10:07 server2 kernel: [887576.433159]  [<c011dfd1>] ? __dequeue_entity+0x21/0x40
Mar 16 07:10:07 server2 kernel: [887576.433163]  [<c010483b>] hypervisor_callback+0x43/0x4b
Mar 16 07:10:07 server2 kernel: [887576.433166]  [<c0106535>] ? xen_safe_halt+0x15/0x40
Mar 16 07:10:07 server2 kernel: [887576.433169]  [<c0109fa4>] xen_idle+0x34/0xd0
Mar 16 07:10:07 server2 kernel: [887576.433172]  [<c01033c7>] cpu_idle+0x47/0x70
Mar 16 07:10:07 server2 kernel: [887576.433176]  [<c04b8765>] rest_init+0x55/0x60
Mar 16 07:10:07 server2 kernel: [887576.433181]  [<c062cb77>] start_kernel+0x2d3/0x2d9
Mar 16 07:10:07 server2 kernel: [887576.433184]  [<c062c6c1>] ? unknown_bootoption+0x0/0x1ad
Mar 16 07:10:07 server2 kernel: [887576.433187]  [<c062c05d>] i386_start_kernel+0x5d/0x64
Mar 16 07:10:07 server2 kernel: [887576.433189] Mem-Info:
Mar 16 07:10:07 server2 kernel: [887576.433190] DMA per-cpu:
Mar 16 07:10:07 server2 kernel: [887576.433192] CPU    0: hi:    0, btch:   1 usd:   0
Mar 16 07:10:07 server2 kernel: [887576.433194] CPU    1: hi:    0, btch:   1 usd:   0
Mar 16 07:10:07 server2 kernel: [887576.433196] Normal per-cpu:
Mar 16 07:10:07 server2 kernel: [887576.433197] CPU    0: hi:  155, btch:  38 usd: 152
Mar 16 07:10:07 server2 kernel: [887576.433199] CPU    1: hi:  155, btch:  38 usd:  97
Mar 16 07:10:07 server2 kernel: [887576.433201] HighMem per-cpu:
Mar 16 07:10:07 server2 kernel: [887576.433202] CPU    0: hi:  155, btch:  38 usd:  27
Mar 16 07:10:07 server2 kernel: [887576.433204] CPU    1: hi:  155, btch:  38 usd:  39
Mar 16 07:10:07 server2 kernel: [887576.433208] Active_anon:21159 active_file:114614 inactive_anon:70967
Mar 16 07:10:07 server2 kernel: [887576.433209]  inactive_file:66060 unevictable:0 dirty:1163 writeback:0 unstable:0
Mar 16 07:10:07 server2 kernel: [887576.433210]  free:97303 slab:62334 mapped:9780 pagetables:0 bounce:0
Mar 16 07:10:07 server2 kernel: [887576.433214] DMA free:2872kB min:76kB low:92kB high:112kB active_anon:0kB inactive_anon:0kB active_file:7204kB inactive_file:72kB unevictable:0kB present:16256kB pages_scanned:0 all_unreclaimable? no
Mar 16 07:10:07 server2 kernel: [887576.433216] lowmem_reserve[]: 0 698 1710 1710
Mar 16 07:10:07 server2 kernel: [887576.433222] Normal free:42460kB min:3340kB low:4172kB high:5008kB active_anon:400kB inactive_anon:38616kB active_file:227928kB inactive_file:122780kB unevictable:0kB present:715256kB pages_scanned:0 all_unreclaimable? no
Mar 16 07:10:07 server2 kernel: [887576.433224] lowmem_reserve[]: 0 0 8095 8095
Mar 16 07:10:07 server2 kernel: [887576.433229] HighMem free:343880kB min:512kB low:1720kB high:2932kB active_anon:84236kB inactive_anon:245252kB active_file:223324kB inactive_file:141388kB unevictable:0kB present:1036260kB pages_scanned:0 all_unreclaimable? no
Mar 16 07:10:07 server2 kernel: [887576.433232] lowmem_reserve[]: 0 0 0 0
Mar 16 07:10:07 server2 kernel: [887576.433235] DMA: 482*4kB 4*8kB 1*16kB 0*32kB 0*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 2872kB
Mar 16 07:10:07 server2 kernel: [887576.433244] Normal: 8451*4kB 774*8kB 106*16kB 10*32kB 7*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 42460kB
Mar 16 07:10:07 server2 kernel: [887576.433252] HighMem: 20554*4kB 20748*8kB 4978*16kB 465*32kB 18*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 343880kB
Mar 16 07:10:07 server2 kernel: [887576.433261] 197423 total pagecache pages
Mar 16 07:10:07 server2 kernel: [887576.433262] 12617 pages in swap cache
Mar 16 07:10:07 server2 kernel: [887576.433264] Swap cache stats: add 58229, delete 45612, find 4735794/4737784
Mar 16 07:10:07 server2 kernel: [887576.433266] Free swap  = 762596kB
Mar 16 07:10:07 server2 kernel: [887576.433267] Total swap = 917496kB
Mar 16 07:10:07 server2 kernel: [887576.439039] 447488 pages RAM
Mar 16 07:10:07 server2 kernel: [887576.439041] 263170 pages HighMem
Mar 16 07:10:07 server2 kernel: [887576.439042] 10262 pages reserved
Mar 16 07:10:07 server2 kernel: [887576.439043] 138763 pages shared
Mar 16 07:10:07 server2 kernel: [887576.439045] 262435 pages non-shared
Permlink Replies: 5 | Pages: 1 - Last Post: Dec 27, 2010 11:12 AM by: Berend de Boer
Replies
Re: Page allocation failure. order:5, mode:0x20
Posted by: addady
Posted on: May 10, 2010 11:09 PM
in response to: Berend de Boer in response to: Berend de Boer
  Click to reply to this thread Reply
Hi,

I have the same problem using this kernel:
2.6.18-xenU-ec2-v1.2

--kernel aki-f5c1219c --ramdisk ari-dbc121b2

Which kernel you are using?

Addady
Re: Page allocation failure. order:5, mode:0x20
Posted by: Berend de Boer
Posted on: Aug 15, 2010 7:24 PM
in response to: addady in response to: addady
  Click to reply to this thread Reply
Sorry addady, never saw your reply. Still have this issue.

# uname -a
Linux server2.xplainhosting.com 2.6.31-302-ec2 #7-Ubuntu SMP Tue Oct 13 19:06:04 UTC 2009 i686 GNU/Linux

That's kernel aki-5f15f636 and ramdisk ari-d5709dbc
Re: Page allocation failure. order:5, mode:0x20
Posted by: Berend de Boer
Posted on: Dec 27, 2010 11:05 AM
in response to: Berend de Boer in response to: Berend de Boer
  Click to reply to this thread Reply
Get exactly the same thing with new server on latest Ubuntu 10.04 LTS. I'm monitoring memory allocation with collectd, absolutely no sign I'm running out of memory one way or another.
Re: Page allocation failure. order:5, mode:0x20
Posted by: Berend de Boer
Posted on: Dec 27, 2010 11:08 AM
in response to: Berend de Boer in response to: Berend de Boer
  Click to reply to this thread Reply
Re: Page allocation failure. order:5, mode:0x20
Posted by: Berend de Boer
Posted on: Dec 27, 2010 11:12 AM
in response to: Berend de Boer in response to: Berend de Boer
  Click to reply to this thread Reply
Hmm, claim here it is actually a messae about the need to tune the kernel? https://bugs.launchpad.net/ubuntu/+source/linux-ec2/+bug/648721

Have followed suggestion, see how this goes.