Discussion Forums
Discussion Forums > Category: Compute > Forum: Amazon Elastic Compute Cloud (EC2) >Thread: Instance reachability check failed Ubuntu 15.10 Wily c4.xlarge
Advanced search options
Instance reachability check failed Ubuntu 15.10 Wily c4.xlarge
Posted by: LaurenceRoweAtStanford
Posted on: Oct 26, 2015 2:42 PM
Attachment c4-system.log (63.1 KB)
  Click to reply to this thread Reply
ubuntu , wily , ec2
This question is answered.
I've made several attempts to launch a c4.xlarge and c4.8xlarge instances using Ubuntu 15.10 Wily but am unable to ping the instance after it has started running. The console shows that the instance reachability check failed.

I am able to successfully launch c4.xlarge instances using Ubuntu 14.04 and t2.large instances using Ubuntu 15.10.

I've tried with both of these instance AMIs:

ubuntu/images/hvm-ssd/ubuntu-wily-15.10-amd64-server-20151021 - ami-225ebd11
ubuntu/images-testing/hvm-ssd/ubuntu-wily-daily-amd64-server-20151026 - ami-ea20cdd9

Might there be a problem with the Ubuntu Kernel in 15.10 for the c4 instances?

Looking at the system log it seems that the network never comes up:

[  140.699509] cloud-init[1469]: 2015-10-26 20:45:49,887 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [0/120s]: request error [('Connection aborted.', OSError(101, 'Network is unreachable'))]


https://gist.github.com/lrowe/7dec7ff47cb44bd680ff

Edited by: LaurenceRoweAtStanford on Oct 26, 2015 2:57 PM
Permlink Replies: 8 | Pages: 1 - Last Post: Dec 10, 2015 2:57 PM by: PaulNamePros
Replies
Re: Instance reachability check failed Ubuntu 15.10 Wily c4.xlarge
Posted by: Neel@AWS
Posted on: Oct 26, 2015 8:16 PM
in response to: LaurenceRoweAtStanford in response to: LaurenceRoweAtStanford
  Click to reply to this thread Reply
Hello LaurenceRoweAtStanford

Thanks for reporting this issue. I am able to recreate this issue. We will investigate it further and will post our findings.

Regards
Neel
Re: Instance reachability check failed Ubuntu 15.10 Wily c4.xlarge
Posted by: LaurenceRoweAtStanford
Posted on: Oct 27, 2015 10:44 AM
in response to: LaurenceRoweAtStanford in response to: LaurenceRoweAtStanford
  Click to reply to this thread Reply
I've also reported this in Launchpad: https://bugs.launchpad.net/ubuntu/+bug/1510345

Looking through the log...

The network card does seem to be detected
[1367092.591159] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 2.12.1-k
[1367092.596324] ixgbevf: Copyright (c) 2009 - 2012 Intel Corporation.
[1367092.600410] FDC 0 is a S82078B
[1367092.614541] ixgbevf 0000:00:03.0: 02:f5:f9:85:fa:ed
[1367092.617407] ixgbevf 0000:00:03.0: MAC: 1
[1367092.619974] ixgbevf 0000:00:03.0: Intel(R) 82599 Virtual Function


But there is a failure waiting for it to come up:
[FAILED] Failed to start Wait for all "auto"...be up for network-online.target.
See 'systemctl status ifup-wait-all-auto.service' for details.
[  OK  ] Reached target Network is Online.
         Starting /etc/rc.local Compatibility...
         Starting Initial cloud-init job (metadata service crawler)...


No IP address is assigned to the network interface.
[  140.683912] cloud-init[1469]: ci-info: +++++++++++++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++++++++++++
[  140.685164] cloud-init[1469]: ci-info: +--------+-------+------------------------------+---------------+-------+-------------------+
[  140.686158] cloud-init[1469]: ci-info: | Device |   Up  |           Address            |      Mask     | Scope |     Hw-Address    |
[  140.686616] cloud-init[1469]: ci-info: +--------+-------+------------------------------+---------------+-------+-------------------+
[  140.688236] cloud-init[1469]: ci-info: |  ens3  | False |              .               |       .       |   .   | 02:f5:f9:85:fa:ed |
[  140.689527] cloud-init[1469]: ci-info: | lxcbr0 |  True |           10.0.3.1           | 255.255.255.0 |   .   | 6e:42:be:b1:2c:95 |
[  140.690922] cloud-init[1469]: ci-info: | lxcbr0 |  True | fe80::6c42:beff:feb1:2c95/64 |       .       |  link | 6e:42:be:b1:2c:95 |
[  140.691826] cloud-init[1469]: ci-info: |   lo   |  True |          127.0.0.1           |   255.0.0.0   |   .   |         .         |
[  140.692765] cloud-init[1469]: ci-info: |   lo   |  True |           ::1/128            |       .       |  host |         .         |
[  140.693148] cloud-init[1469]: ci-info: +--------+-------+------------------------------+---------------+-------+-------------------+
Re: Instance reachability check failed Ubuntu 15.10 Wily c4.xlarge
Posted by: OmarS@AWS
Posted on: Oct 28, 2015 11:08 AM
in response to: LaurenceRoweAtStanford in response to: LaurenceRoweAtStanford
  Click to reply to this thread Reply
Hi LaurenceRoweAtStanford,

Please try creating a /etc/network/interfaces.d/ens3.cfg with:

auto ens3
iface ens3 inet dhcp

As in our testing it seemed to fix the issue.

Best Regards,
Omar S.
Re: Instance reachability check failed Ubuntu 15.10 Wily c4.xlarge
Posted by: LaurenceRoweAtStanford
Posted on: Oct 28, 2015 12:54 PM
in response to: OmarS@AWS in response to: OmarS@AWS
  Click to reply to this thread Reply
OmarS@AWS wrote:

Please try creating a /etc/network/interfaces.d/ens3.cfg with:

auto ens3
iface ens3 inet dhcp

As in our testing it seemed to fix the issue.



I've added this using to my cloud-init script:

write_files:
  • path: /etc/network/interfaces.d/ens3.cfg
content: |
auto ens3
iface ens3 inet dhcp


And while ens3 is now listed at the end of Net device info rather than at the beginning, it is still shows Up: False and the 'Network is unreachable' errors remain.
Re: Instance reachability check failed Ubuntu 15.10 Wily c4.xlarge
Posted by: B. Howard RealName(TM)
Posted on: Oct 29, 2015 6:57 AM
in response to: LaurenceRoweAtStanford in response to: LaurenceRoweAtStanford
  Click to reply to this thread Reply
Canonical is currently addressing this issue and will be releasing updated images within the next day or so. We have confirmed the fix via a daily build which is in the process of being published to AWS. Once the images have passed automated publication and QA, I'll manually verify the fix and then re-release 15.10 for AWS.

For those interested in tracking the official progress, please see https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/1510345
Re: Instance reachability check failed Ubuntu 15.10 Wily c4.xlarge
Posted by: JoshuaF@AWS
Posted on: Oct 29, 2015 12:37 PM
in response to: B. Howard in response to: B. Howard
  Click to reply to this thread Reply
Thanks Ben.

I've verified that the daily you just published works on a c4.xlarge for me:
http://cloud-images.ubuntu.com/wily/20151029/

By the way, here's a little script to grab the latest GP2 "ebs-ssd" AMI ID:
#!/bin/bash
#set -x
#https://cloud-images.ubuntu.com/query/
 
RELEASE=${RELEASE:-wily}
PLATFORM=${PLATFORM:-hvm}
VERSION=${VERSION:-daily}
 
if [ -z $AWS_DEFAULT_REGION ]; then
  echo "Please set region"
  exit 1
fi
 
if [ ! -z $1 ]; then
  echo export RELEASE=$RELEASE
  echo export PLATFORM=$PLATFORM
  echo export VERSION=$VERSION
  exit 0
fi
 
URL=https://cloud-images.ubuntu.com/query/$RELEASE/server/$VERSION.current.txt
curl -s $URL  |grep "ebs-ssd.amd64.$AWS_DEFAULT_REGION" | grep $PLATFORM | cut -f 8


Regards,
Joshua F.
Re: Instance reachability check failed Ubuntu 15.10 Wily c4.xlarge
Posted by: B. Howard RealName(TM)
Posted on: Oct 29, 2015 4:29 PM
in response to: LaurenceRoweAtStanford in response to: LaurenceRoweAtStanford
  Click to reply to this thread Reply
Correct
New 15.10 images have been released at 1, 2.

Thank you all for your patience as we resolved this issue.

[1] http://cloud-images.ubuntu.com/releases/wily/release/
[2] http://cloud-images.ubuntu.com/locator/ec2/
Re: Instance reachability check failed Ubuntu 15.10 Wily c4.xlarge
Posted by: PaulNamePros
Posted on: Dec 10, 2015 2:57 PM
in response to: B. Howard in response to: B. Howard
  Click to reply to this thread Reply
This issue appears to have regressed. It's once again impossible to launch a new 15.10 instance on EC2. Tested with ami-9e723bf4.