Using the EC2 Serial Console to access the Microsoft Server boot manager to fix and debug boot failures

This post is written by Pallavi Ravishankar a Senior Product Manager and Jason Nicholls an Enterprise Solutions Architect.

Failure management is a key part of the reliability pillar within the AWS Well-Architected Framework. But things fail, and operating systems are no exception. An operating system update, application update, a misconfiguration, missing driver, or incorrect security permissions can prevent systems from starting up correctly.

In a previous post, we demonstrated how you can access the GNU GRand Unified Boot-loader (GRUB) using EC2 Serial console to fix a failed Linux kernel load. In this blog post, we show you how you can use EC2 Serial Console to debug and fix your Amazon EC2 Windows Instances.

Configuration changes or software updates are two examples that could result in an Amazon EC2 Windows Instance start-up failure. In this post, you simulate a network failure caused by a misconfiguration of your Amazon EC2 Windows Instance. Then use the Microsoft Windows Special Administration Console (SAC) to debug and fix your EC2 Windows Instance.

Before you simulate the network failure, you must configure SAC to read and write from the instance’s virtual serial port.

Configuring SAC

The SAC interface lets you interact with the Microsoft Windows Operating System, providing administrative access even if network connectivity is not functional. SAC is not enabled by default and must be configured.

You can configure SAC via the Windows’ Command shell or PowerShell. Or you can set up SAC during your instance creation by using EC2 user data. User data is a feature of EC2 that allows you to specify parameters for configuring your instance, or include a simple script. The simple script is carried out at launch.

To launch an EC2 instance running Windows Server, choose an instance family that is built on the AWS Nitro System. The EC2 Serial Console access is only available for EC2 instances based on the AWS Nitro System. Configure the user data to set up SAC access with the following script:

<script>

bcdedit /ems {current} on

bcdedit /emssettings EMSPORT:1 EMSBAUDRATE:115200

bcdedit /set {bootmgr} displaybootmenu yes

bcdedit /set {bootmgr} timeout 15

bcdedit /set {bootmgr} bootems yes

</script>

Ensure that the operating system is Windows Server 2019, the user data is set, and the instance family is correct before launching your instance.

Once your instance is initialized, retrieve the Windows password from the EC2 console or AWS Command Line Interface (CLI). You have now successfully configured SAC access via the instance’s virtual serial port.

Accessing the SAC Menu

EC2 Serial Console can be used to access the EC2 instance’s virtual serial port. EC2 Serial Console access is not permitted by default at the account level. Enabling EC2 Serial Console requires that your user has permission to call EC2 API EnableSerialConsoleAccess. You can enable or disable EC2 Serial Console from the EC2 Console screen or via the CLI.

Enabling or disabling EC2 Serial Console applies to all instances in your account. Service Control Policies can be used to control access to EC2 Serial Console at an organization level. AWS Identity and Access Management (IAM) permissions control access at an instance level. You can exercise more granular controls at the instance level by setting a resource group or tag-based IAM policy. For more information about allowing access to EC2 Serial Console, see documentation section “Configure access to the EC2 Serial Console.”

Simulating a failed networking configuration

Previously, if an EC2 instance became unresponsive, the only available recourse was to shut it down, and mount the disk on a secondary EC2 instance. You could then use the secondary instance to fix the issue. Today, you can use EC2 Serial Console to debug the problem.

Let’s simulate a complete network failure on your newly created EC2 instance by shutting down the Ethernet service. Use Remote Desktop to connect to your EC2 instance. Once you’re connected, open a Windows command shell in administrative mode and run the following command:

netsh interface show interface

The command should show a list of network interfaces available. If you are using an AWS Windows Amazon Machine Image (AMI), the interface “Ethernet 3” should show as enabled. An example of what you should see is depicted in the following image.

Run the following netsh command to disable the network interface:

netsh interface set interface name=”Ethernet 3” admin=DISABLED

The network interface is disabled the moment you press enter and your Remote Desktop connection should shut down, as shown in the following screenshot.

Even if you reboot the instance, you will see that the connection to the instance fails. This is because the network interface has been disabled.

Fixing the network connection

EC2 Serial Console provides a Secure Shell (SSH) to securely access SAC via your Windows EC2 instance’s virtual serial port. Connection to the virtual serial port does not require instance network connectivity. Therefore, you can use EC2 Serial Console to fix the networking misconfiguration.

The SSH session is authorized using an SSH key pair. You can access the EC2 Serial Console using the:

• Amazon EC2 Console with a single click connection (browser based)
• AWS CLI
• Any SSH Client of your choice – openSSH, PuTTY, AWS CloudShell

In order to connect to EC2 Serial Console, you must generate a one-time SSH key locally on your client. To do this, use the AWS CLI to push the public key to the EC2 Serial Console service and use SSH to connect to the EC2 Serial Console endpoint. The Amazon EC2 Console combines all these steps into a single-click access. Detailed instructions of this process are available here.

For this blog post, we use AWS CloudShell. AWS CloudShell is a browser-based shell that makes it easy to securely manage, explore, and interact with your AWS resources. AWS CloudShell is pre-authenticated with your console credentials. You can launch AWS CloudShell directly from the AWS Management Console.

1. 1. From the AWS Management Console, choose the AWS CloudShell console by pressing the CloudShell icon:
   2. Generate a one-time SSH key pair using ssh-keygen.
      ssh-keygen -t rsa -f my_rsa_key
   3. Push your public key to EC2 Serial Console using the AWS CLI installed on AWS CloudShell.
      aws ec2-instance-connect send-serial-console-ssh-public-key \
--instance-id i-00123EXAMPLE \
--serial-port 0 \
--ssh-public-key file://my_rsa_key.pub
--region $REGION
   4. Start an SSH session to EC2 Serial Console.ssh -i my_rsa_key i-00123EXAMPLE.port0@serial-console.ec2-instance-connect.{region}.aws

Once you’re connected, press enter to see the SAC prompt. You can then run ch to see a list of available channels. To start a command shell channel, type cmd. You can then use ch -si 1 to access the newly created command shell channel. An example of the procedure is depicted in the following screenshot.

The console then presents you with the channel information screen after selecting the command shell channel, similar to the following image.

Press the enter key to be dropped into the channel.

The channel requests a username, domain, and password. The username is Administrator, the domain is empty, and the password is the password you retrieved earlier.

Now that you are authenticated, use the command shell to fix the problem.

Run the netsh command:

netsh interface show interface

After running the preceding commands, the command shell shows that we disabled the network interface. An example of this is illustrated in the following image.

Let’s undo our misconfiguration by running the netsh command:

netsh interface set interface name=”Ethernet 3” admin=ENABLED

You can now use Remote Desktop to access the instance again.

Without closing the EC2 Serial Console, reboot the instance by running the following command:

shutdown -r -t 0

Your instance then reboots, which you can see via EC2 Serial Console. First, it drops back to the SAC menu to inform you of a reboot. Notice that the Microsoft Windows boot manager menu on reboot as seen in the following image.

The advanced boot options screen presented in the following image let you start Windows in advanced troubleshooting modes. These modes include repairing the instance, rolling back to a previous configuration, debugging your instance, or starting up in safe mode. To access the advanced boot options, press Esc + 8 on the Windows Server [EMS Enabled] menu option.

By following this post, you setup SAC to read and write to the virtual serial port. You then disabled ethernet access. After confirming that you could no longer access your instance you used EC2 Serial Console to regain access and revert the changes.

Clean up

After you’ve finished with the instance you created for this post, you should clean up by deleting the instance. This will prevent you from incurring any additional costs. To delete the instance:

1. 1. 1. In the navigation pane, choose Instances. In the list of instances, select the instance.
      2. Choose Instance state, Terminate instance.
      3. Choose Terminate when prompted for confirmation.

Amazon EC2 shuts down and deletes your instance. After your instance is deleted, it remains visible on the console for a short while, and then the entry is automatically deleted. You cannot remove the deleted instance from the console display yourself.

Conclusion

EC2 Serial Console offers virtual serial port access to a Microsoft Windows EC2 Instance running on the AWS Nitro System. EC2 Serial Console facilitates the interaction with Special Administration Console to fix and debug instance issues. You can also use EC2 Serial Console to access the Microsoft boot menu to launch an instance in safe mode. You can also connect to the EC2 Serial Console of your Linux instances, which we covered in previous blog post. To learn more regarding EC2 Serial Console, see AWS Documentation or follow this Qwiklabs hands-on lab.