Use Git pre-commit hooks to avoid AWS CloudFormation errors

Do you struggle with formatting AWS CloudFormation templates? Do you wonder if your Python and shell scripts are 100% neat and error-free? Do you spend time and mental energy fixing missing semicolons and trailing white space at the end of your files? If so, there’s a better way to avoid errors: Use GitHub pre-commit hooks.

These hooks catch and sometimes even fix formatting issues and messy scripts before you commit changes to your Git repository.

With the approach we propose, you configure pre-commit hooks for your project and commit your changes to the repo. Then Git inspects your files and returns a list of any issues that must be fixed. If Git flags issues, the commit fails. If not, your code is committed to the project.

For example, here on the AWS Integration and Automation team, when we build AWS Quick Starts, we commonly use the following pre-commit hooks:

cfn-lint: AWS CloudFormation linter. Validates general mappings, resources, parameters, and other components in both YAML and JSON-based CloudFormation templates.
black: Formats your Python code.
isort: Sorts your imports in alphabetical order.
seed-isort-config: Sorts your third-party imports and integrates them into isort.
flake8: Enforces style consistency in Python code.
shellcheck: Checks shell scripts for accuracy.
mixed-line-ending: Replaces or checks mixed line endings.
trailing-whitespace: Trims trailing white space.

In this post, we walk you through an exercise that helps you learn, first-hand, how pre-commit hooks work.

About this blog post
Time to read	~15 min.
Time to complete	~25 min.
Cost to complete	~$0
Learning level	Intermediate (200)
AWS services	AWS CloudFormation Amazon Elastic Kubernetes Service (Amazon EKS)

Process overview

Our exercise has five main steps:

Fork and then clone the test repo.
Install Python, pip, and the AWS CloudFormation linter hook.
Configure your system and Git project for the pre-commit hooks.
Install the pre-commit hooks.
Test your pre-commit hooks.

For the test repo, we chose the Amazon EKS Quick Start because it represents complex configurations in multiple use cases. This Quick Start uses an AWS CloudFormation template to automate the deployment of the reference architecture shown in Figure 1. This architecture includes an Amazon EKS cluster, Kubernetes nodes spanning two or three Availability Zones, and a Linux bastion host in a public subnet.

Figure 1. Architecture diagram for the test repo, the Amazon EKS Quick Start

Prerequisites

This blog post assumes that you have a basic understanding of the following:

Git and GitHub
AWS CloudFormation templates
AWS Quick Starts
YAML, Python, and shell scripts

Walkthrough

You start this exercise by forking the Amazon EKS Quick Start in your GitHub account and then cloning it to your local machine, as described in step 1. When you work with a clone, you don’t accidentally push unwanted changes upstream to the active repo that’s in production.

Step 1: Fork and then clone the test repo

Log in to your GitHub account at https://github.com/login.
Go to https://github.com/aws-quickstart/quickstart-amazon-eks.
In the upper right corner of the page, choose Fork.
Choose your own account as the location of the fork.
Clone the repo to a local directory on your system by running the following command. Replace <your-account> with the name of your account.
```
git clone https://github.com/<your-account>/quickstart-amazon-eks.git
```

Step 2: Install Python, pip, and the AWS CloudFormation linter hook

To use pre-commit hooks, you need both Python and pip. (Python 3 is the programming language that we use to install the pre-commit hooks. Pip 3 is the standard package manager used with Python to manage additional libraries of code.) You need the AWS CloudFormation linter to run validation tests of your CloudFormation template files.

Install Python 3 on your system. Follow the instructions in the Python 3 Installation & Setup Guide. For specific Windows releases of Python 3, see Python Releases for Windows.
Install pip 3 on your system. Follow the instructions in Installing pip/setuptools/wheel with Linux Package Managers.
Install the AWS CloudFormation linter hook by running this command: pip install cfn-lint

Step 3: Configure your system and Git project for the pre-commit hooks

In this section, you use pip to install the pre-commit hooks: isort (including the script-must-have-extension hook), seed-isort-config, black, flake8, mixed-line-editing, trailing-whitespace, and shellcheck. If you’re working with a non-Windows operating system, such as macOS, see the pre-commit documentation site for additional information.

Install the pre-commit package manager by running this command: pip install pre-commit
Navigate to the root directory of the Quick Start project that you forked earlier, and create the .pre-commit-config.yaml file by running the following command: touch .pre-commit-config.yaml

Copy then paste the following text into the .pre-commit-config.yaml file. This step lets the installer know which pre-commit hooks to include in your project. Save your file.

repos:
 -   repo: https://github.com/pre-commit/pre-commit-hooks
     rev: v2.3.0
     hooks:
     -   id: mixed-line-ending
     -   id: trailing-whitespace
     -   id: flake8
 -   repo: https://github.com/ambv/black
     rev: stable
     hooks:
     - id: black
       language_version: python3.7
 -   repo: https://github.com/asottile/seed-isort-config
     rev: v2.1.1
     hooks:
     -   id: seed-isort-config
         args: [--exclude=templates/]
 -   repo: https://github.com/pre-commit/mirrors-isort
     rev: v4.3.21  # Use the revision sha / tag you want to point at
     hooks:
     -   id: isort
         files: ^source/
         types: [file, python]
 -   repo: https://github.com/jumanjihouse/pre-commit-hooks
     rev: 2.0.0
     hooks:
     - id: script-must-have-extension
       name: QuickStart policy is to use .sh extension for shell scripts
       files: >
         (?x)^(
              function/scource/*|
              scripts/*
         )$
       types: [shell, executable]
     - id: shellcheck
       types: [shell, executable]

In the root directory of the project, create the configuration file for the isort Python library by running this command: touch .isort.cfg

In the new .isort.cfg file, add the following text, and save the file.

[settings]
line_length = 88
multi_line_output = 3
include_trailing_comma = True
known_third_party = boto3,crhelper,requests,ruamel

While still in the root directory of the project, create the configuration file for the black Python library by running this command: touch pyproject.toml

In the new pyproject.toml file, add the following text, and save the file. Note that the pyproject.toml file excludes certain directories when the Quick Start doesn’t include Python files.

[tool.black]
 line-length = 88
 target_version = ['py36', 'py37', 'py38']
 include = '\.py'
 exclude = '''
   /(
       \.eggs         # exclude a few common directories in the
     | \.git          # root of the project
     | templates
     | docs
   )/
 '''

Still in the root directory of the project, create the configuration file for the flake8 Python library by running this command: touch .flake8
In the new .flake8 file, add the following text, and save the file. Note that we added a line to let flake8 know which errors and warning codes to ignore when it does its magic. To see a complete list of codes, see the pycodestyle documentation.
```
[flake8]
 max-line-length = 100
 max-complexity = 18
 select = B,C,E,F,W,T4,B9
 ignore = E203, E266, E501, W503, F403, F401
```

Step 4: Install the pre-commit hooks

Navigate to the root of your forked Quick Start directory, and install the hooks from the .pre-commit-config.yaml file by running the following command: pre-commit install
If you add or remove hooks in the .pre-commit-config.yaml file, apply the changes by running the following command: git add .pre-commit-config.yamlIf you don’t run that command after adding or removing hooks, then when you use your pre-commit hooks, you’ll receive the message “[ERROR] Your pre-commit configuration is unstaged. `git add .pre-commit-config.yaml` to fix this,” as shown in Figure 2.

Figure 2. Error message you may receive

Step 5: Test your pre-commit hooks

Run the pre-commit hooks against all the files in the Quick Start project with the following command: pre-commit run --all-filesThis command identifies all errors in the project at one time. If you encounter a similar naming issue in another Quick Start project, determine how shell scripts are called in the code. If you add the .sh file extension to your shell scripts, either update your template files or exclude those files from the pre-commit hook check.
Test a commit for errors as follows.
(a) In the templates directory of the Quick Start, add Default: 10.0.0.1/24 to the Parameters section in the amazon-eks-master.template.yaml file. Figure 3 gives an example of what your file might look like after you add this text:

Figure 3. Example file content

(b) Add and commit your text addition to Git by running the following commands:
```
git add amazon-eks-master.template.yaml
 $ git commit -m "Updated Parameter RemoteAccessCIDR"
```
Your output shows that the tests that passed or were skipped, as shown in Figure 4:

Figure 4. Tests passed or skipped

Cleanup

To clean up from the previous exercises, remove the forked repo and your local clone as follows.

Remove your local clone of the repo by running the following command: rm -Rf /path/to/QuickStart
Remove the forked repo from your GitHub account as follows.
a. Navigate to your forked repo page, and choose Settings.
b. IMPORTANT: At the bottom of the page, in the Danger Zone section, confirm that you’re in your forked repo, not the primary Quick Start for Amazon EKS Architecture repo. You know that you’re in the forked repo because the path includes your account name:<your-account>/quickstart-amazon-eks
c. Choose Delete this repository.
d. Enter the name of your forked repo.
(Optional) Remove the pre-commit hooks by running this command: pip uninstall pre-commit

Conclusion

Walking through the exercise in this post gives you a taste of how pre-commit hooks can save you time and frustration. We hope that you’re now thinking of ways to use pre-commit hooks with your AWS CloudFormation projects and others. As you use the hooks, keep exploring ways that they can benefit your work. If you feel inspired, develop your own pre-commit hook!

To learn more about pre-commit hooks and how you can use them to make your life easier, see the pre-commit documentation.

Please let us know your thoughts in the comments.

Integration & Automation