Introducing Smithy IDL 2.0

The AWS Smithy team is happy to announce the 2.0 release of the Smithy Interface Definition Language (IDL). This release focuses on improving the developer experience of authoring Smithy models and using code generated from Smithy models. It contains numerous new features, such as reduced nulls and optional types in generated code, custom default values, mixins to reduce model duplication, resource properties to improve consistency across operations, dedicated enumeration shapes, and syntax improvements.

Smithy is a protocol-agnostic IDL and set of tools for generating clients, servers, documentation, and other artifacts. It’s purpose-built for code generation, can be extended with custom traits, and enables automatic API standards enforcement. Smithy is Amazon’s next-generation API modeling language, based on our experience building tens of thousands of services and generating SDKs.

Fewer nulls in generated code

One goal of Smithy IDL 2.0 is to reduce the amount of nullable properties in generated code. With the addition of Rust, Kotlin, and Swift as supported programming languages in AWS SDKs, it became apparent that we needed a better approach for defining nullability in Smithy models to make generated code in these languages more idiomatic. However, we also had to balance this desire against the business requirement of allowing services to safely evolve their models over long periods of time (sometimes decades).

Prior to Smithy 2.0, Smithy code generators essentially treated every structure member as nullable. This was because: 1) nullability was somewhat confusing and people modeling structures didn’t realize that default values were provided by only some of the shapes targeted by members, and 2) the @required trait could be backward compatibly removed at any time because it was considered the removal of a constraint.

Consider the following Smithy IDL 1.0 example:

$version: "1.0"
namespace smithy.example

structure Message {
    // Non-null because it has a default value of false.
    delivered: MyBoolean,

    // Nullable because the required trait can be removed.
    @required
    message: String,
}

// Not marked with the box trait, so it provides a default
// value to members that target it.
boolean MyBoolean

In the above example:

1. The delivered member of Message has a default value of false because MyBoolean isn’t marked with the @box trait. These semantics were inherited from Smithy’s predecessor within Amazon, and many teams get confused by how these semantics work because it’s a confusing kind of “action at a distance” from the member definition.
2. The message member is nullable because the @required trait can be removed at any time. This allowed teams using Smithy 1.0 to remove the @required trait from a member if they ever needed to due to changing business requirements. As such, using the @required trait to influence code-generated types was expressly prohibited by the specification.

Smithy IDL 2.0 introduces custom default values for structure members. The previous example is written in Smithy IDL 2.0 as:

$version: "2"
namespace smithy.example

structure Message {
    delivered: MyBoolean = false

    @required
    message: String
}

boolean MyBoolean

With the introduction of default values in Smithy IDL 2.0, we realized that we can still provide almost the same level of model evolution flexibility while still reducing nullability in generated code. In Smithy IDL 2.0, if the @required trait is ever removed from a member, the member has to be provided a default value. This new affordance means that code generators can safely use the @required trait and default values to generate types that contain non-nullable properties. For example, if a structure member is @required, then accessing the member will always return a value. The same is true for members that have default values: they will always return a value even if one wasn’t explicitly provided by the end-user.

Without these updates, languages like Rust that explicitly model nullability in their type systems would generate optional accessors. This would cause users to have to unsafely dereference most structure members.

The nullability improvements in Smithy IDL 2.0 simplify how nullability is modeled, still provides developers the ability to evolve models over time if they need to remove the @required trait, and explicitly model default values of structure members rather than only relying on documentation.

Please note that implementing these Smithy IDL 2.0 nullability semantics in Smithy code generators and AWS SDKs is still a work in progress, but they will be incorporated before they made are generally available.

You can read more about member nullability in the Smithy documentation about structure member optionality.

Mixins to reduce model duplication

When authoring a Smithy model, the same members often need to be re-defined in related structures. Manually copy-pasting parts of shapes in a model is not only tedious; it’s also error prone and can lead to unintentional drift between shapes over time as different parts of a service are updated. Mixins are a kind of model-time copy-paste that eliminates this kind of model duplication. You can learn more about mixins in the Smithy documentation for mixins.

Defining resources in Smithy 1.0 often requires a significant amount of duplication across inputs and outputs. The following example describes a single operation in Smithy 1.0:

$version: "1.0"
namespace smithy.example

resource City {
    identifiers: { cityId: String },
    create: CreateCity
}

operation CreateCity {
    input: CreateCityInput,
    output: CreateCityOutput
}

@input
structure CreateCityInput {
    name: String,
    population: Integer,
    foundedOn: Timestamp,
}

@output
structure CreateCityOutput {
    cityId: String,
    name: String,
    population: Integer,
    foundedOn: Timestamp,
}

Utilizing mixins for this API reduces the repetition and removes surface area for inconsistencies in the model (especially as more operations are added). Here is an updated version of the above model, using resource properties (explained later in the blog post) and mixins:

$version: "2.0"
namespace smithy.example

resource City {
    identifiers: { cityId: String }
    properties: {
        name: String
        population: Integer
        foundedOn: Timestamp
    }
    create: CreateCity
}

operation CreateCity {
    input := with [CityData] {}
    output := with [CityRecord] {}
}

@mixin
structure CityData for City {
    $name
    $population
    $foundedOn
}

@mixin
structure CityRecord with [CityIdentifiers, CityData] {}

@mixin
structure CityIdentifiers for City {
    @required
    $cityId
}

Mixins can be used with other shape types as well, like enumerations of various platform options, operations with a consistent set of errors, and services that share common operations like this example:

service WeatherService with [TaggableService] {
    // This service inherits the operations of TaggableService.
}

@mixin
service TaggableService {
    operations: [
        TagResource
        UntagResource
        ListTagsForResource
    ]
}

Resource properties

Smithy models can define resources to communicate the “things” in an API. Resources in Smithy 1.0 only included operations and identifiers. This allowed services to define the resources in their APIs and ensure that operations are provided the right identifiers, but the lack of validation to ensure that resources expose properties consistently across operations can lead to usability issues. For example, it was easy to define a service with a PutFoo operation that accepts an idList member in its input while a GetFoo operation returns a corresponding member as ids. This kind of inconsistency is undesirable because it puts a usability burden on end users to determine that these members mean the same thing.

Smithy 2.0 adds the ability to define the properties of a resource. With this new declaration, Smithy will automatically detect when a team accidentally uses the wrong property name in the input or output of a resource based operation.

$version: "2"
namespace smithy.example

resource City {
    identifiers: { cityId: CityId }
    properties: {
        name: String
        population: Integer
        foundedOn: Timestamp
    }
    create: CreateCity
}

Each property defines a name and the shape it targets. The input and output of resource operations are limited to members that match these property names and targets (with a few caveats explained in the specification). The new @notProperty trait can be used to indicate that a member of an instance operation isn’t a resource property, like a dryRun boolean or an idempotency token.

Smithy 2.0 also introduces a new feature called member elision to refer to resource identifiers, resource properties, or mixin members. To use member elision, add a $ character before a member name and omit the target of the member so it is inherited. This syntax eliminates the need to redefine the shape targeted by the referenced component, and makes it clear that the target is inherited.

operation CreateCity {
    input := for City {
        @required
        $name

        @required
        $population

        @required
        $location
    }
}

Note: In order to refer to resource properties or identifiers with member elision syntax, bind the resource to a structure using the following for syntax.

You can learn more about these new features in the Smithy documentation for resource properties and target elision.

Dedicated enumeration shapes

Smithy 1.0 previously defined enumerations using an @enum trait on a string shape. This resulted in several issues: the syntax used to define enums was verbose, enum values weren’t members so removing enum values from different projections (or views) or a Smithy model required special-casing, and the metadata associated with an enum value duplicated traits like @documentation and @tags.

Smithy 2.0 introduces the enum and intEnum shapes and deprecates the @enum trait. enum and intEnum are used to represent a fixed set of string and integer values. You can learn more in the Smithy documentation for enum shapes and intEnum shapes.

Making enums dedicated shape types improves the modeling experience and makes their use with other parts of Smithy more consistent. Enum values are now members of enum shapes, meaning documentation, tags, and other properties now use the standard Smithy traits instead of additional properties of enum value definitions. Since enums are now shapes with their values as members, they’re affected by model transformations without needing to write specialized code.

This example shows how enums were defined in Smithy 1.0:

$version: "1.0"
namespace smithy.example

@enum([
    {
        value: "t2.nano",
        name: "T2_NANO",
        documentation: """
            T2 instances are Burstable Performance
            Instances that provide a baseline level of CPU
            performance with the ability to burst above the
            baseline.""",
        tags: ["ebsOnly"]
    },
    {
        value: "t2.micro",
        name: "T2_MICRO",
        documentation: """
            T2 instances are Burstable Performance
            Instances that provide a baseline level of CPU
            performance with the ability to burst above the
            baseline.""",
        tags: ["ebsOnly"]
    },
    {
        value: "m256.mega",
        name: "M256_MEGA",
        deprecated: true
    }
])
string MyString

This changes significantly when using an enum shape in Smithy 2.0, utilizing traits and documentation comments instead.

$version: "2"
namespace smithy.example

enum MyString {
    /// T2 instances are Burstable Performance Instances that
    /// provide a baseline level of CPU performance with the
    /// ability to burst above the baseline.
    @tags["ebsOnly"]
    T2_NANO = "t2.nano"
    
    /// T2 instances are Burstable Performance Instances that
    /// provide a baseline level of CPU performance with the
    /// ability to burst above the baseline.
    @tags["ebsOnly"]
    T2_MICRO = "t2.micro"
    
    @deprecated
    M256_MEGA = "m256.mega"
}

The intEnum shape type provides the ability to define enumerated integers, rather than just strings. This enables compatibility with various serialization formats that use integers for enums rather than strings.

$version: "2"
namespace smithy.example

intEnum FaceCard {
    JACK = 1
    QUEEN = 2
    KING = 3
    ACE = 4
    JOKER = 5
}

$version: "1.0"
namespace smithy.example

operation CreateCity {
    input: CreateCityInput,
    output: CreateCityOutput,
}

@input
structure CreateCityInput {
    @required
    name: String,

    population: Integer,
    foundedOn: Timestamp,
}

@output
structure CreateCityOutput {
    @required
    cityId: String,

    @required
    name: String,

    population: Integer,
    foundedOn: Timestamp,
}

$version: "2"
namespace smithy.example

operation CreateCity {
    input := {
        @required
        name: String

        population: Integer
        foundedOn: Timestamp
    }
    output := {
        @required
        cityId: String

        @required 
        name: String

        population: Integer
        foundedOn: Timestamp
    }
}

$version: "2"
namespace smithy.example

operation CreateCity {
    input := for City {
        $name
        $population
        $foundedOn
    }
    output := for City {
        @required
        $cityId

        $name
        $population
        $foundedOn
    }
}

To see the complete list of changes for this release, check out the Smithy repository changelog.

Next Steps

• Already using Smithy 1.0 and ready to migrate to Smithy 2.0? Get started with the IDL 1.0 to 2.0 migration guide.
• If you are new to Smithy and IDL 2.0 and want to explore more, take a look at the Smithy documentation and watch this introductory video from Michael Dowling, one of Smithy’s creators.
• For feature requests, bug reports, contributions, or feedback of any kind, head over to the Smithy GitHub repository.