Enabling file system sharing on Amazon FSx for NetApp ONTAP across multiple AWS accounts

Enterprise users are looking for a scalable and resilient network-attached storage (NAS) solution with high throughput performance on AWS to support distributed systems running Windows and Linux workloads. Moreover, users like to have flexible deployment models so that storage over the network can be consolidated and managed by the infrastructure account for cost optimization and ease of management. In addition, data can be shared to other AWS accounts within the same AWS Organization to achieve cost and operational efficiencies in their environments.

With Amazon FSx for NetApp ONTAP, AWS provides multi-protocol workloads with fully managed shared storage that provides scalable storage capacity and high performance with low latency storage over the network. FSx for ONTAP offers both Single-Availability Zone (AZ) and Multi-AZ file systems to address users’ system resiliency requirements. In this post, we demonstrate the approach of using VPC networking with AWS Transit Gateway and AWS Resource Access Manager (AWS RAM) to achieve cross-account access on FSx for ONTAP running in a Multi-AZ environment. Multi-AZ file systems support all the availability and durability features of Single-AZ file systems, and they are designed to provide continuous availability to data even when an AZ is unavailable. We also demonstrate how to configure fine-grained access control at the volume level using the ONTAP CLI to make sure of secure access in this solution.

As an alternative, you can consider using VPC sharing announced in December 2023. VPC Sharing is a feature that allows resource owners (“owner accounts”) to share one or more VPC subnets with other accounts (“participant accounts”) in their organization. This allows participants to manage their own application resources in the subnets shared with them within the FSx for ONTAP user interface. With this new feature, participant accounts can easily run an even broader spectrum of business-critical production workloads in VPCs that have been shared with them. You can create Multi-AZ FSx for ONTAP file systems from shared VPC participant accounts in all AWS Regions where Amazon FSx for ONTAP is available.

FSx for ONTAP sharing and access control concepts

Before discussing this solution, we emphasize some important concepts for accessing FSx for ONTAP, as well as commands provided by FSx for ONTAP.

Endpoint IP addresses

During the creation of Multi-AZ FSx ONTAP file systems, endpoint IP addresses are provisioned for clients to access the FSx for ONTAP file system. As the endpoint IP addresses help facilitate seamless file system failover across Multi-AZ, they should not conflict with existing CIDR ranges in the VPC route tables being used by FSx ONTAP. For details, see Managing file systems in the FSx for ONTAP User Guide.

There are two types of endpoint IP addresses used by FSx for ONTAP:

1. The IP addresses of the file system’s administration management endpoint
2. The IP addresses of the storage virtual machines’ management and NFS/CIFS/iSCSI endpoints

The file system’s management endpoint is used to administer the file system using the AWS API Gateway, ONTAP CLI, or REST API. You can SSH into this management endpoint to execute ONTAP commands. Storage virtual machine (SVM) endpoints are used for the clients to mount and access their ONTAP volumes through NFS exports and/or SMB shares. The management endpoint uses fsxadmin commands to operate at the file system level and let you manage a specific file system, such as creating, deleting, and modifying file systems. vsadmin commands operate at the SVM level, which is an isolated file server with its own administrative and data access endpoints for administering and accessing data.

When you create a Multi-AZ FSx for ONTAP file system, you may choose Quick or Standard Create from the AWS Management Console. With the Quick Create option, you may select the Unallocated IP address option. With Standard Create, you can specify your own IP address range. For example, with Quick Create, you may select Unallocated IP address range from your VPC to have Amazon FSx use the last 64 IP addresses from the VPC’s primary CIDR range as the endpoint IP address range for the file system. Alternatively, you may choose Floating IP address range outside your VPC to have Amazon FSx use a 198.19.x.0/24 address range that isn’t already used by any other file systems with the same VPC and route tables. In Figure 1, we used Standard Create and selected Floating IP address range outside your VPC, and we were able to SSH to file system management IP at 198.19.255.112 to access the ONTAP CLI.

Figure 1: FSx for ONTAP Management endpoint

With an SVM endpoint IP addresses of 198.19.255.23, Windows and Linux clients can mount FSx volumes through NFS exports and/or SMB shares as in Figure 2.

Figure 2: FSx for ONTAP SVM endpoint

If you need to use different subnet address other than the default range (198.19.0.0/16), then you can choose to customize different endpoint IP address subnet ranges through the “Standard create” option when creating your FSx for ONTAP file system as in Figure 3.

Figure 3: FSx for ONTAP Standard create

VPC Route Tables

As client workloads need to send traffic to SVM endpoints to mount volumes and shares, you need to make sure of network connectivity between the client network and FSx for ONTAP SVM endpoints, especially when the client VPC sits on different AWS accounts.

AWS RAM helps you securely share your resources across AWS accounts, within your Organization or organizational units (OUs). Transit Gateway is designed to connect Amazon Virtual Private Clouds (Amazon VPCs) to AWS accounts. Then, advertise the FSx for ONTAP endpoint subnet and client subnets into the Transit Gateway route table. Finally, adjust your corresponding VPC local route tables to make them routable. Note that your AWS accounts must be under the same Organization to share the resources. For more information on AWS RAM, visit the AWS RAM home page.

Once you verify the connectivity between client workloads and FSx endpoints, your clients’ workloads can mount NFS exports and CIFS shares through the SVM endpoints.

Access Control

In addition to using Security Groups and the Network Access Control List (ACL) to provide security protection, FSx for ONTAP offers additional fine-grained ACLs through the ONTAP CLI and REST API. These security measures allow you to verify the identity of your clients before connecting to the file system mount point and accessing the data.

If your client workload accesses ONTAP volumes by using the SMB protocol to mount shares, then you can configure ACLs on top of CIFS share mapping to the FSx for ONTAP volume. The ACL authorizes the access requests based on a lookup of the user account in the AD domain of which the SVM is a member. For more information on joining your SVM to an AD domain, refer to the guide Joining an SVM to an Active Directory using the AWS Management Console, AWS CLI, and API.

If your client workload accesses ONTAP volumes by using the NFS protocol to mount exports, then you can configure the “volume export policy” with rules defining access control based on client IP addresses and protocol versions. Then modify the FSx for ONTAP volume to apply the corresponding policy. The default export policy allows read and write access to all clients, which means that any client that can reach the SVM can access its data. It is important to review and customize the export policies to meet your specific security and access requirements.

Step 1: Setting up an AWS account and Amazon EC2 instances

For this post, we use two AWS accounts in the same organization: the owner account and the participant account. Both accounts are under the same organization. The owner account is responsible for managing and providing storage service in a centralized manner to achieve cost and operational efficiencies. It hosts a Transit Gateway and FSx for ONTAP file system, which is configured with one SVM containing multiple volumes. The participant account is hosting clients connecting and utilizing FSx for ONTAP resources at the owner account. It consists of a Windows 2019 EC2 instance and an Amazon Linux 2 EC2 instance.

To share these resources, we use AWS RAM to share the Transit Gateway inside the owner account with the participant account. To facilitate file sharing, the Windows VM hosted by the participant account joins the same AD as the SVM in FSx ONTAP. If your Organization uses two or more AD domains and has configured an SMB server for a data SVM, then you can use the security login domain-tunnel to use SVM as a gateway or tunnel for AD access to the cluster.

For more details about setup, refer to Figure 4:

Figure 4: FSx for ONTAP sharing solution architecture diagram

Walk-through of AD, FSx for ONTAP, Transit Gateway, and RAM configuration

This post uses the console and the NetApp ONTAP CLI for configuration. To access the ONTAP CLI, we must SSH into the FSx file system management endpoint. For detailed configurations on FSx for ONTAP, refer to in Table 1.

Table 1: FSx for ONTAP configuration details

Step 2: Setting up Active Directory (AD)

This post uses AWS Managed Microsoft AD to support both the SVM in FSx (hosted by the owner account) as well as the Windows EC2 instance (hosted by the participant account). For more information about setting up AWS Managed Microsoft AD, see Create your AWS Managed Microsoft AD directory in the AWS Directory Service user guide.

In our example, we define the BUILD.LOCAL as our tested domain in AD. We sign in to the Windows domain using our Windows client at the participant account. Inside the “Active Directory Users and Computers” snap-in (Start menu > Administrative Tools > Active Directory Users and Computers), we add two domain users (user1 and user2). They are used for testing connectivity from the Windows EC2 instance to our FSx ONTAP file shares, as shown in the following Figure 5.

Figure 5: AD Users and Computers setting

Step 3 – Setting up FSx for ONTAP

In our example, we set up an FSx for ONTAP file system with 1 TiB SSD storage, as shown in Figure 6. For more information about creating an FSx for ONTAP file system, see Create an Amazon FSx for ONTAP file system in the FSx for ONTAP user guide.

Figure 6: FSx for ONTAP File System settings

After provisioning our Multi-AZ FSx for ONTAP file system, we create an SVM (named “svm22”) to host multiple storage volumes on FSx for ONTAP, as shown in Figure 7. When you access data in your FSx for ONTAP file system, your clients and workstations interface with an SVM using the SVM’s endpoint IP address. For more information about creating an SVM and volumes, see Managing FSx for ONTAP storage virtual machines in the FSx for ONTAP user guide.

For CIFS connectivity, our SVM must join an AD domain. In our case this domain is called BUILD.LOCAL. Note that we do not have full domain administrative access if we use Managed AD, and so we must supply the provisioned OU (i.e., OU=build,DC=build,DC=local). You may retrieve the DNS server IP addresses from the Console if you are using the Managed AD. If not, then contact your Windows administrator.

Figure 7: FSx for ONTAP SVM configuration

Figure 8: FSx for ONTAP SVM setting

When our SVM (svm22) is ready (shown in the preceding Figure 8), we create two volumes: one called usergrp1 with 100 MiB, and one called usergrp2 with 200 MiB for our testing, as shown in Figure 9.

Figure 9: FSx for ONTAP volumes setting

Step 4 – Setting up Transit Gateway and AWS RAM

To recap, the FSx for ONTAP file system hosted by our owner account consists of 10.0.0.0/24 and 198.19.0.0/16 subnets. Our participant account uses the 10.11.0.0/24 subnet. We use AWS RAM and Transit Gateway to make FSx accessible to a participant account. See How resources sharing works session of the AWS RAM user guide for more information.

Here are the high-level procedures in order, also shown in Figure 10.

Figure 10: Transit Gateway and VPC route configuring procedures

1. On the owner account, create the Transit Gateway (fsx-ontap-tgw-share) and Transit Gateway attachment to include the owner account’s subnets 10.0.0.0/24.
2. Go to the Transit Gateway route table and add “static route” 198.19.0.0/16 to cover FSx for ONTAP endpoint subnet 198.19.0.0/16. The next hop should point to the same Transit Gateway attachment as local subnet 10.0.0.0/24.
3. Share the Transit Gateway resource to the participant account using AWS RAM as in Figure 11.

Figure 11: AWS RAM sharing resources on the owner account

4. Go to the participant account to accept resources being shared by Transit Gateway as in Figure 12.

Figure 12: AWS RAM shared resources on participant account

5. In the Transit Gateway console, locate the Transit Gateway shared by the owner account and create Transit Gateway attachments to include the participant account’s subnet 10.11.0.0/24.

6. Move back to the owner Go to the Transit Gateway route table and verify routes covering subnets 10.0.0.0/24, 198.19.0.0/16, and 10.11.0.0/24.

7. Modify the owner account’s VPC route table of FSx for ONTAP by adding route 10.11.0.0/24 to cover the participant account subnets and forward traffics to Transit Gateway as in Figure 13.

8. Modify the participant account’s VPC route table of the client VPC by adding routes covering FSx for ONTAP subnets 10.0.0.0/24 and 198.19.0.0/16.

9. In the participant account’s console, go to the Linux instance and ping the FSx for ONTAP management IP address (in this case, 198.19.255.112) to verify reachability.

Figure 13: Detailed Transit Gateway and VPC route setting

Step 5: Mounting the FSx for ONTAP volume with Linux and Windows instances from the participant account

Once network connectivity between the VPC of FSx for ONTAP at the owner account and the VPC of clients at the participant account is ready, we can mount FSx for ONTAP volumes using Windows and Linux instances hosted by the participant account. Make sure that you have the right network traffic rules in your security group to map your Amazon FSx file system’s file share to a folder on your supported compute instance. Refer to the File System Access Control with Amazon VPC user guide for more information on Security Groups settings.

Mount cross-account FSx for ONTAP volume on Linux

1. SSH to Linux instance on the participant For details, refer to Mounting on Linux clients.
2. Create the mount points and mount the FSx for ONTAP volumes usergrp1 and usergrp2 created. The full endpoint IP address and junction path name can be retrieved by going to the FSx Console, selecting the Volumes tab, selecting the volume, and then selecting the Attach button.

[ec2-user@linuxEC2/]$ sudo mount -t nfs svm-01234567890abdef0.fs-01234567890abcdef1.fsx.us-east-1.amazonaws.com:/usergrp1 /fsx_usergrp1

[ec2-user@linuxEC2/]$ sudo mount -t nfs svm-01234567890abdef0.fs-01234567890abcdef1.fsx.us-east-1.amazonaws.com:/usergrp2 /fsx_usergrp2

CIFS Share creation for Windows client

1. SSH to FSx for ONTAP management console (198.19.255.112) using the fsxadmin account. Use the fsxadmin password you created when you created the file system to sign in.
2. Create the CIFS share using “vserver cifs share create”. Administrator or Power Users can also create/manage shares through Windows MMC.

FSxId0123456789::> vserver cifs share create -vserver svm22 -share-name user2share –path /usergrp2

3. Verify the created share.

Mount CIFS Share on Windows client

Use Remote Desktop to connect to the Windows EC2 instance with AD user account “user1@build.local”. Select “Map network drive” on File Explorer as in Figure 14 to connect to the CIFS Share (user1share at Z: and user2share at Y:) mapping to the corresponding volume paths as in Figure 15.

Figure 14: Example of mounting volume from Windows instance

Figure 15: Example of mounted volumes on Windows instance

Step 6: Configuring access control policies using ONTAP CLI

To provide fine-grained access control at the FSx for ONTAP volume level, we use the Linux instance IP address and the Windows user identity from AD to control access from the Linux and Windows environments respectively.

Linux instance access control

FSx for ONTAP allows us to use vserver (i.e., SVM) policies, known as export policies, to provide access to individual volumes. The steps use the vserver command to create policy, define rules contributing to the policy, and apply the policy to a specific volume. For details about export policy and export policy rules, refer to the export policy create and export policy rule create user guide.

The following are the CLI commands used. In our example, we want to control the access so that only clients with a source IP address of 10.11.0.150 can access volume usergrp1 with policy1, and clients with a source IP address of 10.11.0.151 can access volume usergrp2 with policy2.

1. Create new policies (policy1 and policy2) on SVM svm22.

FSxId0123456789::> vserver export-policy create -vserver svm22 -policy policy1
FSxId0123456789::> vserver export-policy create -vserver svm22 -policy policy2
FSxId0123456789::> vserver export-policy show -vserver svm22
Vserver		Policy Name         
-------------- ------------- 
svm22		default
svm22		fsx-root-volume-policy
svm22		policy1
svm22		policy2

2. Define export-policy rules to authorize clients from specific IP addresses to access the exports as follows:

FSxId0123456789::> vserver export-policy rule create -vserver svm22 policy1 -ruleindex 1 -clientmatch 10.11.0.150 -rorule any -rwrule any
FSxId0123456789::> vserver export-policy rule create -vserver svm22 policy2 -ruleindex 2 -clientmatch 10.11.0.151 -rorule any -rwrule any

3. Modify the volume policy setting to apply the policies on the volume.

FSxId0123456789::> volume modify -vserver svm22 -volume usergrp1 -policy1
Volume modify successful on volume usergrp1 of Vserver svm22

FSxId0123456789::> volume modify -vserver svm22 -volume usergrp2 -policy2
Volume modify successful on volume usergrp2 of Vserver svm22

4. Verify the policy rule details.

FSxId0123456789::> volume modify -vserver svm22 -volume usergrp1 -policy1
		Policy		Rule		Access		Client		RO
Vserver		Name		Index		Protocol	Match		Rule
-------------- -------------	--------------	--------------	------------- 	--------
svm22		policy1		1		any		10.11.0.150	any

FSxId0123456789::> volume show -vserver svm22 -policy policy1
Vserver	  Volume	Aggregate	State	Type	Size	Available	Used%
--------  -----------	--------------	------	------	------ 	----------	--------
svm22	  usergrp1	aggr1		online	RW	100MB	94.71MB		0%

Now we can mount usergrp1 volume from Linux client 10.11.0.150. However, we receive an error when trying to mount usergrp2 volume from the same client.

[ec2-user@linuxEC2/]$ sudo mount -t nfs svm-01234567890abdef0.fs-01234567890abcdef1.fsx.us-east-1.amazonaws.com:/usergrp1 /fsx_usergrp1
[ec2-user@linuxEC2/]$ sudo mount -t nfs svm-01234567890abdef0.fs-01234567890abcdef1.fsx.us-east-1.amazonaws.com:/usergrp2 /fsx_usergrp2
mount.nfs: access denied by server while mounting svm-01234567890abdef0.fs-01234567890abcdef1.fsx.us-east-1.amazonaws.com:/usergrp2

Windows instance access control

Similar to Linux instance access control, we use vserver commands to apply user access control based on the AD username or user group.

In our example, both CIFS shares user1share and user2share allow Everyone with Full Control. We can verify this as follows:

FSxId0123456789::> vserver cifs share show -vserver svm22
Vserver		Share		Path		Properties	Comment		ACL
-------------- -------------	--------------	--------------	------------- 	----------
svm22          c$		/		oplocks		-		BUILTIN\Ad
						browsable			ministrato
						changenotify			rs / Full 
 						show-previous			Control
						-versions
svm22		ipc$		/		browsable	-		-
svm22		usershare1	/usergrp1	oplocks		-		Everyone /
						browsable			Full Contr
						changenotify			ol
						show-previous
						-version
svm22		usershare2	/usergrp2	oplocks		-		Everyone
						browsable			Full Contr
						changenotify			ol
						show-previous
						-version 

Now we modify the ACL used by both CIFS shares. We remove Everyone on both share points. We assign “User1 with Full Control” to user1share as well as “User2 with Full Control” to user2share.

FSxId0123456789::> vserver cifs share access-control create -vserver svm22 -user-or-group “User1” -permission Full_Control -share user1share
FSxId0123456789::> vserver cifs share access-control create -vserver svm22 -user-or-group “User2” -permission Full_Control -share user2share
FSxId0123456789::> vserver cifs share access-control delete -vserver svm22 -user-or-group “Everyone” -share user1share
FSxId0123456789::> vserver cifs share access-control delete -vserver svm22 -user-or-group “Everyone” -share user2share

After performing the preceding instructions, we verify that only User1 has full control on user1share and User2 has full control on user2share.

FSxId0123456789::> vserver cifs share show -vserver svm22
Vserver		Share		Path		Properties	Comment		ACL
-------------- -------------	--------------	--------------	------------- 	----------
svm22          c$		/		oplocks		-		BUILTIN\Ad
						browsable			ministrato
						changenotify			rs / Full 
 						show-previous			Control
						-versions
svm22		ipc$		/		browsable	-		-
svm22		usershare1	/usergrp1	oplocks		-		User1 /
						browsable			Full Contr
						changenotify			ol
						show-previous
						-version
svm22		usershare2	/usergrp2	oplocks		-		User2
						browsable			Full Contr
						changenotify			ol
						show-previous
						-version

When we go back to our Windows VM, clients signing on with User1 can still access user1share. However, clients are unable to access user2share. Figure 16 shows CIFS share rule validation.

Figure 16: CIFS share rule validation

Cleaning up

If you followed the steps in this post for testing and no longer need the environment, then make sure to clean up the resources that were deployed to mitigate charges to your account. As part of this post, you may have created an FSx for ONTAP file system, an AWS Managed Microsoft AD domain, and two EC2 instances. To remove the EC2 instances, you can use the Amazon EC2 console. To delete the FSx for ONTAP file system, you must delete all exports, shares, volumes, and SVMs. To remove the AWS Managed Microsoft AD domain, you can use the Directory Service console. To remove Transit Gateway, you can use the VPC console.

Conclusion

In this post, we walked you through how underlying clients communicate with your FSx file systems using ONTAP SVMs. We demonstrated how to use Transit Gateway and AWS RAM to share resources across different AWS accounts within the same Organization. We also shared using native NetApp CLI commands on FSx for ONTAP that can setup fine-grained access control to Linux and Windows instances while resources are sharing over the network.

The sharing of single FSx for ONTAP filesystem across AWS accounts within AWS Organizations lets you simplify your environment by consolidating multiple FSx file systems while serving multiple AWS accounts at the same time. It helps you achieve better cost optimization and maximized operational efficiency through the sharing mechanism leveraging AWS Transit Gateway and AWS RAM proposed in this post. More importantly, we provide walkthroughs on using export policy and CIFS shares to allow you to implement good security practices with access control on both Linux and Windows client environment.

Thank you for reading this post on how to share FSx for ONTAP resources across multiple AWS accounts. If you have any comments or questions, then do not hesitate to leave them in the comments section.