AWS Open Source Blog

Keeping clients of OpenSearch and Elasticsearch compatible with open source

The OpenSearch project is a long-term investment in a secure, high-quality, Apache-2.0 licensed search and analytics suite with a rich roadmap of innovative functionality. OpenSearch aims to provide wire compatibility with open source distributions of Elasticsearch 7.10.2, the software from which it was derived. This makes it easy for developers to migrate their applications to OpenSearch: Existing clients, connectors, and low- and high-level APIs will just work. Our experience at AWS is that developers find it painful to update their already-deployed applications to use new version of server software, so backward compatibility for clients and APIs weighs heavily in our designs.

Many developers who use Elasticsearch and OpenSearch in their applications also make use of the open source client libraries maintained by Elastic, which provide convenient high-level interfaces for several popular programming languages. Over the past few weeks, Elastic added new logic to several of these clients that rejects connections to OpenSearch clusters or to clusters running open source distributions of Elasticsearch 7, even those provided by Elastic themselves. While the client libraries remain open source, they now only let applications connect to Elastic’s commercial offerings.

The most broadly adopted open source projects generally emphasize flexibility, inclusion, and avoidance of lock-in. Developers often rely on this expectation of interoperability when choosing the dependencies they’re going to use, so changes like this are disruptive. And indeed, we’ve heard from developers that these changes are creating challenges for them. This includes anyone who depends on these clients and runs open source versions of Elasticsearch, Open Distro for Elasticsearch, OpenSearch, and customers of Amazon Elasticsearch Service, in addition to developers and open source project maintainers who want to support the broadest range of search backends in their applications and projects.

To give these users a clear path forward, the OpenSearch project will add a set of new open source clients that make it easy to connect applications to any OpenSearch or Elasticsearch cluster. These clients will be derived from the last compatible versions of corresponding Elastic-maintained clients before product checks were added. In the spirit of openness and interoperability, we will make reasonable efforts to maintain compatibility with all Elasticsearch distributions, even those produced by Elastic. These clients will let developers continue running their current version of OpenSearch or Elasticsearch with minimal changes to their application code. The new clients will offer the same APIs and functionality they use today. The list of clients we will create forks of includes:

In the meantime, for anyone using one of the above distributions of OpenSearch or Elasticsearch, we do not recommend updating to the latest version of any Elastic-maintained clients, as this may cause applications to break. Anyone who has already updated their clients and is experiencing issues can resolve these issues by using any of the client versions outlined in the OpenSearch documentation.

Great open source software is built together, with a diverse community of contributors. Open source development gives stakeholders the opportunity to shape a project’s direction by contributing to it directly, and by serving as leaders. As part of this initiative, we are seeking community comaintainers for each of these new client libraries. Maintainers will have a leadership position for the library and will play a key role in setting its direction and priorities. If you are interested in participating at any level in the development and roadmap of these client libraries, visit the OpenSearch website for details.

Within the next few weeks, we anticipate opening repositories for each new client library, in the OpenSearch GitHub organization. We will then work quickly to publish the clients to all the corresponding package managers and repositories. Together, we will ensure that developers have easy access to well-supported client libraries for open source Elasticsearch, Open Distro for Elasticsearch, and OpenSearch.

TAGS:
Eli Fisher

Eli Fisher

Eli Fisher is the Senior Product Manager for OpenSearch at AWS. Eli was with the AWS team that launched Open Distro for Elasticsearch and has a long history of working with database and analytics software. When not working, he enjoys hiking and running through the many parks in the Pittsburgh area. You can find him on Twitter @EliRFisher.

Carl Meadows

Carl Meadows

Carl Meadows is Director of Product Management at AWS and is responsible for Amazon Elasticsearch Service, OpenSearch, Open Distro for Elasticsearch, and Amazon CloudSearch. Carl has been with Amazon Elasticsearch Service since before it was launched in 2015. He has a long history of working in the enterprise software and cloud services spaces. When not working, Carl enjoys making and recording music.

Jules Graybill

Jules Graybill

Jules Graybill is the Director of Search Services at AWS where he leads the development of OpenSearch and Open Distro for Elasticsearch. With an Amazon career spanning three decades, he has been involved in a variety of projects from technology to human resources. He lives in Seattle and enjoys tinkering with electronics and retro operating systems in his spare time. You can find him on Twitter at @JulesGraybill.

Kyle Davis

Kyle Davis

Kyle is the Senior Developer Advocate on the Valkey project. He has a long history with open source software development; he was a founding contributor to the OpenSearch project and most recently worked to build a community around Bottlerocket OS . When not working, Kyle enjoys 3D printing and getting his hands dirty in his Edmonton, Alberta-based home garden.

Mehul Shah

Mehul Shah

Mehul A. Shah is the GM for Search Services at AWS, including Amazon Elasticsearch Service, OpenSearch, Open Distro for Elasticsearch, and Amazon CloudSearch. His passion is leveraging the cloud to build smarter, more efficient, and easier to use data systems. He has three girls, and, therefore, he has no spare time.