亚马逊AWS官方博客

使用 Elasticsearch OSS 构建和运行 Open Distro For Elasticsearch SQL 插件

Open Distro For Elasticsearch 包含四个插件:

  • Security – 支持节点到节点加密、五种类型的身份验证、基于角色的访问控制、审计日志记录,以及跨群集搜索。
  • Alerting – 可在来自一个或多个 Elasticsearch 索引的数据满足特定的条件时通知您。
  • Performance – 这是一个 REST API,可让您查询集群的许多性能指标。
  • SQL 支持 – 借助此功能,您可以使用 SQL 语句来查询集群。

RPM 和 Docker 容器为二进制版本。如果您希望根据我们的代码库进行开发工作,或要基于您自己的 Elasticsearch 二进制文件安装我们的插件,则您必须下载源代码并基于源代码进行构建。我们会发布多篇博客文章来向您介绍如何执行此操作。在本篇文章中,我们首先介绍 SQL 插件。其他插件有各自不同的代码库和编译方法。请随时关注!

下载并安装 Elasticsearch

为了测试使用 Elasticsearch OSS 安装和运行 Open Distro for Elasticsearch 的 SQL 插件,我首先运行了一个 EC2 实例,并运行了 Amazon Linux 2 AMI (ID: ami-095cd038eef3e5074)。您需要安装 JDK 11 的开发版本,才能安装 Elasticsearch 并构建插件。我关闭了 Amazon Corretto。您可以在 Downloads for Amazon Corretto 11 页面上找到最新版本。我还通过编码,确保在安装时为 JAVA_HOME 指定路径:

sudo yum update
wget https://d2jnoze5tfhthg.cloudfront.net/java-11-amazon-corretto-devel-11.0.2.9-2.x86_64.rpm
sudo yum -y install java-11-amazon-corretto-devel-11.0.2.9-2.x86_64.rpm
echo "export JAVA_HOME=/usr/lib/jvm/java-11-amazon-corretto/" | sudo tee --append /etc/profile.d/javahome.sh
source /etc/profile.d/javahome.sh

构建并安装插件

要下载 SQL 插件,您需要安装 Git:

sudo yum install git

安装 Git 后,从 GitHub 下载并构建源代码:

git clone https://github.com/opendistro-for-elasticsearch/sql.git
cd sql
./gradlew build

现在,您可以在 Elasticsearch 中安装该插件并启动 Elasticsearch。(注意:如果您已经启动了 Elasticsearch,请在运行 elasticsearch-plugin 后使用“sudo systemctl restart elasticsearch.service”。)

sudo /usr/share/elasticsearch/bin/elasticsearch-plugin install _file:///home/ec2-user/sql/build/distributions/opendistro_sql-0.7.0.0.zip_
sudo systemctl start elasticsearch.service

测试是否一切正常:

curl 'localhost:9200/_cat/plugins?v&s=component&h=name,component,version,description'

您应当看到类似于以下示例的结果:

name    component      version description
2_EXXx_ opendistro_sql 0.7.0.0 Open Distro for Elasticsearch SQL

测试插件

我们需要加载一些数据才能测试插件。我选择了几部电影。您可以复制以下代码并粘贴到命令行:

curl -XPOST localhost:9200/movies/movie/_bulk -H "Content-type: application/json" -d '{"index" : { } }
{"title": "Star Trek Into Darkness", "directors": ["J.J.Abrams"], "genres": ["Action", "Adventure", "Sci-Fi"], "actors": ["Chris Pine", "Zachary Quinto", "Zoe Saldana"], "id": "tt1408101"}
{"index" : { } }
{"title": "Star Wars", "directors": ["George Lucas"], "genres": ["Action", "Adventure", "Fantasy", "Sci-Fi"], "actors": ["Mark Hamill", "Harrison Ford", "Carrie Fisher"], "id": "tt0076759"}
{"index" : { } }
{"title": "Rush", "directors": ["Ron Howard"], "genres": ["Action", "Biography", "Drama", "Sport"], "actors": ["Daniel Br\u00fchl", "Chris Hemsworth", "Olivia Wilde"], "id": "tt1979320"}
{"index" : { } }
{"title": "Gravity", "directors": ["Alfonso Cuar\u00f3n"], "genres": ["Drama", "Sci-Fi", "Thriller"], "actors": ["Sandra Bullock", "George Clooney", "Ed Harris"], "id": "tt1454468"}
{"index" : { } }
{"title": "The Avengers", "directors": ["Joss Whedon"], "genres": ["Action", "Fantasy"], "actors": ["Robert Downey Jr.", "Chris Evans", "Scarlett Johansson"], "id": "tt0848228"}
{"index" : { } }
{"title": "The Dark Knight Rises", "directors": ["Christopher Nolan"], "genres": ["Action", "Crime", "Thriller"], "actors": ["Christian Bale", "Tom Hardy", "Anne Hathaway"], "id": "tt1345836"}
{"index" : { } }
{"directors": ["Quentin Tarantino"], "genres": ["Adventure", "Drama", "Western"], "title": "Django Unchained", "actors": ["Jamie Foxx", "Christoph Waltz", "Leonardo DiCaprio"], "id": "tt1853728"}
'

您可以使用以 _opendistro/_sql 终端节点为目标的简单查询 API(基于 URL)向 Elasticsearch 发送 SQL 命令:

$ curl -H "Content-type: application/json" -XGET localhost:9200/_opendistro/_sql?sql=SELECT%20title%20FROM%20movies
{"took":9,"timed_out":false,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0},"hits":{"total":7,"max_score":1.0,"hits":[{"_index":"movies","_type":"movie","_id":"wDgvb2kBQ4VucTC3NDK9","_score":1.0,"_source":{"title":"Django Unchained"}},{"_index":"movies","_type":"movie","_id":"ujgvb2kBQ4VucTC3NDK9","_score":1.0,"_source":{"title":"Star Trek Into Darkness"}},{"_index":"movies","_type":"movie","_id":"vDgvb2kBQ4VucTC3NDK9","_score":1.0,"_source":{"title":"Rush"}},{"_index":"movies","_type":"movie","_id":"vjgvb2kBQ4VucTC3NDK9","_score":1.0,"_source":{"title":"The Avengers"}},{"_index":"movies","_type":"movie","_id":"vTgvb2kBQ4VucTC3NDK9","_score":1.0,"_source":{"title":"Gravity"}},{"_index":"movies","_type":"movie","_id":"vzgvb2kBQ4VucTC3NDK9","_score":1.0,"_source":{"title":"The Dark Knight Rises"}},{"_index":"movies","_type":"movie","_id":"uzgvb2kBQ4VucTC3NDK9","_score":1.0,"_source":{"title":"Star Wars"}}]}}

请注意,由于使用了 curl,我必须对 SQL 语句进行 URL 编码。根据所用的连接方式,您可能也需要进行 URL 编码。在正文中使用 POST 命令发送 SQL 请求是一种更为可靠的方法:

$ curl -H "Content-type: application/json" -XPOST localhost:9200/_opendistro/_sql -d'{
    "query" : "SELECT title FROM movies" 
}'
{"took":3,"timed_out":false,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0},"hits":{"total":7,"max_score":1.0,"hits":[{"_index":"movies","_type":"movie","_id":"wDgvb2kBQ4VucTC3NDK9","_score":1.0,"_source":{"title":"Django Unchained"}},{"_index":"movies","_type":"movie","_id":"ujgvb2kBQ4VucTC3NDK9","_score":1.0,"_source":{"title":"Star Trek Into Darkness"}},{"_index":"movies","_type":"movie","_id":"vDgvb2kBQ4VucTC3NDK9","_score":1.0,"_source":{"title":"Rush"}},{"_index":"movies","_type":"movie","_id":"vjgvb2kBQ4VucTC3NDK9","_score":1.0,"_source":{"title":"The Avengers"}},{"_index":"movies","_type":"movie","_id":"vTgvb2kBQ4VucTC3NDK9","_score":1.0,"_source":{"title":"Gravity"}},{"_index":"movies","_type":"movie","_id":"vzgvb2kBQ4VucTC3NDK9","_score":1.0,"_source":{"title":"The Dark Knight Rises"}},{"_index":"movies","_type":"movie","_id":"uzgvb2kBQ4VucTC3NDK9","_score":1.0,"_source":{"title":"Star Wars"}}]}}

测试完毕后,请务必取消您的实例,以免产生不必要的费用。

小结

祝贺您! 您已经完成了安装 Elasticsearch OSS 6.5.4、下载并构建 Open Distro for Elasticsearch SQL 插件、加载数据,以及运行 SQL 查询。在将来的博客文章中,我们将探讨如何构建及修改其他插件。如果您有要补充的内容、发现了需要修复的错误,或是有其他好想法,欢迎您提交请求和发表评论

有问题或疑问? 希望参与讨论? 您可以在我们的论坛上获得帮助并讨论 Open Distro for Elasticsearch。您可以在这里提出问题

本篇作者

Jon Handler

Jon Handler

Jon Handler (@_searchgeek) 是总部位于加利福尼亚州帕罗奥图市的 Amazon Web Services 的首席解决方案架构师。Jon 与 CloudSearch 和 Elasticsearch 团队密切合作,为想要将搜索工作负载迁移到 AWS 云的广大客户提供帮助和指导。在加入 AWS 之前,Jon 作为一名软件开发人员,曾为某个大型电子商务搜索引擎编写代码长达四年。Jon 拥有宾夕法尼亚大学的文学学士学位,以及西北大学计算机科学和人工智能理学硕士和博士学位。