CLI for Indexing data from MySQL to Elasticsearch

存储架构 2017-11-15

ElasticSearch is fantastic for indexing and filtering data. But hey, you have your data in a MySQL database. How do you copy all data from the database to Elastic? And that too making sure that nothing goes wrong.

Don’t worry, this is indeed possible. We at have made an awesome CLI tool called ABC which will allow you to do this with a single command.

abc import --src_type=mysql --src_uri= 

THAT’s it. Seriously, this is all you need to sync a MySQL database to an ElasticSearch index. Here’s a video showing the process.

The Steps

The first step is to install ABC if you have not done so already. So go to the GitHub releases page for ABC and download the most recent version. It’s a single no-dependancy binary so put it anywhere you like. We recommended putting it inside a PATH directory so that it can be access from anywhere in the terminal.

Ensure that ABC is working by running the following command.

abc version

abc version output

Now, let’s take a MySQL database and we are going to sync it to ElasticSearch index hosted on

So first of all we take a new empty database and then we are going to create a table called ‘users’ with the following schema.

CREATE table users (
 id INT,
 email varchar(500) NOT NULL,
    name varchar(100) NOT NULL,
 bio text,

After that we will add some sample data to it.

insert into users values(1, '', 'Avi', 'Developer');
insert into users values(2, '', 'Jon', 'King in the north');

The table looks like as follows now -

users table

The MySQL test source is now complete. It’s URL looks as follows


Next we are going to create the sink ElasticSearch index. We go to and create a new app called abcmysqltest . The complete URL to this index looks like the following.

So now we have both the source and the sink ready. It’s time for some ABC magic. Using this source and sink, we can build the import command. It will be as follows

abc import --src_type=mysql --src_uri="USER:PASS@tcp(HOST:PORT)/DBNAME" ""

Once you run this command, you should see that the command will finish in some time with no errors. Now if you visit appbaseio dashboard, you can see that the data has been transferred to the target ElasticSearch index.

appbaseio dashboard

Voila. Everything works. The data has been transferred to ElasticSearch and that too without doing anything at all. Next we will see how to transform and manipulate data as it goes from source to the sink.

Transforming Data before Indexing into Elasticsearch

There are times when you don’t need the data to go as it is from source to the sink. You might like to change the target type name (example — users to accounts) or you might like to remove certain fields (e.g. bio ) or create new fields. For all this, we have the transforms feature in ABC. It can be used by the transform_file parameter.

abc import --src_type=mysql --src_uri="USER:PASS@tcp(HOST:PORT)/DBNAME" --transform_file="transform_file.js" ""

The transform_file parameter takes the path to the transform file. That file in turn contains the JavaScript transforms that should be applied to the pipeline. Let’s take the contents of transform_file.js as follows.

t.Source("source", source, "/.*/")
 .Save("sink", sink, "/.*/")

In the above transform, you can see that we are going to omit the bio field from the data transfer. Now when we run the new import command, we should have the following result.

updated app data

As you can see, bio field was omitted when data reached the sink. More documentation on transform file can be found on GitHub. It supports lots of inbuilt functions like omit and even supports running custom JavaScript code as a transform. It’s a good idea to explore its documentation.


This article is part of the article series “How to index data from into ElasticSearch using ABC”. If you found this interesting, consider giving this article a :clap|type_1_2: so that others readers can find it on Medium too. ��

Further Reading

ABC’s README is a good place to start if you want to learn more about the application. You can also have a look at the MYSQL adaptor docs . Furthermore, you may :star:️ the repo on GitHub and watch it to stay tuned for updates.

责编内容by:All things #datastreams (源链)。感谢您的支持!


Testing with a MySQL 5.7 Database On Codeship Every site I've used Codeship on—until today—has either used SQLite in testin...
CentOS7源码安装lamp 环境介绍 虚拟机 : VMware Workstation 14 Pro 镜像 : CentOS Linux release 7.4.1708...
slurm-16.05.3任务调度系统部署与测试(1) 1.概述 本博客通过VMware workstation创建了虚拟机console,然后在console内部创建了8台kvm虚拟机,使用这8台虚拟机作为集群...
How to add TINYINT MySQL type to Doctrine in Symfo... Hello! A few days ago I needed to define an entity with a TINYINT type colum...
MYSQL indexing problem I am having some difficulties finding an answer to this question... For simplici...