Using Elastic Curator To Clean Up ELK

I recently setup ELK in order to begin collecting logs from Sysmon for security monitoring in my lab. The problem I could foresee running into was the issue of disk space. Unfortunately when my ELK server runs out of space, it runs out of space. I needed a way to clean up the logs when the server began to reach a threshold. This led me to Elastic Curator
. Curator allows us to manage our indices which includes deleting indices over a given number of days ago. However, this is just a one-off command we can run so I wanted to add some logic to the process.

Disclaimer: This is strictly for one node setups, if you have a large setup with multiple clusters you will want a different solution. (You can use this method still but you might get some weird issues)

Install Curator:

sudo pip install elasticsearch-curator

Create Curator Directory:

sudo mkdir /etc/curator

Create config file:

sudo nano /etc/curator/config.yml

Paste in:

Create delete config action file:

sudo nano /etc/curator/delete-after.yml

Paste in:

Unless you followed this guide
you will most likely have to change some of the details in this action config file. Namely the filters –> value field. You will need to put the name of your index here.

Create Script file:

sudo nano /etc/curator/cleanup.sh

Paste in:

Warning: The above script will check that the disk space is less than 80 percent. It will check the free space of the root or whatever drive is mounted on “/”. If the free space is less than 20 percent, it will begin to delete the oldest indices, once it deletes the oldest one, it will check again and delete the next oldest until the disk space usage is below 80 percent. It will stop deleting regardless of disk space if there is less than 2 days of indices left. Since I couldn’t find an easy way to simple delete the oldest indices, I start at 90 days ago and move forward until it starts to find indices, if you have more than that, please adjust the script (the days variable at line 8). You may also want more than 2 days as a safety net.

Add script to cron:

sudo su
(crontab -l 2>/dev/null; echo "5 0 * * * /etc/curator/cleanup.sh") | crontab -

This cronjob will run the script 5 minutes after midnight forever.

Curator should be added to the HELK
soon. If you aren’t aware of the HELK
and want to get into Threat Hunting (Or just want a super quick way to spin up and ELK stack) you should definitely look into the HELK.
The incredible HELK (Hunting, Elasticsearch, Logstash, Kibana) makes it is easy as running a script to setup a reliable ELK stack tailored for threat hunting.

Check out these labs: https://cyberwardog.blogspot.com/2017/02/setting-up-pentesting-i-mean-threat_98.html
for an in-depth guide on how to set this stuff up manually as well as build the lab around it.

If you run into any issues, feel free to reach out to me on Twitter
or byemail!

The advice and scripts contained and referenced in this point are provided with no warranty. As always, never blindly trust scripts off the internet let alone throw them into a cron job running as root.

责编内容来自:jordanpotti (源链) | 更多关于

阅读提示:酷辣虫无法对本内容的真实性提供任何保证,请自行验证并承担相关的风险与后果!
本站遵循[CC BY-NC-SA 4.0]。如您有版权、意见投诉等问题,请通过eMail联系我们处理。
酷辣虫 » 后端存储 » Using Elastic Curator To Clean Up ELK

喜欢 (0)or分享给?

专业 x 专注 x 聚合 x 分享 CC BY-NC-SA 4.0

使用声明 | 英豪名录