Skip to content

Latest commit

 

History

History
54 lines (45 loc) · 1.52 KB

README.md

File metadata and controls

54 lines (45 loc) · 1.52 KB
Raw

File system analysis with Elasticsearch and Kibana

Motivation

When moving from a filesystem based to a dms like alfresco it´s important to know how many files were added over time. Additionally it would be good to filter by some special file type (no media files, only office files for example).

This repo contains different scripts and settings for an elk stack used as an dynamic file system analysis tool.

elk filesystem analysis sample

Quick start

  • check out this repo
git clone
cd elk-filesystem-analysis
  • start elk stack with docker-compose
docker-compose up
  • prepare index with bash script
cd index
./prepare_index.sh
  • import kibana settings containing dashboard and visualization settings
  • open kibana and set default index to fsanalyzer with ctime as default time field
  • use python script to index a directory
python3.6 filestructure-elastic-loader.py /path/to/index

Description

The python script walks through the given directory recursively. For each file in the directory the script creates a json object which is sent to elasticsearch.

{
  'name': 'filename01.jpg',
  'fullpath': '/path/to/index/pictures/xmas2016/filename01.jpg',
  'filesize': 3400000
  'suffix': '.jpg',
  'ctime': '2017-04-12T10:36:55'
}

Features

  • trace file size usage (over time)
  • trace file count (over time)
  • filter by filetype
  • filter by directory
  • filter by specific time intervals