Extraction of information from log files using Python programming and Tableau

Abstract

Application servers generate daily log files with a significant part of their activity. This information is recorded sequentially over time but mixes various types of information. The absence of a standard for formatting the data record and the respective volume, make it difficult to extract the corresponding information. The lack of work, specifically in the treatment of SOA server log files, did not allow the comparisson with pre-existing Key Performance Indicators (KPI) or a set of best practices that could be followed. This work results in a description of the process that can serve as a guide for: definition of a logging structure; construction of a data extraction process; definition of a data structure to support the extracted information; definition of control metrics; definition of analysis and control processes for the extracted data.. Given the size of the files and the diversity of types of information that existed, it was necessary to use Python programming for data extraction and pre-treatment, Excel for data pre-treatment, Tableau for statistical treatment and presentation of results.