Parsing#
Each Log-file goes through up to 4 different kinds of parsing steps:
Overview Image
Image showing what is parsed by which part.
🟨 -> Header-Text
🟪 -> Startup-Entries
🟥 -> Meta-Data
🟧 -> Multiline entry
Meta-Data Parsing#
To parse the meda-data, the whole file has to be available. The Meta-Data parser goes through the file in chunks and tries to find each attribute via regex.
It takes a ‘LogFile’-Instance as input and copies all Meta-Data attributes that the ‘LogFile’ already has to itself. This means it only searches for those attributes that are missing on the ‘LogFile’.
Game Map
This has to be the arma internal used name (no spaces, can also differ from the display name like virolahti -> vt7)
2022/11/13, 11:52:47 Mission world: cam_lao_nam
🔗explanation with regex101.com
re.compile(r"\sMission world\:\s*(?P<game_map>.*)")
Version
A semi-semvar-version is needed to be parseable
2022/11/13, 11:53:11 2022-11-13 19:53:11:377 | Antistasi | Info | File: A3A_fnc_initServer | Server version: 2.6.1.872dbb3
🔗explanation with regex101.com
re.compile(r"\s*((MP server version)|(Server version)):\s*(?P<version>.*?)(?=\s|$)")
Mods
It looks for the special delimited block of mods that arma puts in the logs by default and then processes each line as a possible mod.
11:51:09 ============================================================================================= List of mods =============================================================================================== 11:51:09 modsReadOnly = true 11:51:09 safeModsActivated = false 11:51:09 customMods = true 11:51:09 hash = '06E03E1C463E72E69E7B56CB8BD8C7F859180514' 11:51:09 hashShort = 'f1973b83' 11:51:09 name | modDir | default | official | origin | hash | hashShort | fullPath 11:51:09 ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 11:51:09 @ACECompatSOG | @ACECompatSOG | false | false | GAME DIR | 58c416e22842b40b72edf1dde9ddc97cb1f6667a | 75251e92 | C:\TCAFiles\Users\Antistasi\2433\@ACECompatSOG 11:51:09 Gruppe Adler Trenches | @GruppeAdlerTrenches | false | false | GAME DIR | 1469bb9106e667d7257e5117e40516bea48b6001 | a300f1d0 | C:\TCAFiles\Users\Antistasi\2433\@GruppeAdlerTrenches 11:51:09 VET_Unflipping - 1.3.2 | @VETUnflipping | false | false | GAME DIR | b87f16fa85a76c4522fe0fe67521f1ab4a7a7fb0 | dc68cef1 | C:\TCAFiles\Users\Antistasi\2433\@VETUnflipping 11:51:09 @ZeusEnhancedACE3Compatibility | @ZeusEnhancedACE3Compatibility | false | false | GAME DIR | 3ba9d33273e6b3891df3e596d51295136f59ec40 | 5c267667 | C:\TCAFiles\Users\Antistasi\2433\@ZeusEnhancedACE3Compatibility 11:51:09 Zeus Enhanced 1.12.2 | @ZeusEnhanced | false | false | GAME DIR | 6f138882698bd41e062699f5b0bcc1ea79468511 | 4eb0fe50 | C:\TCAFiles\Users\Antistasi\2433\@ZeusEnhanced 11:51:09 TFAR | @TaskForceArrowheadRadioBETA | false | false | GAME DIR | 6e889b7a92591e469d9e037aa3631fb8e5a954fc | d11c0c7f | C:\TCAFiles\Users\Antistasi\2433\@TaskForceArrowheadRadioBETA 11:51:09 Enhanced Movement | @EnhancedMovement | false | false | GAME DIR | fbc1f582c89f11919f9c6ead5d842e6c1aea9b71 | a47fddc8 | C:\TCAFiles\Users\Antistasi\2433\@EnhancedMovement 11:51:09 Community Base Addons v3.15.8 | @CBAA3 | false | false | GAME DIR | 1d7bc59f1f4d1d2a3604e1c33be8fdd686a5617f | b09d9bc5 | C:\TCAFiles\Users\Antistasi\2433\@CBAA3 11:51:09 Advanced Combat Environment 3.15.2 | @ace | false | false | GAME DIR | f5413c6e60e6aa9e5bbeda3f1dcdc3e029727f68 | 56cefafc | C:\TCAFiles\Users\Antistasi\2433\@ace 11:51:09 [Dev] Antistasi - Community 2.6.1 | @Dev1AntistasiDevBuild | false | false | GAME DIR | 1c0cf6a7b0eaf2c81d3a479c327a5c1712e28070 | 19619fb8 | C:\TCAFiles\Users\Antistasi\2433\@Dev1AntistasiDevBuild 11:51:09 Arma 3 Creator DLC: S.O.G. Prairie Fire | vn | false | true | GAME DIR | b086dbcc2986a92aef3b0fd6204c8bc54ee8a8d4 | f66fee82 | C:\TCAFiles\Users\Antistasi\2433\vn 11:51:09 Arma 3 Art of War | aow | true | true | GAME DIR | 6ba7b160d357ca64590614f6a6c4579168efc1d0 | bc4189c2 | C:\TCAFiles\Users\Antistasi\2433\aow 11:51:09 Arma 3 Contact (Platform) | enoch | true | true | GAME DIR | 9f6919562e719e9f70bf666194a7862d3274185a | 87e92527 | C:\TCAFiles\Users\Antistasi\2433\enoch 11:51:09 Arma 3 Tanks | tank | true | true | GAME DIR | b4f3c4faa775f1c81c77f02f97cfa1aa18da5209 | bbdf79e2 | C:\TCAFiles\Users\Antistasi\2433\tank 11:51:09 Arma 3 Tac-Ops | tacops | true | true | GAME DIR | 8fcf9c1f0a1fa5b8d354b64967ffd6583fc43d3e | fb047e5c | C:\TCAFiles\Users\Antistasi\2433\tacops 11:51:09 Arma 3 Laws of War | orange | true | true | GAME DIR | 12abb4e3868e4b3036e9cea47138bdeb4abe518d | f564c92a | C:\TCAFiles\Users\Antistasi\2433\orange 11:51:09 Arma 3 Malden | argo | true | true | GAME DIR | a844aa131096258aab8c000fddcc940284385494 | 49d733fb | C:\TCAFiles\Users\Antistasi\2433\argo 11:51:09 Arma 3 Jets | jets | true | true | GAME DIR | 3f599a4f98ca3a0cd3c5e5e574acc16737c061ef | aae4f413 | C:\TCAFiles\Users\Antistasi\2433\jets 11:51:09 Arma 3 Apex | expansion | true | true | GAME DIR | 26e79f13d9a9e6ae48a6119621f73d53cf411dbd | fe5acf19 | C:\TCAFiles\Users\Antistasi\2433\expansion 11:51:09 Arma 3 Marksmen | mark | true | true | GAME DIR | 6f0fbfd365214eb67dc377503d2583c6718ff19b | 21bc50ea | C:\TCAFiles\Users\Antistasi\2433\mark 11:51:09 Arma 3 Helicopters | heli | true | true | GAME DIR | f61b0de0ac2bd0602413e01b40b80f56b2b32130 | 5e3c056 | C:\TCAFiles\Users\Antistasi\2433\heli 11:51:09 Arma 3 Karts | kart | true | true | GAME DIR | 952859b5e0bac3b093582ba87f54bdafea848df9 | e2ac0c1 | C:\TCAFiles\Users\Antistasi\2433\kart 11:51:09 Arma 3 Zeus | curator | true | true | GAME DIR | 531b60d7d72729c65b0bb6a87f6cd48b6d6a7041 | 3cafe885 | C:\TCAFiles\Users\Antistasi\2433\curator 11:51:09 Arma 3 | A3 | true | true | NOT FOUND (Empty) | | | 11:51:09 @TaskForceEnforcer | @TaskForceEnforcer | false | false | GAME DIR | da39a3ee5e6b4b0d3255bfef95601890afd80709 | 11fdd19c | C:\TCAFiles\Users\Antistasi\2433\@TaskForceEnforcer 11:51:09 @utility | @utility | false | false | GAME DIR | da39a3ee5e6b4b0d3255bfef95601890afd80709 | 11fdd19c | C:\TCAFiles\Users\Antistasi\2433\@utility 11:51:09 ==========================================================================================================================================================================================================
🔗explanation with regex101.com
re.compile(r"""^([0-2\s]?\d) [^\d] ([0-6]\d) [^\d] ([0-6]\d) \s?\={25,}\sList\sof\smods\s\={25,} \n (?P<mod_lines>(^([0-2\s]?\d) [^\d] ([0-6]\d) [^\d] ([0-6]\d) \s(?!\=).*\n) + ) ^([0-2\s]?\d) [^\d] ([0-6]\d) [^\d] ([0-6]\d) \s?\={25,}""", re.VERBOSE | re.MULTILINE)
Campaign ID
Is only allowed to consist of digits. Currently this attribute also sets if the Log-file is from a new Campaign or loaded and continued from a saved Campaign.
2022/07/04, 06:11:28 2022-07-04 13:11:28:248 | Antistasi | Info | File: A3A_fnc_initServer | Loading last campaign ID 93344
🔗explanation with regex101.com
re.compile(r"((?P<text_loading>(Loading last campaign ID)|(Loading campaign with ID))|(?P<text_creating>Creating new campaign with ID))\s*(?P<campaign_id>\d+)")
UTC offset of the log-file
Danger
if this attribute is not found, the log-file is marked as unparsable and will not be parsed any further.
It does this by looking for the first log-entry it can find, that contains the full local datetime AND the full UTC datetime and calculate the offset from those two.
2022/11/13, 11:53:11 2022-11-13 19:53:11:377 | Antistasi | Info | File: A3A_fnc_initServer | Server init started
🔗explanation with regex101.com
re.compile(r"""^ (?P<local_year>\d{4}) / (?P<local_month>[01]\d) / (?P<local_day>[0-3]\d) \,\s+ (?P<local_hour>[0-2]\d) \: (?P<local_minute>[0-6]\d) \: (?P<local_second>[0-6]\d) \s (?P<year>\d{4}) \- (?P<month>[01]\d) \- (?P<day>[0-3]\d) \s (?P<hour>[0-2]\d) \: (?P<minute>[0-6]\d) \: (?P<second>[0-6]\d) \: (?P<microsecond>\d{3}) (?=\s)""", re.VERBOSE | re.MULTILINE)
To not use too much memory when reading large log-files and to not accidentally cut one of the search texts in half with chunking, it always keeps 2 chunks and processes the combined text of both. When reading a new chunk, the older one of the 2 stored gets discarded and the other one gets combined with the new one to create the new text to search.
The Mechanism of how the files are read.#
It will continue reading chunks and searching until one of the following conditions is fulfilled:
☐ end of file reached
☐ all meta-attributes found
☐ it has read more than the set limit (in bytes), as all attributes should be able to be found (if they exist) in the beginning of a log-file.
Header-Text Parsing#
The Header-text, that each arma log-file has, is parsed by collecting lines, until a line that starts with a simple timestamp. That line is not consumed.
Note
The file-object position is not reset to zero after parsing the header-text.
Startup-Entries Parsing#
All lines from the end of the Header-text to the first line that has a full local timestamp are collected and stored unprocessed. This is done because most of the time these messages do not contain data that is usually checked and would bloat the database and slow down parsing.
Note
The file-object position is not reset to zero after parsing the header-text.
Entries Parsing#
Note
This programm assumes that each line is a unique entry and only in special cases, does an entry stretched over more than a single line.
These special cases must either have a concrete syntax or start with special markers.
It is not advised to use multiline entries, not with the Logbook, but also not in General.
The Parser read the log-file line by line (using the FileLineProvider which gives easy access to the previous, current and the next line), if the line starts with a full local timestamp, it stores the line and looks at the next line. If the next line
does not start with a full local timestamp
has a entry continuing-marker after the timestamp
then it adds the next line also to the stored lines, if not it yields the stored lines as an entry and clears the line-storage.
Flowchart
Future Plans
This will probably change in the future, to be more flexible. If I am able to achieve this, the parsing will change to statemachine parser, that is extendable. It is mostly necessary for the default arma error entries, that often need knowledge of lines about 4 lines after the current one to check if they are still part of the entry.
Record Processing#
Warning
Most Code here will be changed in the near future as it currently is almost hardcoded specifically to Antistasi-logs. It will be changed to be more generic and also easier to extend.