One of the most powerful features of LimaCharlie’s Adapters is the ability to ingest any data type and utilize the Adapter configuration to customize the telemetry to your liking.
One thing I always found useful was the client_options.mapping.parsing_re option, within an Adapter configuration. This lets you write a regular expression for the data, creating custom fields.
Consider the following:
#Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Referer) sc-status sc-substatus sc-win32-status time-taken
2016-09-13 21:45:10 ::1 GET /webapp2 - 80 - ::1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/52.0.2743.116+Safari/537.36 - 500 0 0 5502
We can use the following regex in our configuration to parse this event:
(?P<date>\d{4}-\d{2}-\d{2})\s+(?P<time>\d{2}:\d{2}:\d{2})\s+(?P<s_ip>\d+\.\d+\.\d+\.\d+)\s+(?P<cs_method>\S+)\s+(?P<cs_uri_stem>\S+)\s+(?P<cs_uri_query>\S+)\s+(?P<s_port>\d+)\s+(?P<cs_username>\S+)\s+(?P<c_ip>\d+\.\d+\.\d+\.\d+)\s+(?P<cs_user_agent>.+?)\s+(?P<cs_referer>\S+)\s+(?P<sc_status>\d+)\s+(?P<sc_substatus>\d+)\s+(?P<sc_win32_status>\d+)\s+(?P<time_taken>\d+)
I know, I know, it’s not the cleanest. However, it’s a fast way to get the data we want ingested, with the fields appropriately named.
Favorite regex testing tool: https://regexr.com/