pfSense/Suricata export logs as JSON. Raw suricata logs example:
client http: 01/11/2024-09:31:18.494809 www.fitbit.com[**]/robots.txt[**]Mozilla/5.0 (compatible;PetalBot;+https://webmaster.petalsearch.com/site/petalbot)[**][**]GET[**]HTTP/1.1[**]200[**]399 bytes[**]192.168.7.1:45388 -> 192.168.7.32:80
client http: 01/11/2024-09:29:04.903059 www.fitbit.dk[**]/index.php/lost-found/49-how-to-become-root-in-centos[**]Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)[**]https://www.fitbit.dk/index.php/lost-found[**]GET[**]HTTP/1.1[**]200[**]3518 bytes[**]192.168.7.1:17400 -> 192.168.7.32:80
client http: 01/11/2024-09:27:08.568444 conncheck.opensuse.org[**]/[**][**][**]GET[**]HTTP/1.1[**]204[**]0 bytes[**]192.168.97.192:51229 -> 91.193.113.65:80
client http: 01/11/2024-09:27:08.568423 conncheck.opensuse.org[**]/[**][**][**]GET[**]HTTP/1.1[**]204[**]0 bytes[**]192.168.7.6:58350 -> 91.193.113.65:80
client http: 01/11/2024-09:25:16.953029 dk.archive.ubuntu.com[**]/ubuntu/pool/main/g/glibc/libc-bin_2.35-0ubuntu3.6_amd64.deb[**]Debian APT-HTTP/1.3 (2.4.11) non-interactive[**][**]GET[**]HTTP/1.1[**][**]0 bytes[**]192.168.7.20:56872 -> 130.225.254.116:80
client http: 01/11/2024-09:25:02.408557 dk.archive.ubuntu.com[**]/ubuntu/dists/jammy-security/main/i18n/by-hash/SHA256/e51eac42bc540a3b48bbef1269ba06340ad3434bb64e54b398ba4700bcaaebfd[**]Debian APT-HTTP/1.3 (2.4.11) non-interactive[**][**]GET[**]HTTP/1.1[**][**]0 bytes[**]192.168.7.20:32888 -> 130.225.254.116:80
client http: 01/11/2024-08:35:51.678910 www.notfitbit.dk[**]/index.php?option=com_ajax&format=raw&plugin=RunSchedulerLazy&group=system[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36[**]https://www.notfitbit.dk/index.php/lost-found/28-tightvnc-remote-desktop-connections-using-opensuse-as-client-or-server[**]GET[**]HTTP/1.1[**]200[**]0 bytes[**]192.168.7.1:26650 -> 192.168.7.32:80
client http: 01/12/2024-14:37:44.264891 ctldl.windowsupdate.com[**]/msdownload/update/v3/static/trustedr/en/disallowedcertstl.cab?8cafb0b347f01d4e[**]Microsoft-CryptoAPI/10.0[**][**]GET[**]HTTP/1.1[**]304[**]0 bytes[**]192.168.7.101:58755 -> 212.10.14.163:80
GROK patten:
%{WORD:client}\s%{WORD:protocol}:\s%{MONTHNUM:month}[/-]%{MONTHDAY:day}[/-]%{YEAR:year}-%{TIME:timestamp}\s%{HOSTNAME:host}\[%{DATA:UNWANTED}\]%{URIPATHPARAM:request}\[\*\*\]%{DATA:user_agent}\[\*\*\]%{DATA:referrer}\[%{DATA:UNWANTED}\]%{WORD:http_method}\[%{DATA:UNWANTED}\]%{DATA:http_version}\[%{DATA:UNWANTED}\]%{NUMBER:response_code}\[%{DATA:UNWANTED}%{NUMBER:response_size}?%{DATA:UNWANTED}\[%{DATA:UNWANTED}\]%{IPV4:sourceip}:%{NUMBER:sourceport}(\s)?(->)?(\s)?(%{IPV4:destinationip})?(:)?(%{NUMBER:destinationport})?
Explanation of some key patterns used:
- `%{WORD:client}`: Matches "client" and assigns it to the field "client."
- `%{HOSTNAME:host}`: Matches the hostname (www.fitbit.com) and assigns it to the field "host."
- `%{URIPATHPARAM:request}`: Matches the URI path ("/robots.txt") and assigns it to the field "request."
- `%{DATA:user_agent}`: Matches the user agent string and assigns it to the field "user_agent."
- `%{WORD:http_method}`: Matches the HTTP method ("GET") and assigns it to the field "http_method."
- `%{DATA:http_version}`: Matches the HTTP version ("HTTP/1.1") and assigns it to the field "http_version."
- `%{NUMBER:response_code}`: Matches the HTTP response code ("200") and assigns it to the field "response_code."
- `%{NUMBER:response_size}`: Matches the response size in bytes ("399") and assigns it to the field "response_size."
- `%{IP:client_ip}:%{NUMBER:client_port}`: Matches the client IP and port and assigns them to the fields "client_ip" and "client_port."
- `%{IP:server_ip}:%{NUMBER:server_port}`: Matches the server IP and port and assigns them to the fields "server_ip" and "server_port."