Project: Have to know my openwrt access point a little better. I.e. gather logs from the access point and point these logs to a central location and somehow display and search the logs. Possible even an alert, when something bad is happening. On the backburner is adding other devices and adding some fancy displays.
Need to be running 24/7 and therefore hardware must be very power efficient. So any of my old servers HPE/Dell servers was out from the start. Had to be a RPI4 project.
Log server will need some persistent storage for the logs. I started out with a SD card for OS and Docker. Mounted an external mechanical USB disk for data. Should be an OK setup, as logs really can use up some storage space. SD card ... not so sure. I could not get this setup to work properly and switched to a sata-usb cable with a SSD drive instead.
Disclaimer: Not an expert and have possibly made some stupid mistake. This is not a write-up. It is more like a basic starting point for running a Graylog server on RPI4, if you are starting out from scratch with especially Docker/docker-compose.
Hardware: RPI4 (4GB). Sata to USB adapter and a cheap 120GB SSD drive. I think there is a point in avoiding the SD cart.
OS: raspios lite arm64 (64bit) BETA with docker/docker-compose [https://downloads.raspberrypi.org/raspios_lite_arm64/images/]
Graylog Docker image: graylog:4.2-arm64 [https://hub.docker.com/r/graylog/graylog/]
Biggest issue was location of permanent storage. I was originally using a SD card for OS and external UBS disk for storage. For backup, I would like to know where data is located. I am also not a Docker expert, but Docker offer support for host volumes. Which is exactly what I wanted. It goes something like this (extract from first yml file):
- /media/greylog/greylog_data:/usr/share/graylog/data
Syntax: [host path]:[/container path]. I.e. mounted USB to "/media/greylog/greylog_data" (Yes - I know I misspelled 'greylog' several times. It is of course 'Graylog'). You can also use bind volumes in somewhat the same fashion. Elasticsearch and Mongodb containers had no problem with host volumes. Started up just fine. However, the Graylog container did not like it and the result was a lot of access violations. Something like this:
mkdir: cannot create directory ‘/usr/share/graylog/data/journal’: Permission denied
I never got to fix this properly, as I do not understand the problem 100%. Graylog container is running with an '1100' user, and it should be possible to add an '1100' user to your local OS. I.e. create a 1100 user and give this user the correct access rights to a local path. Not sure - was a dead end for me, with my limited Linux skills. Also tried different environment variables for the Graylog container. Suspect Graylog container is not using these - i.e. you cannot change the Graylog container user. Oddly enough, the container user was able to create some directories on the mounted disk when using host volumes.
Alternative: Using named volumes instead of host volumes. With named volumes, Docker is keeping track of the location of these volumes. You just give the volume a name and Docker handles the rest. Not a general Docker discussion on setting up volumes, but simply a work-a-round as the Graylog container apparently have no problem with named volume. It is easy enough to find the location of the named volumes, if you really want to - for backup and stuff. Snip from yml file:
volumes:
- es_data:/usr/share/elasticsearch/data
Simply a random name "es_data" : Path inside container (you will also need to define the volume in the yaml file). You lose some control here: Would be nice to point to a specific path and keep track of disk size/use. For the path inside the container, I have reused the path used in Graylog 'Example # 2'. Do not change. See link below.
So hardware needed changing too: Instead of SD card and mounted external USB disk, I boot directly on an SSD dive (with a sata-usb cable). It is not ideal, as I am not sure about Linux partitions - but you get to skip the SD card. The point is, storage is now just limited by the disk size (or the relevant disk partition) and it is much nicer to work on a SSD drive, than a SD card. Before starting on the docker-compose file, I would recommend looking through: [https://archivedocs.graylog.org/en/latest/pages/installation/docker.html] The below is heavily influenced by example '2'.
So ... the yml file.
| version: | This is the version of the compose file format (just mention this, as I used it for my own version notes and made a mess). |
| container_name: | Simply to reference the running container. Optional. |
| volume: | For each container (mongo, elasticsearch and graylog), each volume must be defined. Like: ... volumes: - mongo_data:/data/db ... ... volumes: mongo_data: driver: local ... |
| user: | Have no effect. |
| ports: | Port for incoming logs. I have configured each source with a different port number. I.e. switch is using port 1515 and pfsense is using 1516 and so on. |
| volumes: | The 3 named volumes. The same three names mention in the service section. |
Graylog environment variables
| GRAYLOG_ROOT_TIMEZONE | Not sure, this is correct. Still have some issues with time zones. |
| GRAYLOG_ROOT_PASSWORD_SHA2 | Have a look here: https://archivedocs.graylog.org/en/latest/pages/installation/docker.html#settings |
# Default version: Named volumes
version: "2"
services:
#MongoDB: https://hub.docker.com/_/mongo/
mongodb:
image: mongo:4.2
container_name: mongodb
volumes:
- mongo_data:/data/db
#Elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/7.10/docker.html
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.2
container_name: elasticsearch
volumes:
- es_data:/usr/share/elasticsearch/data
environment:
- http.host=0.0.0.0
- transport.host=localhost
- network.host=0.0.0.0
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
mem_limit: 1gb
# Graylog
graylog:
# user: $USER
image: graylog/graylog:4.2-arm64
container_name: greylog
volumes:
- greylog_data:/usr/share/graylog/data
environment:
- GRAYLOG_PASSWORD_SECRET=[change password]
# Password admin
- GRAYLOG_ROOT_PASSWORD_SHA2=[change this - have a look at the above link]
- GRAYLOG_HTTP_EXTERNAL_URI=http://127.0.0.1:9000/
# Timezone
- GRAYLOG_ROOT_TIMEZONE=Europe/Copenhagen
entrypoint: /usr/bin/tini -- wait-for-it elasticsearch:9200 -- /docker-entrypoint.sh
links:
- mongodb:mongo
- elasticsearch
restart: always
depends_on:
- mongodb
- elasticsearch
ports:
# Graylog web interface and REST API
- 9000:9000
# Syslog TCP
- 1514:1514
# Syslog UDP
- 1514:1514/udp
- 1515:1515/udp
- 1516:1516/udp
- 1517:1517/udp
# GELF TCP
- 12201:12201
# GELF UDP
- 12201:12201/udp
volumes:
mongo_data:
driver: local
es_data:
driver: local
greylog_data:
driver: local
For testing and finding out Docker, I went with VirtualBox/Ubuntu VM instead of Docker for Windows. Installed docker/docker compose on an Ubuntu VM and used Visual Code from a Windows computer. There is some plug-ins to Visual Code you need to figure out, but it is all fairly simple and Visual Code is a great help with yml. Only problem I ran into, was a newly created VM with at re-used IP. Visual Code was sure; it was some kind of man-in-the-middle attack. Never did figure out how to reuse a VM IP - simply changed the VM ip.
Performance with the Pi: Below is a small snip of a timestamp, uptime, temp, load and total/free memory (Room temperature is around 15-17). Throughput for my setup is minimal. openwrt access point (5-7 devices), Pfsense, pi-hole and a switch.
10:48:16 8 days 27.7 0.18, 0.24, 0.26 3.7Gi 2.6Gi
10:49:16 8 days 26.2 0.18, 0.24, 0.26 3.7Gi 2.6Gi
10:50:16 8 days 27.2 0.41, 0.27, 0.27 3.7Gi 2.6Gi
10:51:16 8 days 26.2 0.41, 0.31, 0.28 3.7Gi 2.6Gi
10:52:16 8 days 26.7 0.15, 0.25, 0.26 3.7Gi 2.6Gi
10:53:16 8 days 26.7 0.16, 0.23, 0.26 3.7Gi 2.6Gi
10:54:16 8 days 26.7 0.38, 0.29, 0.28 3.7Gi 2.6Gi
10:55:16 8 days 28.2 0.32, 0.29, 0.27 3.7Gi 2.6Gi
10:56:16 8 days 27.2 0.55, 0.34, 0.29 3.7Gi 2.6Gi
10:57:16 8 days 26.7 0.29, 0.31, 0.28 3.7Gi 2.6Gi
10:58:16 8 days 27.7 0.25, 0.30, 0.28 3.7Gi 2.6Gi
10:59:17 8 days 26.7 0.24, 0.29, 0.27 3.7Gi 2.6Gi
11:00:17 8 days 26.2 0.13, 0.25, 0.26 3.7Gi 2.6Gi
11:01:17 8 days 27.2 0.22, 0.26, 0.26 3.7Gi 2.6Gi
11:02:17 8 days 27.2 0.22, 0.26, 0.26 3.7Gi 2.6Gi
11:03:17 8 days 27.7 0.08, 0.21, 0.25 3.7Gi 2.6Gi
11:04:17 8 days 26.7 0.23, 0.23, 0.25 3.7Gi 2.6Gi
11:05:17 8 days 27.2 0.24, 0.24, 0.26 3.7Gi 2.6Gi
11:06:17 8 days 26.2 0.67, 0.35, 0.30 3.7Gi 2.6Gi
11:07:17 8 days 26.2 0.59, 0.39, 0.31 3.7Gi 2.6Gi
11:08:17 8 days 27.2 0.31, 0.35, 0.30 3.7Gi 2.6Gi
11:09:17 8 days 26.2 0.31, 0.33, 0.29 3.7Gi 2.6Gi
11:10:17 8 days 28.2 0.43, 0.36, 0.30 3.7Gi 2.6Gi
11:11:17 8 days 27.7 0.16, 0.29, 0.28 3.7Gi 2.6Gi
11:12:17 8 days 26.7 0.18, 0.27, 0.27 3.7Gi 2.6Gi
11:13:17 8 days 27.2 0.41, 0.32, 0.29 3.7Gi 2.6Gi
11:14:17 8 days 26.7 0.48, 0.36, 0.30 3.7Gi 2.6Gi
11:15:18 8 days 27.7 0.41, 0.36, 0.30 3.7Gi 2.6Gi
11:16:18 8 days 26.7 0.19, 0.31, 0.29 3.7Gi 2.6Gi
11:17:18 8 days 27.7 0.64, 0.40, 0.31 3.7Gi 2.6Gi
11:18:18 8 days 27.2 0.98, 0.56, 0.38 3.7Gi 2.6Gi
11:19:18 8 days 27.2 0.48, 0.49, 0.36 3.7Gi 2.6Gi
11:20:18 8 days 27.2 0.49, 0.49, 0.37 3.7Gi 2.6Gi
11:21:18 8 days 26.7 0.47, 0.48, 0.37 3.7Gi 2.6Gi
11:22:18 8 days 27.2 0.21, 0.40, 0.35 3.7Gi 2.6Gi
11:23:18 8 days 27.2 0.38, 0.41, 0.35 3.7Gi 2.6Gi
11:24:18 8 days 26.2 0.14, 0.33, 0.33 3.7Gi 2.6Gi
11:25:18 8 days 28.2 0.46, 0.37, 0.34 3.7Gi 2.6Gi
11:09:17 8 days 26.2 0.31, 0.33, 0.29 3.7Gi 2.6Gi
How I got these numbers:
#!/bin/bash
while true
do
temp=$(vcgencmd measure_temp | egrep -o '[0-9]*\.[0-9]*')
timestamp=$(date "+%T")
uptime=$(uptime | grep -ohe 'up .*' | sed 's/,//g' | awk '{ print $2" "$3 }')
load=$(uptime | grep -ohe 'load average[s:][: ].*' | awk '{ print $3" "$4" "$5 }')
totalmem=$(free -h | grep 'Mem' | awk '{print $2}')
usedmem=$(free -h | grep 'Mem' | awk '{print $3}')
printf "%-10s%10s%10s%20s%10s%10s\n" "$timestamp" "$uptime" "$temp" "$load" "$totalmem" "$usedmem"
printf "%-10s%10s%10s%20s%10s%10s\n" "$timestamp" "$uptime" "$temp" "$load" "$totalmem" "$usedmem" >> temp.txt
sleep 60
done
It really is a lot of fun working with Docker and Graylog. A little amazed I actually got it working in the end.