DevOps Tips: Automate Your Homelab CI/CD

Don't learn with production.. learn in the lab!

Intro

My DevOps series will cover self-hosting services that will help you learn some DevOps principles. My first post was on why you should version control your homelab, available here. I won’t be teaching introductory level topics because there is already a ton of great content out there. So I won’t cover the basics of docker, linux, networking, etc.

In this post I’m covering using Gitea, Gitea Actions, and a few best practice concepts for automating CI/CD in your lab. This will help you think through solutions in your own lab and whatever you may do for work. This lightweight CI/CD can be used to deploy setup configs, security indicators, network configs, and more! It can be integrated with ansible as well, which I may cover in a later post.

Why Gitea?

There is are a few big players in the code repository space and they are all capable of what I cover. I chose Gitea mainly because it is lightweight and can be self-hosted. In my homelab, Gitea actually syncs with a private Git repository. This is done because I want a backup of my homelab setup outside my homelab. I am treating that as an offsite backup, which your production environment should have a similar design. Back to Gitea, you can deploy it on basically any hardware. I am running it on a Raspberry Pi 4B that boots from a 1TB SATA SSD. Here are some more details on Gitea.

Key Benefits

• Lightweight & fast: Gitea is built in Go and the project has a goal of being lean and uses few resources

• Self-hosted: You can maintain full control of the code and infrastructure

• Built-In CI/CD: Gitea Actions allows automating builds, tests, and deployments and just requires an additional docker container

• Documentation: Gitea has decent documentation and a plethora of writeups on what to do

Gitea Flow

My instructions require you have docker and docker compose available on a host in your homelab. One of the most common ways to deploy Gitea is using Docker Compose. The services are then all accessed using Nginx Proxy Manager. I can do a writeup later on that. If you want to set this up without Nginx then you will need to use a port mapping like - 3000:3000 in your compose files. Here is my code and action flow.

Gitea

I have it use services_default network that all my services run on so they can communicate without exposing ports. I also have it mapping all of the data to /data/gitea so I can easily make backups to protect against hardware failure.

version: '3.8'
services:
  gitea:
    image: gitea/gitea:latest
    container_name: gitea
    volumes:
      - /data/gitea:/data
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    networks:
      - services_default
    restart: unless-stopped
networks:
  services_default:
    external: true

Gitea Runner

This uses the same network as earlier. There are two additional notes with the Runner though. You need to pass environment variables to it so it can connect to your Gitea instance and you will want to map volumes of services you want your Gitea Actions to be able to update easily. This allows me to pull code changes to the mapped volumes for MkDocs and Homepage and then add in necessary API keys for these services after the updates are retrieved.

version: '3.8'
services:
  runner:
    image: gitea/act_runner:nightly
    environment:
      GITEA_INSTANCE_URL: '${INSTANCE_URL}'
      GITEA_RUNNER_REGISTRATION_TOKEN: '${REGISTRATION_TOKEN}'
      GITEA_RUNNER_NAME: '${RUNNER_NAME}'
      GITEA_RUNNER_LABELS: '${RUNNER_LABELS}'
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /data/mkdocs:/docs
      - /data/homepage:/homepage
    networks:
      - services_default
networks:
  services_default:
    external: true

CI/CD Flow

The Gitea Actions are defined in a foldier .gitea/workflows that is also part of the code repository. You will need to build an action and point it to a flow for this to work. This flow is a basic workflow that will pull updates to mapped volumes and then replace API keys and passwords. It uses stored secrets to access the code repository. Then it runs a shell script that does a string replace with the keys and passwords and their respective environment variable that has been setup. Here is the action workflow.

name: Deploy Docs

on:
  push:
    branches:
      - main

jobs:
  deploy:
    runs-on: self-hosted
    steps:
      - name: Install git
        run: |
          apk update
          apk add --no-cache git
      - name: Clone Repository
        run: |
          if [ ! -d "/tmp/labgeek" ]; then
            git clone https://${{ secrets.REPO_USER }}:${{ secrets.REPO_TOKEN }}@gitea.labgeek.io/ed/labgeek.git /tmp/labgeek
          fi
          cd /tmp/labgeek
          git fetch origin
          git reset --hard origin/main
      - name: Copy docs to docs
        run: |
          cp -r /tmp/labgeek/* /docs/
      - name: Copy homepage to homepage
        run: |
          cp -r /tmp/labgeek/docs/include/homepage/* /homepage/
      - name: Replace API keys
        run: |
          # Navigate to homepage directory
          cd /homepage

          # Load environment and replace variables
          chmod +x ./replace_api_keys.sh
          
          # Replace keys
          ./replace_api_keys.sh

Next Steps

I did not cover every component but some of the core ones. I hope this inspires you to explore how you could test out CI/CD in your homelab. If you want to take this to the next level, here are some other components you can integrate.

1. Use a secrets manager service, not environment variables

2. Improve automation with ansible

3. Integrate ssh for services not on the same host (instead of mapping volumes)

There are also some things to consider when automating these components. Managing ssh keys, file permissions, and network permissions were some additional components I had to work through for this to work.

Final Thoughts

This setup will lay the groundwork for understanding a CI/CD pipeline, and you won’t risk blowing up production at work! This entire flow can be setup with a couple Raspberry Pi’s to help simulate network segmentation, ssh keys, and remote file permission issues. The most critical skill you can gain is the desire to learn and troubleshoot. Everyone’s environment will be a bit different and will require research on how to fix it.

Thank you for reading and feel free to reach out for questions or help troubleshooting! I would love to hear from you all.