Using Caddy as a reverse proxy in 2022

I’ve been wanting to move my load balancer system for WP Support HQ to Caddy web server for a while now. I took the slow holiday time to finally get it done!

The load balancer is important when hosting websites for people, so I can move the server they’re on without having them update their DNS. There are other advantages; SSL termination, Web Application Firewalls, and Rate Limiting.

Why Caddy?

Caddy is fast. It has automatic SSL certificates. It’s easy to configure. At least easier than Nginx.

It is also easy to compile in other modules, I learned.

SSL Termination

This is handed by Caddy almost automatically. Because I wasn’t doing this before, I needed to figure out how to proxy to a secure connection also. Termination is good, because it’s one less certificate to manage and faster to connect to the backend.

Web Application Firewalls

This helps protect my website from 0 days and potentially poorly written custom code. While it only helps, it is still worth having in place. I used the OWASP Coraza middleware for Caddy.

Rate Limiting

This helps protect the servers from being overloaded by crawlers and bots. A lot of crawlers and bots target WordPress sites. However, some clients use Cloudflare, so I needed to exclude those from these limits. I used the HTTP rate limiting module for Caddy 2.

The Install

I setup a new server on DigitalOcean with Ubuntu 22.04. Then installed Caddy using the deb packages. Install Caddy then install XCaddy. We’ll use XCaddy to build in the other modules for the WAF and Rate Limiting.

xcaddy build --with github.com/corazawaf/coraza-caddy --with github.com/mholt/caddy-ratelimit

systemctl stop caddy
cp caddy /usr/bin/caddy
systemctl start caddy

UDP Receive Buffer Size

sysctl -w net.core.rmem_max=2500000

The Configuration

I used the Caddyfile configuration format. Everything from here on is in the /etc/caddy/ directory.

/etc/caddy/
  - Caddyfile
  - trusted_proxies
  - waf/
  - vhosts/

Tip: Run Caddy directly to see configuration errors. caddy run --config /etc/caddy/Caddyfile. Stop the service first.

The Caddyfile

{
        debug

        email dusty@wpsupporthq.com
        order coraza_waf first
        order rate_limit before basicauth

        admin off

        log {
                output file /var/log/caddy/caddy.log {
                        roll_keep_for 30d
                }
        }
}

(waf) {
        coraza_waf {
                include /etc/caddy/waf/coraza/coraza.conf-recommended
                include /etc/caddy/waf/coreruleset/crs-setup.conf.example
                include /etc/caddy/waf/coreruleset/rules/*.conf
        }
}

(trusted) {
        import /etc/caddy/trusted_proxies
}

(secure_headers) {
        header {
                Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
                Content-Security-Policy "upgrade-insecure-requests"
                Referrer-Policy "strict-origin-when-cross-origin"
                ?Cache-Control "public, max-age=15, must-revalidate"
        }
}

# not CloudFlare IPs
(rate_limits) {
        @not_excepted not remote_ip 173.245.48.0/20 103.21.244.0/22 ...
        rate_limit @not_excepted {
                zone site {
                        key {http.request.host}
                        window 5m
                        events 1000
                }
                zone ip {
                        key {http.request.remote.host}
                        window 5m
                        events 250
                }
        }
}

:80 {
        root * /usr/share/caddy
        file_server
        import rate_limits
}

wps-lb-2.wpserverhq.com:443 {
        root * /usr/share/caddy
        file_server
        import rate_limits
}

import /etc/caddy/vhosts/*

This is the main configuration file. There are global options and snippets for use in the vhost files.

trusted_proxies

Is a list of trusted proxies for the reverse_proxy directive. We need this to get the correct IP from these proxies, Cloudflare, in my case.

trusted_proxies 173.245.48.0/20 103.21.244.0/22 ...

The waf directory holds the default rules for the firewall. I followed the setup instructions for Using OWASP Core Ruleset.

There is one newer rule that doesn’t work right now. The solution is to just delete it until it’s supported. GitHub Issue

# Setup
cd waf/
git clone git@github.com:coreruleset/coreruleset.git

rm /etc/caddy/waf/coreruleset/rulesREQUEST-922-MULTIPART-ATTACK.conf

mkdir coraza
cd coraza
wget https://raw.githubusercontent.com/corazawaf/coraza/v3/dev/coraza.conf-recommended

The vhosts directory holds the configuration of the sites.

Tip: When saving a vhosts file, I had to also re-save the Caddyfile otherwise I’d get an odd configuration error. Not sure why?

Error: adapting config using caddyfile: /etc/caddy/vhosts/test.wpsupporthq.com:1: unrecognized directive: test.wpsupporthq.com:443
Did you mean to define a second site? If so, you must use curly braces around each site to separate their configurations.

vhosts/test.wpsupporthq.com

test.wpsupporthq.com:443 {
        encode gzip

        import secure_headers

        # Change to https for secure upstream
        reverse_proxy http://10.134.0.10 {
                # Uncomment to keep a TLS connection to the upstream server
                # transport http {
                #       tls
                #       tls_insecure_skip_verify
                #       compression off
                # }
                header_up Host {http.request.host}
                header_up X-Real-IP {http.request.remote}
                header_up X-Forwarded-Port {http.request.port}

                import trusted
        }

        import waf

        import rate_limits
}

Next Steps

Redirect www to non-www and vis-versa.

Automate the vhost files with Ansible.