Skip to content

Caddy and Cloudflare

Caddy is our reverse proxy software. Better than nginx. We pair this with Cloudflare to get DDOS protection and importantly IP anonymization.

Caddy Install

sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https curl
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/testing/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-testing-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/testing/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-testing.list
sudo apt update
sudo apt install caddy
systemctl daemon-reload
systemctl restart caddy
systemctl status caddy

LetsEncrypt Certs and Cloudflare Problem

Basically LetsEncrypt requires HTTP traffic be allowed over the domain and if you have Cloudflare set up to only allow https it doesn't work. The dumb solution to this is to turn off proxying for the initial cert setup phase and then switch it back on as the cert refresh (every 30 days) works fine. Here's the good way:

to allow caddy to get certs you need to add a config rule under rules->configuration rules to prevent the requests from being redirected to https. to do this:

custom expression -> uri path, starts with, /.well-known/acme-challenge -> automatic https rewrites OFF -> deploy

Cloudflare Config Notes

proxy everything on the domain name used for deadnought activities.
minecraft server srv record example: SRV _minecraft._tcp.SUBDOMAINHERE 0 5 PORTNUMBER SUBDOMAINHERE.proxius.net
SSL/TLS->overview->strict
optimization->protocol->brotli->on
caching->configuration->cahing level->no query string
respect existing headers
caching rules-> if hostname contains proxius.net don't cache. (new at time of writing, will see how it plays out)

Caddy Config Notes

This is how I have Caddy set up for my primary services. There are several items in here which took me a while to figure out

  • Load balancing
    • lb_policy: first, how the endpoints should be prioritized
    • lb_retries: 20, how many times do we try this
    • This rule is used to deal with services getting overloaded / rate limiting the user because I'm bombarding them with weird proxied requests. Most notably fixes the image loading issue on jellyfin I had.
  • header_up
    • header_up X-Forwarded-For {http.request.header.Cf-Connecting-Ip}
      • Passes the origin IP along keyed as X-Forwarded-For. This allows services that use this header to grab a user's actual IP instead of the Cloudflare proxy IP.
  • Path matching
    • Should be clear from the examples whats going on. However this syntax took me quite a while to figure out. I use this to harden my hidden DNS so it appears there's nothing running on it and only valid requests make it through.
  • Compression
    • Use gzip not brotli. 1) you have to xcaddy brotli in and 2) brotli is a bit better, but takes way longer to compute so latency sucks.

Black Arts Cloudflare Redirection

The Old and Slightly More Performant Way

cloudflarecaddyproxiedDNSdirectDNSservice307 redirectclientfiltered reply
cloudflarecaddyproxiedDNSdirectDNSservice307 redirectclientfiltered reply

Cloudflare only allows HTML traffic without violating their TOS, because the bandwidth from media would bankrupt them on this free service. I've figured out a way around this that leverages security through obscurity.

Create page rules in Cloudflare that redirect media traffic back to the server via a non proxied domain name, thus preventing the request from being served over Cloudflare's network. To get a list of endpoints from your app that are doing this, you can analyze the traffic either in Cloudflare or set up a system locally, or just use your browser and check out whats going on.

Procedural Notes:

  • use the network tab in chrome and work your way down the slog of errors you get when trying to perform certain actions on whatever it is you're trying to fix.
  • CORS errors can be fixed in caddy
  • redirect code 307 should be used because it always reuses the last used method. this is what makes proxy bypass file uploading possible.
  • use redirect rules in cloudflare to selectively bypass proxied traffic

On Cloudflare:

  • website->Rules->redirect rules->add
  • Custom filter expression
  • Hostname equals sub.domain.net AND URI contains your_path_here
  • URL redirect Type: Dynamic
  • Expression: concat("https://", "hidden.domain.net", http.request.uri.path)
  • Status code: 307 This is important
  • Preserve query string
  • Place at first

Now if you want to be clean you can lock down your hidden domain to not respond to anything that doesn't match these page rules. You do this in the Caddy config for your hidden domain entries.

TODO A possible enhanced version of this would be to check for a query parameter that functions as a "password". You could slip this into caddy on the proxied IP after a login event or whatever (all logic could be contained in caddy). Then cloudflare just redirects to our hidden domain based off that rule.

The New and Lazy Way

Just do all the redirect math in Caddy.

Infinite Redirect Rule Scaling

Figured this out while needing to be able to scale up an amount of rules for hosting bluemap maps for multiple Minecraft servers.

Create the redirect rule just like the other ones. Here's the magic trick:

concat("https://", "bmap.secretdomain.net", "/", http.host, http.request.uri.path)

We pass our host (http.host) along as a "prefix" in our path! Now, in Caddy we can parse this part of our path and use it as another layer of redirection.

bmap.secretdomain.net {
        header Access-Control-Allow-Origin *

        #our minecraft server subdomain
        handle_path /subdomain.proxius.net/* {
                header {
                        Access-Control-Allow-Origin *
                        Access-Control-Allow-Credentials true
                        Access-Control-Allow-Headers *
                        Access-Control-Allow-Methods *
                        defer
                }
                #bluemap address
                reverse_proxy 192.168.4.5:8100 {

                }
        }
}