CloudflarD to Nginx on Mac Mini Homelab

Backing up my previous home lab setup

I had an public ip address from my ISP. On my router DHCP static ip address were configured for mac mini, as well as 443 port forwarding to it.

So, any traffic from outside to my ip port 443 were forwarded to macmini

On mac mini side, nginx were installed via brew (no docker stuff here)

From a cloudflare, i had an wildcard subdomain pointing to my ip address, and ssl configured to strict mode (no http anywhere, only https)

Here are config snippets from nginx

nginx.conf

worker_processes 1;

events {
  multi_accept on;
  worker_connections 1024;
}

http {
  charset utf-8;
  client_max_body_size 10M;
  default_type application/octet-stream;
  gzip on;
  gzip_vary on;
  include mime.types;
  keepalive_timeout 65;
  sendfile on;
  server_tokens off;
  tcp_nodelay on;
  tcp_nopush on;

  # for ip in $(curl -s https://www.cloudflare.com/ips-v4/)
  # do
  #   echo "set_real_ip_from $ip;"
  # done
  set_real_ip_from 173.245.48.0/20;
  set_real_ip_from 103.21.244.0/22;
  set_real_ip_from 103.22.200.0/22;
  set_real_ip_from 103.31.4.0/22;
  set_real_ip_from 141.101.64.0/18;
  set_real_ip_from 108.162.192.0/18;
  set_real_ip_from 190.93.240.0/20;
  set_real_ip_from 188.114.96.0/20;
  set_real_ip_from 197.234.240.0/22;
  set_real_ip_from 198.41.128.0/17;
  set_real_ip_from 162.158.0.0/15;
  set_real_ip_from 104.16.0.0/13;
  set_real_ip_from 104.24.0.0/14;
  set_real_ip_from 172.64.0.0/13;
  set_real_ip_from 131.0.72.0/22;
  real_ip_header X-Forwarded-For;
  real_ip_recursive on;

  include servers/*;
}

technically nothing fancy here, usual config may be used, the only thing changed is this block with ip addresses, so nginx logs will contain real ip instead of cloudflare

note: we will allow traffic only from cloudflare, the settings above is technically optional and needed only if you wish to see real ip addresses in logs

also, you may notice this include servers/*; that's where all service definitions leave, here are few examples of how it looks like

3000.conf

server {
  server_name 3000.mac-blog.org.ua;

  include cloudflare.conf;
  include common.conf;

  location / {
    proxy_pass http://192.168.105.109:3000;
    include proxy_headers.conf;
  }
}

calendar.conf

server {
  server_name calendar.mac-blog.org.ua;

  include cloudflare.conf;
  include common.conf;

  location / {
    root html;
    fastcgi_pass 127.0.0.1:9000;
    fastcgi_index index.php;
    fastcgi_param SCRIPT_FILENAME /Users/mini/Desktop/calendar/public$fastcgi_script_name;
    include fastcgi_params;
    fastcgi_read_timeout 5s;
  }
}

as, you may see, everywhere we are including two more files, here they are

cloudflare.conf

listen 8080 ssl;
# https
ssl_certificate /opt/homebrew/etc/nginx/cloudflare.pem;
ssl_certificate_key /opt/homebrew/etc/nginx/cloudflare.key;
# mtls
ssl_verify_client on;
ssl_client_certificate /opt/homebrew/etc/nginx/authenticated_origin_pull_ca.pem;

here are are adding cloudflare trusted https certs for strict tls to work, as well as mtls so only traffic from cloudflare will pass thru.

common.conf

location = /favicon.ico {
  log_not_found off;
}

location = /robots.txt {
  log_not_found off;
}

nothing interesting here, just few tuning options, may be good to add some cache control directives for static content, etc.

thanks to brew, nginx is installed as launchd service out of the box so will be running after system restart.

This approach allows to expose not only local services but also service on local network. Also we may play with nginx configuration options to have some basic auth. And the beauty of setup there is no http, as result no need to have redirect to https rules, and the best one only traffic from cloudflare is allowed.

But at moment i have loose my public ip address and because of that need to find alternative approach, very first attempt was around cloudflared which may completely replace nginx on its own, but because i have few small php services and php-fpm which can not be handled by cloudfalred seems like i still need nginx

The key difference: with cloudflared i do not need port forwarding on my router anymore, as well as ssl stuff in nginx - from tunnel everything will be in plain http which is fine for internal net.

so after following up cloudflare wizzard from here https://one.dash.cloudflare.com/a46509e2e3b4f05df1e7bbb3d89af270/networks/connectors

i was able to install cloudflared as easy as

brew install cloudflared
sudo cloudflared service install eyJhIj......bSJ9

https://developers.cloudflare.com/cloudflare-one/networks/connectors/cloudflare-tunnel/do-more-with-tunnels/local-management/as-a-service/macos/

and after starting it, cleanup nginx from all previous cloudflare additions, aka, now configs are as simple as:

server {
  listen 80;
  server_name prometheus.mac-blog.org.ua;

  # include cloudflare.conf;
  # include common.conf;

  location / {
    proxy_pass http://localhost:9090;
    include proxy_headers.conf;
    auth_basic "auth";
    auth_basic_user_file /opt/homebrew/etc/nginx/prometheus.htpasswd;
  }
}

the only thing did not like - brew services list shows one service, but cloudflared installs system daemon at /Library/LaunchDaemons/com.cloudflare.cloudflared.plist

so instead, i have uninstalled everything and following docs from here https://developers.cloudflare.com/cloudflare-one/networks/connectors/cloudflare-tunnel/downloads/ downloaded binary and used it instead

at cloudflare portal i have added so called application

not how cloudflare warns us that it wont create any dns record because of wildcard, so we need to create one manually like so

for a target use: [tunnel_id].cfargotunnel.com

you can retrieve your tunnel id from here https://one.dash.cloudflare.com/a46509e2e3b4f05df1e7bbb3d89af270/networks/connectors