Kubernetes NodePort Load Balancing with nginx

Mostly this is done in a cloud environment where they have Kubernetes integrated with cloud load balancers and you expose kubernetes services as type LoadBalancer.

However I wanted to do this without cloud in my Virtualbox environment. Its not ideal and I wish nginx could add a port when using proxy_pass pointing to upstream.

My configuration is not ideal and does not scale well. I am using it in a POC and it is working so far so documenting for future reference.

NOTE I did not test if upstream is failing over but that is well documented for nginx so I trust it is working. You could of course change upstream mechanisms to round-robin, least-connected or ip-hash.

user www-data;
worker_processes 4;
worker_rlimit_nofile 40000;

events {
    worker_connections 8192;
}

http {
   map $host $serverport {
     "hello.cluster01.local"   "30000";
     "web01.cluster01.local"   "30001";
     "web02.cluster01.local"   "30002";
     default      "no_match";
   }

   upstream hello.cluster01.local-30000 {
      server 172.20.100.10:30000; 
      server 172.20.100.11:30000; 
   }

   upstream web01.cluster01.local-30001 {
      server 172.20.100.10:30001;
      server 172.20.100.11:30001;
   }

   upstream web02.cluster01.local-30002 {
      server 172.20.100.10:30002;
      server 172.20.100.11:30002;
   }

  server {
    listen 80;
    server_name "~(.*).cluster01.local";
    set $upstream $host-$serverport; 
    location / {
      proxy_set_header X-Forwarded-For $remote_addr;
      # if not load balancing pointing to one node like below is fine
      #proxy_pass http://172.20.100.10:$np;
      # with upstream you can't add a port so I have an upstream per service
      #proxy_pass http://backend:$np;
      proxy_pass http://$upstream;
      proxy_set_header Host $host;
    }
  }
}