Add ingress flexibility to the CANFAR science platform

Kubernetes applications generally do not have an opinion about the ingress on the cluster and typically helm chart deployments allow deployers to set the ingressClassName to be used to route the relevant services (and configure any specific settings needed by the application or its services)

Currently, the CANFAR base deployment is deployed with the traefik ingress controller by default and the helm chart has the corresponding dependency.

This is useful in cases, say, where a user is starting with a blank cluster and wants everything included in the box.

However, for kubernetes clusters in the SRCNodes, where multiple other services and constraints exist, an assumption on ingress shouldn’t be made. 

As we aim to deploy a steady state of production services now, we have to be mindful of the software stack we are adopting in SRCNet and the operational overheads of the same. Reducing and avoiding complexity where possible is essential and multiple ingress controllers add risk and bloat the SRCNet software stack.

AC 1: Helm chart to have ingress class choice, such that nginx (or any other ingress provider) can be used.

Stretch AC2: Helm chart to create ingress objects based on parametrised ingressClassName (eg here).