NIC Integration

Automatic Maintenance of NIC Entries for your own Services.

Hosts contained in Istio Gateways can be automatically registered as CNAMEs in the NIC for our Istio Ingressgateways through the annotation wwu.io/nic_node.

The value for this annotation is the entry for which the CNAME record should be generated.

Each cluster can only create CNAMEs for its own NIC entries, which are currently:

  • Production:
    • All Clusters: istio.k8s.wwu.de
    • Einsteinstraße: istio.ms1.k8s.wwu.de (In most cases, this is the correct one)
    • Schlossplatz: istio.ms2.k8s.wwu.de
  • Staging:
    • All Clusters: istio.staging.k8s.wwu.de
    • Einsteinstraße: istio.ms1.staging.k8s.wwu.de (In most cases, this is the correct one)
    • Schlossplatz: istio.ms2.staging.k8s.wwu.de

Only hosts that have been approved in the project application are allowed in the Gateways. Allowing e.g. domain.uni-muenster.de in your namespace will not allow you to use sub.domain.uni-muenster.de, or any other subdomain for that matter. These will need to be specified during the application or requested afterwards. If you would like to use subdomains, please tell us so, because we need to get the networking department’s permission for them.

Wildcard hosts such as * or *.example.com are skipped silently, since there is no single CNAME the controller could create for them — you will not see an event for those.

We run a service that periodically looks for these annotations on all existing istio gateways and automatically creates DNS entries on the NIC-entries of our Ingress Gateways. It also automatically removes entries that no longer have a corresponding gateway. Deleting an annotated Gateway triggers this reconciliation right away, so its entries are removed without waiting for the periodic run. This is important when you are using the multi-cluster annotation on a gateway that is only present on one cluster, because the clusters will “fight” each other, continuously creating and removing the entry. In short, if you use the multi-cluster annotation, make sure your service is actually multi-cluster.

Pointing the annotation at a different node moves the Gateway’s hosts over: the controller deletes the entries on the old node and creates them on the new one. This costs two requests per host but converges in a single reconciliation run.

This would for example configure example.uni-muenster.de as a CNAME in the NIC for the Istio Ingress gateway on the production cluster in the Einsteinstraße:

apiVersion: networking.istio.io/v1
kind: Gateway
metadata:
  annotations:
    wwu.io/nic_node: istio.ms1.k8s.wwu.de
  name: example
  namespace: example
spec:
  selector:
    istio: ingressgateway
  servers:
    - hosts:
        - example/example.uni-muenster.de
      port:
        name: http
        number: 80
        protocol: HTTP
      tls:
        httpsRedirect: true
    - hosts:
        - example/example.uni-muenster.de
      port:
        name: https
        number: 443
        protocol: HTTPS
      tls:
        credentialName: example--example-certificate
        minProtocolVersion: TLSV1_3
        mode: SIMPLE

Controlling public visibility

Whether the CNAMEs of a Gateway are visible in the public DNS is decided by the annotation wwu.io/nic_public_dns on the same resource. It applies to every host of that Gateway and takes the values strconv.ParseBool understands, so true/false as well as 1/0. A Gateway without the annotation is public, which is the behavior every entry had before the annotation existed.

apiVersion: networking.istio.io/v1
kind: Gateway
metadata:
  annotations:
    wwu.io/nic_node: istio.ms1.k8s.wwu.de
    wwu.io/nic_public_dns: "false"
  name: example
  namespace: example
spec:
  # ...

Changing the annotation on a Gateway that already has its entries registered converges as well: since there is no NIC endpoint to change the visibility of an existing entry in place, this happens as a delete followed by a create of the same name.

Two Gateways claiming the same host on the same node with different visibility values are a conflict the controller cannot resolve on its own: the entry gathered first wins, and the other Gateway gets a ConflictingPublicDNS warning event. A value that cannot be parsed as a boolean produces an InvalidPublicDNSAnnotation warning event, and the entries stay public — refusing them outright would drop a working DNS name over a typo.

Debugging

The controller that synchronizes information between Kubernetes resources and the NIC provides feedback to users about various problems by sending events on the affected resources. You can see these either in your namespace’s events (e.g. kubectl --context kube-staging-ms1 -n my-namespace get events) or directly on the resource (e.g. kubectl --context kube-staging-ms1 -n my-namespace describe gateway my-gateway).

Events

Here are some of the most common events:

  • UnknownNode: You are trying to add a CNAME for a host that is not managed by this cluster. This mostly happens if, for example, you try to use istio.ms1.k8s.wwu.de on the staging cluster or istio.staging.k8s.wwu.de on production.
  • HostToNICEntryFailure: Sometimes your host cannot be converted into a proper NIC entry. One of the main reasons for this is that you are trying to use a subdomain that our robot account in the cluster does not have permission to use. You can request that our robot account be granted permission for the subdomain directly from the network department (nic@uni-muenster.de).
  • NICAnnotationOnVirtualService: Previously, the annotations for the NIC entries were on the VirtualService resources in the cluster, but we decided to switch to the Gateway resources. Although it still works to put the annotations on the VirtualService resources for now, we might remove this in the future, and the recommendation is to put them on the Gateway resources.
  • IgnoreHost: We maintain a manual exclusion list for some domains and hosts. If you think your domain or host should not be on that list, please contact us.
  • FailedToAddCNAME: The NIC rejected the changes we tried to apply. This can have several causes; the event description should provide more details. One error we have already seen is inconsistent configuration across clusters. For example, if you set the annotation to istio.ms1.staging.k8s.wwu.de on kube-staging-ms1 and to istio.staging.k8s.wwu.de on kube-staging-ms2, the controllers on those clusters will try to add your CNAME to different nodes, and one of them will fail to apply the change to the NIC.
  • InvalidPublicDNSAnnotation: The value on wwu.io/nic_public_dns could not be parsed as a boolean. The entries stay public until the value is fixed.
  • ConflictingPublicDNS: Another resource already claimed the same host on the same node with a different value for wwu.io/nic_public_dns. The entry that was gathered first wins; this resource’s entry was not applied.