GEP-0036: Self-Hosted Shoot Exposure

Table of Contents

• GEP-0036: Self-Hosted Shoot Exposure
  • Table of Contents
  • Summary
  • Motivation
    • Goals
    • Non-Goals
  • Proposal
    • Shoot API Changes
    • SelfHostedShootExposure Extension Resource
      • Extension Controller Interface
      • Examples of Possible Extension Implementations
    • DNS-Based Control Plane Exposure
    • gardenlet Controller for Updating Control Plane Endpoints
  • Alternatives
    • Service of Type LoadBalancer

Summary

This proposal introduces a standardized mechanism for exposing the API server of self-hosted shoot clusters with managed infrastructure (see GEP-28), e.g., using a load balancer of the underlying infrastructure provider or other strategies. It defines a new extension resource, SelfHostedShootExposure, and describes how control plane exposure can be configured, managed, and reconciled, enabling external access to shoot clusters in a flexible and provider-agnostic way.

Motivation

API servers of hosted shoot clusters can be accessed externally via a DNS name following the pattern api.<Shoot.spec.dns.domain> ("external domain"). The DNS record points to the load balancer of an istio ingress gateway of the hosting seed cluster (see GEP-08). For convenience, shoot owners can omit the Shoot.spec.dns.domain field when creating the Shoot object to use a default domain provided by the operator as the cluster's external domain. This mechanism relies on configuration in the garden cluster, which might not be available at the time of bootstrapping the self-hosted shoot cluster. However, the external domain is required for bootstrapping the cluster as of now, eliminating the option of using a default domain.

Apart from the external domain which might be chosen by the shoot owner, Gardener also creates an "internal domain" for hosted shoot clusters, i.e., a DNS record with the same values. The internal domain is configured by the operator in Seed.spec.dns.internal.domain and cannot be influenced by the shoot owner. Self-hosted shoot clusters only have an external domain based on the Shoot manifest, as there is no Seed object for configuring the internal domain. Hence, the internal domain is not relevant for this proposal.

For convenience and consistency, Gardener should support exposing self-hosted shoot clusters externally via a DNS name using the same pattern as the external domain of hosted shoot clusters. However, in self-hosted shoot clusters, the control plane is hosted within the shoot cluster itself, and there is no hosting seed cluster to provide the necessary exposure mechanism, i.e., no istio ingress gateway.

For self-hosted shoot clusters with managed infrastructure, Gardener reuses many existing components (e.g., extensions and machine-controller-manager) for managing the infrastructure, control plane components, and machines. Similarly, it should provide a standardized way to externally expose the API server of self-hosted shoot clusters using existing components. However, there is no mechanism in these existing components to handle the exposure of a Kubernetes control plane in the desired way. Hence, this proposal introduces a new extension resource for this particular purpose.

In case of a self-hosted shoot cluster with unmanaged infrastructure, Gardener expects the operator to manually set up the necessary DNS record pointing to the control plane nodes or an external load balancer. While the focus of this proposal is on self-hosted shoot clusters with managed infrastructure, some of the described mechanisms could also be applied to expose self-hosted shoot clusters with unmanaged infrastructure.

Goals

• Enable external access to the API server of self-hosted shoot clusters with managed infrastructure
• Use the same DNS name pattern for self-hosted shoot clusters as for hosted shoot clusters (external domain)
• Provide a flexible, extension-based mechanism for control plane exposure by defining a new Gardener extension resource
• Support multiple exposure strategies (e.g., cloud load balancer or DNS) to fit different use cases
• Allow extensions to implement provider-specific/custom logic for exposing shoot control planes
• Allow reusing the exposure mechanisms for shoot clusters with unmanaged infrastructure where applicable

Non-Goals

• Add an internal domain for self-hosted shoot clusters
• Add support for using default domains for self-hosted shoot clusters
• Defining the full lifecycle or implementation details of all possible exposure strategies

Proposal

Shoot API Changes

To specify which exposure mechanism should be used for the control plane of a self-hosted shoot cluster, the Shoot API is extended as follows:

apiVersion: core.gardener.cloud/v1beta1
kind: Shoot
spec:
  provider:
    type: local
    workers:
    - name: control-plane
      controlPlane:
        exposure: # either `extension` or `dns` or omitted/empty
          extension:
            type: local # defaults to `.spec.provider.type`, but could also be different
            providerConfig: {} # *runtime.RawExtension
          dns: {}

In the control plane worker pool, a new optional exposure field is added. It can be used to specify that the control plane should be exposed using a SelfHostedShootExposure extension (via the extension field, see The SelfHostedShootExposure Extension Resource) or directly via DNS (via the dns field, see DNS-Based Exposure). If the exposure field is omitted, the control plane is not exposed externally by Gardener via either mechanism.

The extension.type field specifies which SelfHostedShootExposure extension should be used, defaulting to the value of spec.provider.type if not set. Additional configuration for the extension can be provided via the optional extension.providerConfig field.

The exposure field is mutable and supports switching between the two exposure mechanisms or enabling/disabling control plane exposure during the lifetime of the self-hosted shoot cluster. Switching from one exposure mechanism to another results in the creation/deletion of the SelfHostedShootExposure object and updating the corresponding DNSRecord.spec.values[] list accordingly. When disabling control plane exposure, the SelfHostedShootExposure object is deleted (if it exists), and the corresponding DNSRecord.spec.values[] list is updated one last time to contain the addresses of all control plane nodes (as created during initial bootstrapping). From this point onward, the control plane exposure is no longer managed by Gardener and needs to be handled manually by the operator.

SelfHostedShootExposure Extension Resource

If the new exposure.extension field is set, gardenadm init (for initial bootstrapping) or the gardenlet (after connecting the shoot to a garden) creates/updates a SelfHostedShootExposure object in the kube-system namespace (similar to the other self-hosted shoot extension objects). This resource instructs the corresponding extension controller to manage the necessary resources for exposing the control plane of the self-hosted shoot cluster and allows the extension to report the resulting ingress addresses, e.g.:

apiVersion: extensions.gardener.cloud/v1alpha1
kind: SelfHostedShootExposure
metadata:
  name: example
  namespace: kube-system
spec:
  # extensionsv1alpha1.DefaultSpec
  type: stackit
  providerConfig: {} # *runtime.RawExtension

  credentialsRef: # *corev1.ObjectReference
    apiVersion: v1
    kind: Secret
    namespace: kube-system
    name: cloudprovider

  # control plane endpoints that should be exposed
  port: 443
  endpoints:
  - nodeName: example-control-plane
    addresses: # []corev1.NodeAddress
    - address: 172.18.0.2
      type: InternalIP
    - address: example-control-plane
      type: Hostname
  # - ... more endpoints for HA control planes
status:
  # extensionsv1alpha1.DefaultStatus
  observedGeneration: 1
  lastOperation:
    type: Reconcile
    state: Succeeded

  # endpoints of the exposure mechanism
  ingress: # []corev1.LoadBalancerIngress
  - ip: 1.2.3.4
  - hostname: external.load-balancer.example.com

The spec includes the default set of fields included in all extension resources like type and providerConfig (see GEP-01). For shoots with managed infrastructure, the credentialsRef field references the credentials secret that should be used by the extension controller to manage the necessary infrastructure resources (similar to the Infrastructure extension resource). The port field specifies the port that the API server listens on and that should be exposed via the exposure mechanism. Additionally, the spec.endpoints list contains all healthy control plane node addresses that should be exposed. Each endpoint includes the node name and a list of addresses (based on the Node.status.addresses list).

Control plane nodes are considered healthy if their status.conditions list contains a condition of type Ready with status True and does not contain any condition of type {Disk,Memory,PID}Pressure or NetworkUnavailable with a status other than False and the node has healthy etcd and kube-apiserver pods. Also, control plane nodes that are marked for deletion or maintenance operations (e.g., replacement by machine-controller-manager, cluster-autoscaler scale-down, or in-place update) are excluded from the endpoints list. With this filtering in place, there is no need for a more sophisticated health check mechanism in the SelfHostedShootExposure extension implementation itself.

The status includes the default fields included in all extension resources like observedGeneration and lastOperation. Additionally, the status.ingress list contains resulting addresses of the exposure mechanism, e.g., the IPs or hostnames of a load balancer. The status.ingress field has the same type as Service.status.loadBalancer.ingress as it will be used as the source for the values of the corresponding DNSRecord – similar to how the status of the istio ingress gateway service is used for the DNSRecord values of hosted shoot clusters.

As usual, gardenadm/gardenlet will wait for the object to be reconciled successfully and update the (already existing) DNSRecord object's .spec.values[] with the addresses out of the reported .status.ingress[]. IP addresses are preferred over hostnames when updating the DNSRecord.

Extension Controller Interface

The extension controller implementing SelfHostedShootExposure must reconcile resources for exposing the control plane and update .status.ingress with the resulting addresses. A new controller for the SelfHostedShootExposure resource will be added to the extension library, similar to other existing extension controllers. The corresponding Actuator interface implemented by the extension looks as follows:

type Actuator interface {
  // Reconcile creates/reconciles all resources for the exposure of the self-hosted shoot control plane.
  Reconcile(context.Context, *extensionsv1alpha1.SelfHostedShootExposure, *extensionscontroller.Cluster) ([]corev1.LoadBalancerIngress, error)
  // Delete removes all resources that were created for the exposure of the self-hosted shoot control plane.
  Delete(context.Context, *extensionsv1alpha1.SelfHostedShootExposure, *extensionscontroller.Cluster) error
}

When reconciling a SelfHostedShootExposure object, the extension controller returns the resulting list of LoadBalancerIngress addresses that will be stored in .status.ingress (implemented in the controller of the extension library).

Examples of Possible Extension Implementations

A typical provider extension can implement the SelfHostedShootExposure resource by creating a load balancer on the underlying infrastructure and configuring it to forward traffic to the control plane nodes specified in .spec.endpoints. I.e., the extension controller would ensure a load balancer and the correct target pool similar to the Service controller of a cloud-controller-manager.

For infrastructures or scenarios where creating a load balancer is not possible or desired, an alternative implementation of the SelfHostedShootExposure resource can install a software-defined load balancer (e.g., kube-vip or MetalLB) on the control plane nodes themselves (e.g., via a DaemonSet) – possibly in combination with provider-specific infrastructure resources (e.g., external IPs and NICs). E.g., in an OpenStack environment with layer 2 connectivity but without load balancer support, the extension controller could create a floating IP (external IP) and a port (NIC) in the shoot's network, install kube-vip on the control plane nodes, and configure kube-vip to advertise the port's IP as a virtual IP via ARP.

In provider-local, the SelfHostedShootExposure controller can create a Service of type LoadBalancer in underlying kind cluster and configure it to forward traffic to the control plane nodes. The LoadBalancer service in the kind cluster would simulate a cloud provider load balancer by forwarding traffic from the host machine on a specific IP (bound to the loopback device) to the control plane machines hosted as pods in the kind cluster. Potentially, the LoadBalancer service in the kind cluster could be implemented using cloud-provider-kind (see Hackathon 2025-11 results, separate work stream).

DNS-Based Control Plane Exposure

As an alternative to using a SelfHostedShootExposure extension, the control plane of a self-hosted shoot cluster can also be exposed directly via DNS. In this case, no additional API objects or infrastructure resources for exposing the control plane are created, and the control plane nodes' addresses are passed directly to the DNSRecord object's .spec.values[] by gardenadm/gardenlet.

While this approach is simpler and requires no additional extension controller, it has some limitations compared to using a SelfHostedShootExposure extension. Most notably, the DNS record updates (e.g., when control plane machines are rolled out) might be delayed due to DNS caching. Also, there is no load balancing mechanism in front of the control plane nodes, so clients need to handle multiple addresses themselves. Furthermore, if the control plane nodes are not exposed externally (i.e., do not have external IPs or hostnames), the control plane still cannot be accessed from outside the cluster.

gardenlet Controller for Updating Control Plane Endpoints

The gardenlet responsible for the self-hosted shoot cluster (deployed by gardenadm connect) runs a new controller that watches the Node objects of the control plane worker pool.

If the shoot uses a SelfHostedShootExposure extension, the controller updates the SelfHostedShootExposure.spec.endpoints[] list with the .status.addresses[] of all healthy control plane nodes. Once the SelfHostedShootExposure object has been reconciled successfully, the controller updates the corresponding DNSRecord object's .spec.values[] with the addresses reported by the extension in SelfHostedShootExposure.status.ingress[] if necessary.

Not all extensions implementing SelfHostedShootExposure require continuously updated control plane endpoints. E.g., an extension using kube-vip only needs to create the infrastructure resources once and then kube-vip will elect a leader and dynamically advertise the virtual IP from one of the healthy control plane nodes. To omit unnecessary API requests made by the gardenlet controller, we add a field to the ControllerRegistration API that allows extensions to specify if they need a continuously updated SelfHostedShootExposure.spec.endpoints list or not.

If the shoot uses DNS-based control plane exposure, the controller directly updates the DNSRecord object's .spec.values[] with the addresses of all healthy control plane nodes.

Alternatives

Service of Type LoadBalancer

Instead of introducing a new extension resource, gardenadm/gardenlet could manage a Service of type LoadBalancer in the self-hosted shoot cluster that forwards traffic to the control plane nodes. However, this approach has several drawbacks:

• Services of type LoadBalancer are explicitly not designed for exposing control planes and typically exclude control plane nodes from traffic (see kubernetes/kubernetes#65618, kubeadm, KEP-1143)
• Services of type LoadBalancer would include worker nodes in the target pool by default, which is not desired for control plane exposure
• Using externalTrafficPolicy: Local to restrict the target pool to control plane nodes would result in the loss of connectivity in case kube-proxy is not running on the control plane nodes (even if the API server is available)
• Other exposure strategies (e.g., software-defined load balancers or purely DNS-based exposure) cannot be implemented using a Service of type LoadBalancer