Istio OIDC Authentication

A service mesh is an architectural pattern that provides common network services as a feature of the infrastructure. This typically includes features such as service discovery and policy enforcement to control how services within the mesh can communicate with each other.

Istio is a service mesh implementation which works by running an instance of Envoy alongside each instance of your services to intercept and proxy service traffic. Additionally, fleets of standalone Envoys are deployed to handle traffic entering and leaving the mesh. Istio’s main purpose then is to configure and expose the functionality of Envoy.

In addition to the core features, Istio also supports powerful extension points, as well as the ability to apply custom configuration to the Envoy sidecars. Here we will describe how Istio can be configured to manage the OpenID Connect (OIDC) authentication flow for applications running within the mesh to allow both authentication and authorisation decisions to be offloaded to Istio. There are a number of ways to achieve this with Istio however here we look at two solutions and how their integration points have been affected by changes to Istio’s architecture.

OIDC

OIDC is an identity layer built upon the OAuth 2.0 protocol that allows the identity of a user to be verified based on authentication to an identity provider. When a user visits an HTTP service that implements OIDC, they would typically be redirected to an identity provider (for example Google, Azure or dex) where they would log in and then be redirected again back to the original service. This sequence of redirects would result in a JWT being attached to the request and presented to the service. This JWT is signed by the identity provider with fields (or claims) that contain information about the user who signed in (for example their email address). These claims can then be used to make authentication and authorisation decisions.

OIDC is a common way of delegating the responsibility of managing user credentials to another service and a powerful feature of Istio is that it can be leveraged to manage this flow without the service needing to be aware OIDC is even being used.

For more information on OIDC and associated terminology, Okta has a great primer.

Istio 1.4

Istio 1.4 and earlier included a component called Mixer which formed part of the Istio control plane. When policy checks were enabled, before Envoy made an upstream connection it would make a logical request to Mixer in order to determine whether the connection was allowed and what action to take. Mixer therefore provided an extension point for Istio, allowing integrations with external components that could make these policy decisions on its behalf.

The App Identity and Access Adapter is an example of an external component that interfaces with Mixer in exactly this way; by analysing attributes sent by Envoy to Mixer when making policy decisions it can work out whether a user has already authenticated to a configured identity provider and if not trigger the OIDC flow. The resulting JWT can then be compared against policy confguration to either allow or deny access to the upstream service.

Istio 1.5 and Above

Since Istio 1.5, Mixer has been deprecated in favour of implementing these extensions within Envoy itself. In particular this reduces the latency of policy decisions that would otherwise require a network call to Mixer.

Here we describe in detail an alternative way to configure Istio to manage the OIDC authentication flow and authorisation decisions but without Mixer. For the purpose of this description we will assume we are running on a 1.16 GKE cluster with Istio 1.7.1 installed, but this setup should be compatible with recent versions of both Kubernetes and Istio:

gcloud container clusters create istio \
  --cluster-version 1.16 \
  --machine-type n1-standard-2 \
  --enable-autoscaling \
  --max-nodes=5
# If not on OS X see the releases for other operating systems:
# https://github.com/istio/istio/releases/tag/1.7.1
wget https://github.com/istio/istio/releases/download/1.7.1/istio-1.7.1-osx.tar.gz
tar xzf istio-1.7.1-osx.tar.gz
./istio-1.7.1/bin/istioctl manifest install -y
# https://istio.io/latest/docs/tasks/security/authentication/mtls-migration/#lock-down-mutual-tls-for-the-entire-mesh
kubectl apply -f - <<EOF
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system
spec:
  mtls:
    mode: STRICT
EOF

We will deploy Nginx to the cluster to act as the test application we want to configure authenticaiton and authorisation for. You will need to configure a domain (I will use nginx.lukeaddison.co.uk) to expose Nginx on. It should be pointing at the following IP address:

kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

We also deploy cert-manager to provision a Let’s Encrypt TLS certificate for us.

WARNING: once OIDC authentication is enforced on the Istio ingress gateway, cert-manager will no longer be able to renew the certificate using the HTTP-01 challenge solver configured below. One solution would be to use a DNS-01 challenge solver instead

# Set your Nginx domain
NGINX_DOMAIN=""
kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.0.1/cert-manager.yaml
# Wait for cert-manager to be ready
kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-issuer-account-key
    solvers:
    - http01:
        ingress:
          class: istio
---
apiVersion: v1
kind: Namespace
metadata:
  name: nginx
  labels:
    istio-injection: enabled
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: nginx
spec:
  selector:
    app: nginx
  ports:
  - name: http
    port: 80
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: nginx
  namespace: nginx
spec:
  hosts:
  - $NGINX_DOMAIN
  gateways:
  - istio-system/nginx
  http:
  - route:
    - destination:
        port:
          number: 80
        host: nginx
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: nginx
  namespace: istio-system
spec:
  secretName: nginx-tls
  issuerRef:
    name: letsencrypt
    kind: ClusterIssuer
  dnsNames:
  - $NGINX_DOMAIN
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: nginx
  namespace: istio-system
spec:
  selector:
    app: istio-ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - nginx/${NGINX_DOMAIN}
    tls:
      httpsRedirect: true
  - port:
      number: 443
      name: https
      protocol: HTTPS
    hosts:
    - nginx/${NGINX_DOMAIN}
    tls:
      credentialName: nginx-tls
      mode: SIMPLE
EOF

Nginx should now be exposed to the internet which you can verify by curling your domain.

curl "https://${NGINX_DOMAIN}"

Authentication

We can now configure OIDC authentication. The first thing we need to do is to configure our OIDC provider; we will be using Google here but any compatible provider should work.

For Google specifically, we need to create a Google OAuth application. The Kubeflow documentation walks through the steps to achieve this for IAP. The functional differences for us is that we want to specify Authorized domains to be whatever domain Nginx is exposed on (so nginx.lukeaddison.co.uk for me) and in Authorized redirect URIs specify the /oauth2/callback path of the Nginx domain (so https://nginx.lukeaddison.co.uk/oauth2/callback for me).

For other providers the configuration (especially the requirement to whitelist redirect URIs) should be similar. Whichever provider you use the setup process should return a client ID and secret which you should remember for a later step.

We can now enforce that access to the Nginx service be authenticated using our OIDC provider. Using the discovery URL (supported by most providers) we can retrieve the information required to configure Istio.

# https://developers.google.com/identity/protocols/oauth2/openid-connect#discovery
OIDC_DISCOVERY_URL="https://accounts.google.com/.well-known/openid-configuration"
OIDC_DISCOVERY_URL_RESPONSE=$(curl $OIDC_DISCOVERY_URL)
OIDC_ISSUER_URL=$(echo $OIDC_DISCOVERY_URL_RESPONSE | jq -r .issuer)
OIDC_JWKS_URI=$(echo $OIDC_DISCOVERY_URL_RESPONSE | jq -r .jwks_uri)
kubectl apply -f - <<EOF
apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name: istio-ingressgateway
  namespace: istio-system
spec:
  selector:
    matchLabels:
      app: istio-ingressgateway
  jwtRules:
  - issuer: $OIDC_ISSUER_URL
    jwksUri: $OIDC_JWKS_URI
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: istio-ingressgateway
  namespace: istio-system
spec:
  selector:
    matchLabels:
      app: istio-ingressgateway
  rules:
  - from:
    - source:
        requestPrincipals: ["*"]
EOF

The RequestAuthentication resource says that if a request to the ingress gateway contains a bearer token in the Authorization header then it must be a valid JWT signed by the specified OIDC provider. Istio will then concatenate the iss and sub fields of the JWT with a / separator which will form the principal of the request. The AuthorizationPolicy enforces that a request principal is set, otherwise a request with no Authorization header at all would be allowed through.

Visting your domain again should now show: RBAC: access denied, so the final thing we need to do is to configure Istio to manage the OIDC flow to retrieve a JWT and inject it into the request path.

oauth2-proxy

Envoy provides filters which are a way of extending the functionality of Envoy. Istio provides an EnvoyFilter API as a way of customising Envoy filters. One such filter that Envoy supports out of the box is the external authorization filter which can call out to an external service to decide whether an incoming HTTP request is authorised or not. We can use oauth2-proxy for this purpose; notice that oauth2-proxy is not actually acting as a proxy in this case:

# Set your client ID and secret from your OIDC provider setup
CLIENT_ID=""
CLIENT_SECRET=""
COOKIE_SECRET=$(openssl rand -hex 16)
kubectl apply -f - <<EOF
apiVersion: v1
kind: Namespace
metadata:
  name: oauth2-proxy
  labels:
    istio-injection: enabled
---
apiVersion: v1
kind: Secret
metadata:
  name: oauth2-proxy
  namespace: oauth2-proxy
stringData:
  OAUTH2_PROXY_CLIENT_ID: $CLIENT_ID
  OAUTH2_PROXY_CLIENT_SECRET: $CLIENT_SECRET
  OAUTH2_PROXY_COOKIE_SECRET: $COOKIE_SECRET
---
apiVersion: v1
kind: Service
metadata:
  name: oauth2-proxy
  namespace: oauth2-proxy
spec:
  selector:
    app: oauth2-proxy
  ports:
  - name: http
    port: 4180
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: oauth2-proxy
  namespace: oauth2-proxy
spec:
  selector:
    matchLabels:
      app: oauth2-proxy
  template:
    metadata:
      labels:
        app: oauth2-proxy
    spec:
      containers:
      - name: oauth2-proxy
        image: quay.io/oauth2-proxy/oauth2-proxy:v6.1.1
        args:
        - --provider=oidc
        - --cookie-secure=true
        - --cookie-samesite=lax
        - --cookie-refresh=1h
        - --cookie-expire=4h
        - --cookie-name=_oauth2_proxy_istio_ingressgateway
        - --set-authorization-header=true
        - --email-domain=*
        - --http-address=0.0.0.0:4180
        - --upstream=static://200
        - --skip-provider-button=true
        - --whitelist-domain=${NGINX_DOMAIN}
        - --oidc-issuer-url=${OIDC_ISSUER_URL}
        env:
        - name: OAUTH2_PROXY_CLIENT_ID
          valueFrom:
            secretKeyRef:
              name: oauth2-proxy
              key: OAUTH2_PROXY_CLIENT_ID
        - name: OAUTH2_PROXY_CLIENT_SECRET
          valueFrom:
            secretKeyRef:
              name: oauth2-proxy
              key: OAUTH2_PROXY_CLIENT_SECRET
        - name: OAUTH2_PROXY_COOKIE_SECRET
          valueFrom:
            secretKeyRef:
              name: oauth2-proxy
              key: OAUTH2_PROXY_COOKIE_SECRET
        resources:
          requests:
            cpu: 10m
            memory: 100Mi
        ports:
        - containerPort: 4180
          protocol: TCP
        readinessProbe:
          periodSeconds: 3
          httpGet:
            path: /ping
            port: 4180
EOF

Note that the above configuration tells oauth2-proxy to store session state as a browser cookie. This has the advantage of being stateless but can lead to large HTTP headers if the JWT returned from the OIDC flow is large; an alternative is to use Redis which adds a further operational burden but is more secure and the size of the cookie is small and constant.

An EnvoyFilter can then be used to inject configuration for the external authorisation filter into the ingress gateway proxies to lookaside to oauth2-proxy for authorisation decisions:

kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: istio-ingressgateway
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      app: istio-ingressgateway
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: GATEWAY
      listener:
        filterChain:
          filter:
            name: envoy.http_connection_manager
            subFilter:
              name: envoy.filters.http.jwt_authn
    patch:
      operation: INSERT_BEFORE
      value:
        name: envoy.filters.http.ext_authz
        typed_config:
          "@type": type.googleapis.com/envoy.config.filter.http.ext_authz.v2.ExtAuthz
          http_service:
            server_uri:
              uri: http://oauth2-proxy.oauth2-proxy.svc.cluster.local:4180
              cluster: outbound|4180||oauth2-proxy.oauth2-proxy.svc.cluster.local
              timeout: 10s
            authorizationRequest:
              allowedHeaders:
                patterns:
                - exact: cookie
            authorizationResponse:
              allowedUpstreamHeaders:
                patterns:
                - exact: authorization
EOF

Visiting Nginx again you should be redirected to your OIDC provider. After signing in successfully oauth2-proxy should set an encrypted cookie which on subsequent requests will be decrypted to a JWT and attached as the Authorization header which Istio can validate.

Note that the envoy.filters.http.jwt_authn sub filter we are matching on is only present when a RequestAuthentication resource is selecting the ingress gateway (as configured above).

If you want to authenticate multiple services you may want to configure the --cookie-domain and --whitelist-domain flags on the oauth2-proxy Deployment to include multiple subdomains. For example, I could set both of those flags to .lukeaddison.co.uk and then the cookie would be sent for any subdomain of lukeaddison.co.uk meaning I would only need to sign in once. To see the full set of configuration options go here.

Another powerful use case is to combine the OIDC authentication configuration with Istio’s ability to proxy to external services. For example, if I have a service running outside of Kubernetes but that does not have its own identity-aware authentication mechanism, I can use Istio as a reverse proxy and configure access to that service in a similar way to if it was running within the mesh.

Authorization

With the above configuration in place, we can now make further authorisation decisions based on the attached JWT and corresponding claims. For example, to ensure that the user signed into our configured OAuth application (instead of another Google one) we can restrict access based on the audience claim. This is straightforward since the audience claim must contain our client ID:

kubectl apply -f - <<EOF
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: istio-ingressgateway
  namespace: istio-system
spec:
  selector:
    matchLabels:
      app: istio-ingressgateway
  rules:
  - when:
    - key: request.auth.audiences
      values:
      - $CLIENT_ID
EOF

Of course, since oauth2-proxy is also making a decision about whether to allow a request through depending on the existence of a JWT stored in an encrypted cookie, it shouldn’t be possible for a user to submit a JWT from a different source in the Authorization header. Nevertheless, representing the expected value natively in Istio provides defence in depth.

We can also configure the ingress gateway Envoys to forward the Authorization header to our upstream service to allow for service specific policy (the following assumes that the email claim is present on the JWT):

EMAIL_ADDRESS="luke.addison@jetstack.io"
kubectl apply -f - <<EOF
apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name: istio-ingressgateway
  namespace: istio-system
spec:
  selector:
    matchLabels:
      app: istio-ingressgateway
  jwtRules:
  - issuer: $OIDC_ISSUER_URL
    jwksUri: $OIDC_JWKS_URI
    # Forward JWT to Nginx sidecar
    forwardOriginalToken: true
---
apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name: nginx
  namespace: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  jwtRules:
  - issuer: $OIDC_ISSUER_URL
    jwksUri: $OIDC_JWKS_URI
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: nginx
  namespace: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  rules:
  - when:
    - key: request.auth.audiences
      values:
      - $CLIENT_ID
    - key: request.auth.claims[email]
      values:
      - $EMAIL_ADDRESS
EOF

Restricting access to an email address which differs from yours should give RBAC: access denied as before. Unfortunately, currently only string and string list claims are extracted from the JWT, so in particular the boolean email_verified claim (signifying whether the provider took steps to ensure the email address was controlled by the end user) cannot be matched on. However, as described in the Claim Stability and Uniqueness section of the OIDC specification, the iss and sub claims used together are the only claims that can provide a stable identifier for a user, so if that is the goal then they should be used instead of the email claim. For more details on what is supported by AuthorizationPolicy see the documentation here and here.

The Dex example-app is a useful tool for retrieving a JWT locally to see the claims your provider returns for a particular set of scopes. Note the scopes requested by oauth2-proxy by default. By configuring oauth2-proxy to request different scopes, you can adjust the claims that are present on the returned JWT and thus the attributes that can be matched on for authorisation decisions. Returning group membership for example allows access to particular services to be granted and revoked by simply moving users within your provider, without any changes to the Istio configuration.

As we have demonstrated, a really powerful aspect of this is that our backend service can be completely unaware that OIDC is being used and does not need to support it itself. However, if the service has support for parsing the JWT, then it can also be used to authorise granular access to different features of the service.

Finally, it is worth mentioning that the EnvoyFilter used to lookaside to oauth2-proxy does not need to select the ingress gateway, but could select any application sidecar (using context: SIDECAR_INBOUND instead). This would allow you to expose both OIDC authenticated and unauthenticated services on the same ingress gateway. Alternatively, a separate ingress gateway could be deployed for unauthenticated services.

Future

Another nascent project in this area is authservice which provides an alternative implementation of an external authorization endpoint, specifically for OIDC authentication. One of the features currently being considered on the roadmap is supporting the configuration of authservice using AuthorizationPolicy which would give a more declarative solution.

Going with the theme of WASM extensibility in Istio, managing the OIDC flow may be a good candidate as a WASM extension so that Envoy no longer needs to call out to an external authorization implementation.

Get in Touch

If you want to know more about how Istio’s features can be leveraged provide powerful benefits to your services, Jetstack offers training around Istio from the ground up to get deeper insights into how the various pieces come together.

Tags// , ,