Support for external identity providers #1397

adutra · 2025-04-18T11:15:43Z

Fixes #336 and #976.

Overview

This change is globally backwards-compatible, there are no configuration breaking changes and the default server behavior is the same as today (internal authentication only).

The main interfaces are also unchanged – but some implementations changed.

TODOs

Things to do in follow-up PRs:

Integration tests (these will be tricky to implement, therefore I prefer to tackle that later)
Helm chart changes
Documentation

Authentication Types

To support external identity providers, the notion of "authentication type" has been introduced. The following values are supported:

internal: Polaris internal authentication only (default)
external: Polaris external authentication only (using Quarkus OIDC)
- When this type is enabled globally or for a realm, the realm's internal token endpoint is disabled and returns 501.
mixed: Polaris internal and external authentication (using Quarkus OIDC)
- internal is tried first, then external

The following authentication type is also present, but I would be in favor of removing it:

test: The old TestInlineBearerTokenAuthenticator, currently unused in Polaris

The default authentication type is internal, but can be either changed globally or overridden per realm.

The authentication type is configured in the application.properties file as below:

# Default authentication type
polaris.authentication.type=internal

# Authentication type overrides per realm
polaris.authentication.realm1.type=external
polaris.authentication.realm2.type=mixed

See below for details of each authentication type.

Configuration Changes

The polaris.authentication configuration section remains backwards-compatible, but everything now can be overridden per realm.

The following options are effective only when using internal auth (can be overridden in per realm):

polaris.authentication.token-service.*
polaris.authentication.token-broker.*

The polaris.oidc configuration section is new and is used to further configure the OIDC tenants. See below for examples.

Authenticator Changes

The Authenticator interface remains the same, but BasePolarisAuthenticator has been modified and became DefaultAuthenticator.

This is because Quarkus Security distinguishes two phases of authentication:

Extract the credentials (i.e., decode the token)
Authenticate the credentials

Therefore, the Authenticator is not responsible anymore for decoding the token, but only for authenticating it.

A new PrincipalCredential interface has been introduced to hold the principal ID, name and roles, and pass it to the authenticator. The existing DecodedToken interface, which is used for internal authentication, is now a sub-type of PrincipalCredential.

The default logic for authenticating the principal, i.e. by a metastore lookup by ID or by name, remains the same.

Token Broker Changes

Token brokers now need to be injected directly, instead of injecting their factories. They are now request-scoped beans.

As a bonus, token brokers can now be configured per realm, i.e., one realm could use RSA key pairs, and another one shared secrets.

Authentication Workflows in Detail

Internal

InternalAuthenticationMechanism handles the authentication header
- Calls TokenBroker to decode the token and creates a PrincipalCredential
InternalIdentityProvider creates a first SecurityIdentity with the PrincipalCredential
AuthenticatingAugmentor authenticates the PrincipalCredential
- Calls Authenticator.authenticate()
- Sets AuthenticatedPolarisPrincipal as the SecurityIdentity's principal
ActiveRolesAugmentor sets the active roles in the SecurityIdentity
- Calls ActiveRolesProvider

External

[Quarkus OIDC] OidcAuthenticationMechanism handles the authentication header
[Quarkus OIDC] OidcIdentityProvider creates a first SecurityIdentity with the token
OidcTenantResolvingAugmentor resolves the Polaris OIDC tenant configuration to use and store it as an attribute in the SecurityIdentity
PrincipalCredentialAugmentor maps the JWT claims to a PrincipalCredential and adds it to the SecurityIdentity
AuthenticatingAugmentor authenticates the PolarisCredential
- Calls Authenticator.authenticate()
- Sets AuthenticatedPolarisPrincipal as the SecurityIdentity's principal
ActiveRolesAugmentor sets the active roles in the SecurityIdentity
- Calls ActiveRolesProvider

Mixed

InternalAuthenticationMechanism handles the authentication header
- Calls TokenBroker to decode the token
If the TokenBroker validates the token:
- InternalIdentityProvider creates a first SecurityIdentity with the token as a PolarisCredential
Otherwise:
- InternalAuthenticationMechanism yields to the OIDC mechanism
- [Quarkus OIDC] OidcAuthenticationMechanism handles the authentication header
- [Quarkus OIDC] OidcIdentityProvider creates a first SecurityIdentity with the token
- OidcTenantResolvingAugmentor resolves the Polaris OIDC tenant configuration to use and store it as an attribute in the SecurityIdentity
- PrincipalCredentialAugmentor maps the JWT claims to a PrincipalCredential and adds it to the SecurityIdentity
For both cases: AuthenticatingAugmentor authenticates the PolarisCredential
- Calls Authenticator.authenticate()
- Sets AuthenticatedPolarisPrincipal as the SecurityIdentity's principal
ActiveRolesAugmentor sets the active roles in the SecurityIdentity
- Calls ActiveRolesProvider

OIDC Principal Mapping

Main interface: org.apache.polaris.service.quarkus.auth.external.mapping.PrincipalMapper.

Currently, the following polaris.oidc options are available:

polaris.oidc.principal-mapper.type: the Principal mapper implementation to use
polaris.oidc.principal-mapper.id-claim-path: the claim path to the principal id in the JWT token, e.g. polaris/principal_id
polaris.oidc.principal-mapper.name-claim-path: the claim path to the principal name in the JWT token, e.g. polaris/principal_name

They can be overridden per OIDC tenant (this is different from the realm!). The OIDC tenants will be looked up in the quarkus.oidc configuration section, e.g.:

quarkus.oidc.oidc-tenant1.auth-server-url=http://localhost:8080/realms/polaris
quarkus.oidc.oidc-tenant1.client-id=client1
quarkus.oidc.oidc-tenant1.application-type=service

polaris.oidc.oidc-tenant1.principal-mapper.id-claim-path=polaris/principal_id
polaris.oidc.oidc-tenant1.principal-mapper.name-claim-path=polaris/principal_name

If a tenant is not found in the quarkus.oidc or in the polaris.oidc section, the default OIDC tenant will be used.

Important: the default OIDC tenant is disabled by default, it must be enabled if you intend to use external auth.

OIDC Principal Roles Mapping

Main interface: org.apache.polaris.service.quarkus.auth.external.mapping.PrincipalRolesMapper.

Roles will be mapped from the JWT by Quarkus OIDC. This is done by setting the quarkus.oidc.roles.role-claim-path property:

quarkus.oidc.roles.role-claim-path=polaris/roles

See https://quarkus.io/guides/security-oidc-bearer-token-authentication#token-claims-and-security-identity-roles for more information.

The roles mapped by Quarkus will be available in SecurityIdentity.getRoles(). On the Polaris side, the PrincipalRolesMapper interface converts those roles to Polaris roles; the following properties are available:

polaris.oidc.principal-roles-mapper.type: the Principal Roles mapper implementation to use.
polaris.oidc.principal-roles-mapper.filter: optional regex to filter out roles.
polaris.oidc.principal-roles-mapper.mappings[0].regex: optional regex to modify role names.
polaris.oidc.principal-roles-mapper.mappings[0].replacement: replacement string.

These properties can be overridden per realm.

Mapping Examples

Example: for a JWT token like this (some claims omitted for brevity):

{
  "sub": "a663c299-1118-4b35-ac85-f9a4f38d0699",
  "typ": "Bearer",
  "azp": "client1",
  "scope": "profile email",
  "polaris": {
    "roles": [ "PRINCIPAL_ROLE:ALL" ],
    "principal_name": "root",
    "principal_id": 1
  },
  "preferred_username": "service-account-client1",
  "clientAddress": "172.18.0.1",
  "client_id": "client1"
}

You can configure Polaris like this:

quarkus.oidc.roles.role-claim-path=polaris/roles
polaris.oidc.principal-mapper.id-claim-path=polaris/principal_id
polaris.oidc.principal-mapper.name-claim-path=polaris/principal_name

Another example; if the JWT token is like this:

{
  "sub": "1"
  "typ": "Bearer",
  "azp": "client1",
  "scope": "service_admin catalog_admin profile email",
  "preferred_username": "root",
  "clientAddress": "172.18.0.1",
  "client_id": "client1"
}

You can configure Polaris like this:

quarkus.oidc.roles.role-claim-path=scope
polaris.oidc.principal-mapper.id-claim-path=sub
polaris.oidc.principal-mapper.name-claim-path=preferred_username
polaris.oidc.principal-roles-mapper.filter=^(?!profile$|email$).*
polaris.oidc.principal-roles-mapper.mappings[0].regex=^.*$
polaris.oidc.principal-roles-mapper.mappings[0].replacement=PRINCIPAL_ROLE:$0

The resulting Polaris principal roles will be: PRINCIPAL_ROLE:service_admin and PRINCIPAL_ROLE:catalog_admin.

adutra · 2025-04-18T12:44:36Z

For those willing to test out integration with Keycloak:

Start a Keycloak server on port 8080 e.g.: http://localhost:8080/realms/master
With keycloak admin UI, configure a client to return the following claims (e.g. using hard-coded claim mappers):
a. polaris/principal_id
b. polaris/principal_name
c. polaris/roles
Run Polaris with:

./gradlew :polaris-quarkus-service:quarkusDev \
  -Dpolaris.realm-context.realms=realm1,realm2,realm3 \
  -Dpolaris.bootstrap.credentials="realm1,root,secret;realm2,root,secret;realm3,root,secret" \
  -Dpolaris.authentication.realm2.type=external \
  -Dpolaris.authentication.realm3.type=mixed \
  -Dquarkus.oidc.tenant-enabled=true \
  -Dquarkus.oidc.auth-server-url=http://localhost:8080/realms/master \
  -Dquarkus.oidc.roles.role-claim-path=polaris/roles \
  -Dpolaris.oidc.principal-mapper.id-claim-path=polaris/principal_id \
  -Dpolaris.oidc.principal-mapper.name-claim-path=polaris/principal_name

3 realms will be bootstrapped:
a. realm1 internal auth
b. realm2 external auth
c; realm3 mixed auth
Obtain a token from Keycloak:

token=$(curl -v http://localhost:8080/realms/master/protocol/openid-connect/token \
  -d client_id=client1 \
  -d client_secret=s3cr3t \
  -d grant_type=client_credentials | jq -r .access_token)

Obtain a token from Polaris (realm1 or realm3 – realm2 will return 501):

token=$(curl -v http://localhost:8181/api/catalog/v1/oauth/tokens \
  -H "Polaris-Realm: realm1" \
  -d client_id=root \
  -d client_secret=secret \
  -d grant_type=client_credentials \
  -d scope=PRINCIPAL_ROLE:ALL | jq -r .access_token)

Using the token:

curl -v http://localhost:8181/api/catalog/v1/config\?warehouse\=default \
  -H "Polaris-Realm: realm1"  \
  -H "Authorization: Bearer $token"

adutra · 2025-04-18T13:08:34Z

polaris-core/src/main/java/org/apache/polaris/core/auth/AuthenticatedPolarisPrincipal.java

+   *
+   * <p>By convention, this method returns an empty set when the principal is requesting all
+   * available principal roles.
+   */


I'm trying to document things as I go. This is the best knowledge I can get for this principal attribute.

adutra · 2025-04-18T13:10:07Z

quarkus/defaults/src/main/resources/application.properties

+quarkus.oidc.application-type=service
+quarkus.oidc.resolve-tenants-with-issuer=true
+# Default tenant (disabled by default, set this to true if you use external authentication)
+quarkus.oidc.tenant-enabled=false


We could enable this conditionally with a ConfigSourceInterceptor that would check if any realm is using external or mixed auth, in which case it makes sense to enable the default OIDC tenant, especially if no other tenant is configured. But I'm leaving this as a follow-up improvement.

adutra · 2025-04-18T13:12:44Z

...s/service/src/main/java/org/apache/polaris/service/quarkus/auth/PolarisIdentityProvider.java

-
-/** A custom {@link IdentityProvider} that handles Polaris token authentication requests. */
-@ApplicationScoped
-public class PolarisIdentityProvider implements IdentityProvider<TokenAuthenticationRequest> {


FYI this was just moved to org.apache.polaris.service.quarkus.auth.internal.InternalIdentityProvider but git lost track of it.

adutra · 2025-04-18T13:22:02Z

...e/src/test/java/org/apache/polaris/service/quarkus/it/QuarkusApplicationIntegrationTest.java

@@ -61,10 +58,6 @@ public Map<String, String> getConfigOverrides() {
    }
  }

-  @Inject
-  @Identifier("rsa-key-pair")
-  TokenBrokerFactory tokenBrokerFactory;


adutra · 2025-04-18T13:22:34Z

...vice/src/test/java/org/apache/polaris/service/quarkus/ratelimiter/RateLimiterFilterTest.java

@@ -77,7 +77,11 @@ public Map<String, String> getConfigOverrides() {
          "polaris.metrics.tags.environment",
          "prod",
          "polaris.realm-context.type",
-          "test");
+          "test",


This test would fail without this change because each realm would have a different pair of RSA keys. So switching to shared secret.

adutra · 2025-04-18T13:24:15Z

service/common/src/main/java/org/apache/polaris/service/auth/Authenticator.java

+  /**
+   * Authenticates the given credentials and returns an optional principal.
+   *
+   * <p>If the credentials are not valid or if the authentication fails, implementations may choose


Again trying to document things a posteriori.

I don't really see the value of returning an Optional here, as the semantics of returning "empty" conflict with that of throwing an error. But I didn't want to change the method signature without knowing what others think.

adutra · 2025-04-19T22:48:30Z

polaris-core/src/main/java/org/apache/polaris/core/auth/AuthenticatedPolarisPrincipal.java

+   * available principal roles. When this is true, {@link #getActivatedPrincipalRoleNames()} returns
+   * an empty set.
+   */
+  public boolean allRoles() {


I think we need this method to distinguish the case when the principal is requesting all roles from the case where no roles were found in the token – since both cases translate into an empty set of role names.

But a better solution imho would be to remove roles from this class completely, and let SecurityIdentity.getRoles() expose the roles.

I checked the call sites for getActivatedPrincipalRoles() and it's in fact never used. So it seems that setActivatedPrincipalRoles() could be removed as well.

I removed the method because it doesn't work well with the Resolver. TLDR we can't distinguish "no roles" from "all roles" for now.

adutra · 2025-04-19T22:52:34Z

service/common/src/main/java/org/apache/polaris/service/auth/DefaultAuthenticator.java

      activatedPrincipalRoles.addAll(
-          Arrays.stream(tokenInfo.getScope().split(" "))
+          credentials.getPrincipalRoles().stream()
              .map(
                  s -> // strip the principal_role prefix, if present
                  s.startsWith(PRINCIPAL_ROLE_PREFIX)


I believe this is dangerous and should be changed. For example, if the roles were initially:
["PRINCIPAL_ROLE:catalog-admin', "service-admin"], this would be transformed into: ["catalog-admin', "service-admin"], potentially requesting the service-admin role, but that likely wasn't intentional.

I think it would be safer to discard all roles that don't have the PRINCIPAL_ROLE: prefix.

Agree, this should be stream().filter(s -> s.startsWith(PRINCIPAL_ROLE_PREFIX)).map(...)

adutra requested review from ashvina, dennishuo, dimas-b, eric-maynard, jackye1995, jbonofre, vvcephei, collado-mike, snazy, RussellSpitzer and takidau as code owners April 18, 2025 11:15

github-project-automation bot added this to Basic Kanban Board Apr 18, 2025

adutra requested review from MonkeyCanCode, flyrain and ebyhr as code owners April 18, 2025 11:15

github-project-automation bot moved this to PRs In Progress in Basic Kanban Board Apr 18, 2025

adutra force-pushed the oidc-multi-auth branch from d6e5d21 to 02d37b1 Compare April 18, 2025 12:19

adutra commented Apr 18, 2025

View reviewed changes

adutra force-pushed the oidc-multi-auth branch 6 times, most recently from 0fff0f7 to abb7483 Compare April 19, 2025 17:22

adutra force-pushed the oidc-multi-auth branch 5 times, most recently from dd3d992 to 42befb1 Compare April 19, 2025 22:45

adutra commented Apr 19, 2025

View reviewed changes

adutra force-pushed the oidc-multi-auth branch 3 times, most recently from 7d192a0 to 6b76c39 Compare April 21, 2025 12:02

adutra mentioned this pull request Apr 21, 2025

Add token refresh and token exchange OAuth2 flows to polaris-synchronizer tool. apache/polaris-tools#8

Open

Support for external identity providers

2ef6cb8

adutra force-pushed the oidc-multi-auth branch from 6b76c39 to 2ef6cb8 Compare April 22, 2025 17:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for external identity providers #1397

Support for external identity providers #1397

adutra commented Apr 18, 2025 •

edited

Loading

adutra commented Apr 18, 2025 •

edited

Loading

adutra Apr 18, 2025

adutra Apr 18, 2025 •

edited

Loading

adutra Apr 18, 2025

adutra Apr 18, 2025

adutra Apr 18, 2025

adutra Apr 18, 2025

adutra Apr 19, 2025

adutra Apr 20, 2025

adutra Apr 20, 2025

adutra Apr 19, 2025

collado-mike Apr 23, 2025

Support for external identity providers #1397

Are you sure you want to change the base?

Support for external identity providers #1397

Conversation

adutra commented Apr 18, 2025 • edited Loading

Overview

TODOs

Authentication Types

Configuration Changes

Authenticator Changes

Token Broker Changes

Authentication Workflows in Detail

Internal

External

Mixed

OIDC Principal Mapping

OIDC Principal Roles Mapping

Mapping Examples

adutra commented Apr 18, 2025 • edited Loading

Choose a reason for hiding this comment

adutra Apr 18, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adutra commented Apr 18, 2025 •

edited

Loading

adutra commented Apr 18, 2025 •

edited

Loading

adutra Apr 18, 2025 •

edited

Loading