Kerberos handbook

Here is a kerberos handbook to document all kerberos-related stuff that may be useful to read later-on.

Journey of a kerberos ticket

SSH

When authentication with kerberos (gssapi_keyex or gssapi_with_mic authentication methods), the Kerberos ticket is transfered (delegated) if it asked on the client side (GSSAPIDelegateCredentials).

Ticket Delegation

To do so, a ccache (Credential Cached) is created by the SSH daemon using the krb5_cc_new_unique call in libkrb5.

This will create a ccache, with a name derived from the default but unique among all users’s sessions.

This new ccache will be made the default one with a call to krb5_cc_switch if the ccache type supports it (DIR in 1.10, KEYRING in 1.12, KCM in 1.13). No KRBCCNAME is exported in this case.

If the ccache does not support credential collections, a KRBCCNAME environment variable is exported by SSH.

Ticket Storage

In case the FILE ccache type (the default one), the ticket is stored as-is in the specified file

In case of KEYRING ccache type, the ticket will be stored as a big_key key and then linked in the specified keyring. No easy method exists to list all stored credentials.

In case of KCM ccache type, the ticket storage is handled by a KCM daemon (sssd-kcm or h5l deamons). To list the available credential, use ldbsearch from the ldb-tools package.

# ldbsearch -H /var/lib/sss/secrets/secrets.ldb cn
# record 1
dn: cn=90909b36-1788-47b9-8bb4-c073d901267e-15117,cn=ccache,cn=15117,cn=persistent,cn=kcm

# record 2
dn: cn=default,cn=15117,cn=persistent,cn=kcm

# record 3
dn: cn=66f38325-a79a-4eb6-a570-d7be36a2bdee-0,cn=ccache,cn=0,cn=persistent,cn=kcm

# record 4
dn: cn=ccache,cn=15117,cn=persistent,cn=kcm

# record 5
dn: cn=ccache,cn=0,cn=persistent,cn=kcm

# returned 5 records
# 5 entries
# 0 referrals
Ticket usage

GSS-enabled application can find kerberos ticket with either the KRB5CCNAME environnment variable or the default configured one (in /etc/krb5.conf).

If the ccache supports it, the application may search for multiple principals within a single ccache collection. See kerberos documentation for details.

Ticket cleanup

The credential cleanup is handled by 2 consecutive mechanisms. The first one is the PAM stack. During the session tear-down process, the PAM stack will delete the credential it have created. The second one is the GSS part of SSH, it will delete the credential it have delegated.

Things to keep in mind:

  • If the ccache supports cache collection, the primary cache will be the same for all sessions of a user. Both of them delegates a new ticket and calls krb5_cc_switch, the last one wins. During cleanup, if the primary cache got deleted, the remaining session will go through the collection cache to find a new one.

  • For all GSS-enabled application, if any of them creates a new credential, they must be in charge of deleting it. No expiration (except for keyring ccache type) is done automatically.

  • The SSH/PAM delegation process may not be in the same namespace as the user session. So delgating a ticket in /tmp, for instance, may not work if you have polyinstantiated directories.

Batch application

When a node is allocated to execute a job, it have access to the user credential through the auks service. The main objective is to retrieve it and make it available to both the user and the system for daemon that may require it.

To do so, the auks slurm plugin will retrieve it using a auks API call and the store it in a credential cache.

To choose where to store the ticket, auks will call krb5_cc_new_unique using the system’s configuration. This will create a new unique ccache.

At a minimum, this cache must be available in :

  • the user’s environment : using the KRB5CCNAME environment variable)

  • the userland auks daemon in charge of the renewal : using the KRB5CCNAME environment variable

  • the system daemon in charge of creating GSS contexts with NFS server (rpc-gssd). This daemon will only use the following methods sequentially:

    • Using the system-wide default ccache.

    • Iterating all the available ccaches in the given directory (/tmp by default).

Note

Please note that, with the default configuration (ccache in tmp), the first method never works because the default path is always wrong (/tmp/krb5cc_%UID).

Warning

This does not apply to root authentication (for mount commands). It this case, the rpc.gss should be configured to use a keytab and machine credentials (use-machine-creds in /etc/nfs.conf).

Once the ccache is populated, the ccache full name is exported to the user (and auks process) environment. Thus, overriding any primary ccache set by krb5_cc_switch.

If the ccache supports cache collections, this ccache is available to system daemons because it will search through the complete collection. If not, it will probably fall-back to /tmp scraping.

During cleanup, the auks slurm plugin will delete the ccache it created.