Central identity

Before you read. A contractor finishes on Friday. Their account exists on two hundred servers, because two hundred times somebody ran useradd.

You can write a loop. The loop will miss the eleven machines that were powered off, the four built last week from an image with the account already baked in, and the one nobody wrote down. A fortnight later the account still exists on some subset of the estate and you cannot say which subset.

If an account should not live in /etc/passwd, where should it live?

It should live in one place, and every machine should ask that place. Everything in this topic is a consequence of that: a directory holding the accounts, a protocol for asking it questions, a separate protocol for proving who you are, and a daemon on each machine wiring both into the two interfaces Linux already had.

Most of the confusion comes from two different questions being treated as one. “What is Jane’s user ID, home directory, and shell” is a lookup. “Is the person at this keyboard really Jane” is an authentication. Local machines answered both from files, which is why they feel like one question. Central identity usually answers them with two different protocols, and almost every decision downstream follows from keeping them apart.

Some words you will need

directory

A database optimised for being read, arranged as a tree, holding entries about people, groups, and machines. Read constantly, written rarely.

DN

Distinguished name. The unique name of one entry, written as a path from the entry up to the root of the tree: uid=jsmith,ou=people,dc=example,dc=com.

base DN

The entry a search starts from. Everything below it is in scope; everything above and beside it is not.

objectClass

What kind of thing an entry is. It decides which attributes the entry must have and which it may have. An entry can have several.

bind

The LDAP operation that says who you are for the rest of the connection. Anonymous, or a DN and a password.

realm

A Kerberos administrative domain, written in capitals by convention: EXAMPLE.COM. Usually the DNS domain, shouted.

KDC

Key distribution centre. The Kerberos server. It knows a key for every principal and it is the only thing that does.

TGT

Ticket-granting ticket. The first ticket you get, which you then exchange for tickets to individual services without typing your password again.

What breaks without this

Offboarding becomes unprovable. You can say you removed the account; you cannot demonstrate it, because there is no single place to look. “Show me that this person has no access” is a question the estate cannot answer.

User IDs drift apart, and file ownership is a number. alice is 1004 on one machine and 1007 on another, where 1004 is bob. Put an NFS share between them and Bob owns Alice’s files, with nothing broken in a way any tool will report.

One password change becomes two hundred password changes, and the machines that were down during the change are now the machines with the old password on them.

Nobody can answer “who can log in to this host”. The answer is in a file on the host, a different file on the next host, and the group memberships are in a third place. That question is the first one asked in every incident and every audit.

Two questions that look like one

Locally, lesson 28 covers both halves: /etc/passwd holds the identity (name, number, home, shell) and /etc/shadow holds the hash that answers the authentication. Two files, one machine.

The right-hand column changes; the two interfaces do not. No program was rewritten to understand LDAP. ls -l still calls getpwuid(), sshd still calls PAM, and something behind those interfaces started answering differently. That is why central identity can be added to an estate that already exists.

LDAP is a directory, not a login service

A directory is a tree, and every entry is named by its full path back to the root. That name is the DN.

dc=com
  dc=example                      <- the base DN, the top of what this server serves
    ou=people
      uid=jsmith                  <- an entry, dn: uid=jsmith,ou=people,dc=example,dc=com
      uid=asmith
    ou=groups
      cn=developers

Read a DN right to left and it is a path down from the root: dc=com, then dc=example inside it, then ou=people, then uid=jsmith. LDAP writes it the other way round, most specific part first. dc is a domain component, ou an organisational unit, cn a common name, uid a user identifier, attribute names being used to label a level of the tree.

# Debian 13 (trixie), x86_64
$ ldapsearch -x -H ldap://localhost -b dc=example,dc=com
# extended LDIF
#
# LDAPv3
# base <dc=example,dc=com> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#

# example.com
dn: dc=example,dc=com
objectClass: top
objectClass: dcObject
objectClass: organization
o: Example Inc
dc: example

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1

The lines starting with # are comments from ldapsearch, not data; the entry is the five lines from dn: to dc: example, in LDIF. Its objectClass values are the schema in action: organization is what requires o: Example Inc, dcObject what requires dc: example. An entry that claims an objectClass and omits a required attribute is rejected at write time, which is what stops a directory becoming a pile of half-filled forms.

You will frequently be pointed at a directory and not told its base DN. There is a defined way to ask.

# Debian 13 (trixie), x86_64
$ ldapsearch -x -H ldap://localhost -s base -b "" namingContexts supportedLDAPVersion
# extended LDIF
#
# LDAPv3
# base <> with scope baseObject
# filter: (objectclass=*)
# requesting: namingContexts supportedLDAPVersion 
#

#
dn:
namingContexts: dc=example,dc=com
supportedLDAPVersion: 3

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1

namingContexts is the answer to “what is the base DN”. A server can serve more than one, in which case you get more than one line. Note what dn: is on that entry: nothing. The root DSE is the one entry whose DN is empty, and -s base -b "" is the incantation for reaching it. Memorise it. It works against Active Directory too, and it is the fastest way to confirm that a directory is reachable, is answering, and is the one you think it is.

Adding a user, in the format the directory speaks

LDIF is also the import format. A file of entries, blank line between them:

# Debian 13 (trixie), x86_64
$ cat /root/jsmith.ldif
dn: ou=people,dc=example,dc=com
objectClass: organizationalUnit
ou: people

dn: uid=jsmith,ou=people,dc=example,dc=com
objectClass: inetOrgPerson
objectClass: posixAccount
uid: jsmith
cn: Jane Smith
sn: Smith
uidNumber: 10001
gidNumber: 10001
homeDirectory: /home/jsmith
loginShell: /bin/bash

Two entries in one file, and the order matters: ou=people has to exist before anything can be created inside it. Read the second entry as two things stacked on one object. inetOrgPerson carries the human attributes: cn, sn, optionally mail. posixAccount carries what a Linux machine needs: uidNumber, gidNumber, homeDirectory, loginShell, which are the last four fields of a line in /etc/passwd. That is the whole trick, and something on the client turns one into the other.

# Debian 13 (trixie), x86_64
$ ldapadd -x -D cn=admin,dc=example,dc=com -w secret -f /root/jsmith.ldif
adding new entry "ou=people,dc=example,dc=com"

adding new entry "uid=jsmith,ou=people,dc=example,dc=com"

-D is the bind DN, who you are claiming to be, -w is that DN’s password, and -x selects simple authentication rather than SASL. Yes, the password is on the command line where shell history and ps can see it; -W prompts instead.

# Debian 13 (trixie), x86_64
$ ldapsearch -x -LLL -H ldap://localhost -b dc=example,dc=com "(objectClass=*)" dn
dn: dc=example,dc=com

dn: ou=people,dc=example,dc=com

dn: uid=jsmith,ou=people,dc=example,dc=com

Three entries, and it is the default scope that returned all three. Every search has a base and a scope, and ldapsearch defaults to -s sub: the base entry plus everything beneath it, however deep. The alternatives are -s one, the immediate children of the base and not the base itself, and -s base, the base entry alone, exactly what the root DSE query used. Scope is the first thing to suspect when a search that should match returns nothing, closely followed by a base DN pointing one level too deep, and neither produces an error: both produce numEntries: 0, which reads like the entry does not exist.

# Debian 13 (trixie), x86_64
$ ldapsearch -x -LLL -H ldap://localhost -b dc=example,dc=com "(uid=jsmith)"
dn: uid=jsmith,ou=people,dc=example,dc=com
objectClass: inetOrgPerson
objectClass: posixAccount
uid: jsmith
cn: Jane Smith
sn: Smith
uidNumber: 10001
gidNumber: 10001
homeDirectory: /home/jsmith
loginShell: /bin/bash

Every field a login needs is there, and it came from one server rather than from this machine. -LLL drops the comment blocks, which is what you want once you are reading output rather than learning the format.

Filters are the query language: prefix notation, parenthesised, strange for about a day.

# Debian 13 (trixie), x86_64
$ ldapsearch -x -LLL -H ldap://localhost -b dc=example,dc=com "(&(objectClass=posixAccount)(uidNumber>=10000))" uid uidNumber homeDirectory
dn: uid=jsmith,ou=people,dc=example,dc=com
uid: jsmith
uidNumber: 10001
homeDirectory: /home/jsmith

(&(A)(B)) is A and B, (|(A)(B)) is or, (!(A)) is not. The trailing attribute names limit what comes back, the way naming columns limits a SELECT. (objectClass=posixAccount) is the filter every Linux client uses, because it is how you find the entries that represent Unix accounts rather than printers, groups, or people who have a mailbox and no shell.

Binding, and why that is not the same as logging in

A search happens on a connection, and every connection has an identity, set by the bind operation. If you do not bind as anybody, you are anonymous:

# Debian 13 (trixie), x86_64
$ ldapwhoami -x -H ldap://localhost; echo ---; ldapwhoami -x -H ldap://localhost -D cn=admin,dc=example,dc=com -w secret
anonymous
---
dn:cn=admin,dc=example,dc=com

Anonymous is an identity, not an error. This directory allows an anonymous connection to read entries, which is how every search above worked without a password. Whether that is acceptable is a policy decision, and it is usually the wrong default; the panel below has the argument.

# Debian 13 (trixie), x86_64
$ ldapsearch -x -LLL -H ldap://localhost -D cn=admin,dc=example,dc=com -w wrongpass -b dc=example,dc=com; echo "exit=$?"
ldap_bind: Invalid credentials (49)
exit=49

Invalid credentials (49) is worth knowing by number, because it turns up in application logs where the text has been thrown away. It means the bind failed: either the DN does not exist or the password is wrong, and the server deliberately does not say which, because that difference is a gift to somebody enumerating accounts.

And here is the thing that makes the lookup-versus-authentication distinction concrete. A bind is an authentication. Set a password on Jane’s entry and she can bind as herself:

# Debian 13 (trixie), x86_64
$ ldappasswd -x -H ldap://localhost -D cn=admin,dc=example,dc=com -w secret -s Passw0rd uid=jsmith,ou=people,dc=example,dc=com; ldapwhoami -x -H ldap://localhost -D uid=jsmith,ou=people,dc=example,dc=com -w Passw0rd
dn:uid=jsmith,ou=people,dc=example,dc=com

ldappasswd set the password, and ldapwhoami bound as uid=jsmith,... and got back her own DN. She authenticated to the directory, with two consequences. First, it works by sending the password to the server, on every authentication, fine on a TLS connection and catastrophic without one. Second, the client has to know Jane’s DN before it can try, which means a search before the bind, which means the client needs an identity of its own to do that search. That two-step is why a client configured for LDAP authentication has a bind DN in it that has nothing to do with the user logging in.

Kerberos exists because there is a better answer to the same question.

# Debian 13 (trixie), x86_64
$ grep -E "^ldap" /etc/services
ldap		389/tcp			# Lightweight Directory Access Protocol
ldap		389/udp
ldaps		636/tcp				# LDAP over SSL
ldaps		636/udp

# Debian 13 (trixie), x86_64
$ ldapsearch -Y EXTERNAL -H ldapi:/// -b cn=config -LLL dn 2>&1 | head -20
SASL/EXTERNAL authentication started
SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
SASL SSF: 0
dn: cn=config

dn: cn=module{0},cn=config

dn: cn=schema,cn=config

dn: cn={0}core,cn=schema,cn=config

dn: cn={1}cosine,cn=schema,cn=config

dn: cn={2}nis,cn=schema,cn=config

dn: cn={3}inetorgperson,cn=schema,cn=config

dn: olcDatabase={-1}frontend,cn=config

dn: olcDatabase={0}config,cn=config

Kerberos, tickets instead of a password

Kerberos answers the authentication question with a design goal LDAP binds do not have: the password never crosses the network, and no service you log in to ever learns it.

Step one is the one that surprises people. kinit does not send the password. It derives a key from it locally, encrypts the current time with that key, and sends that. The KDC has the same key in its database, decrypts the timestamp, and if the result is a sensible time then the client must hold the right key. This is pre-authentication, and the timestamp is why: a static proof could be recorded and replayed for ever, and a timestamp cannot, because it is only accepted inside a narrow window.

Step two returns a ticket you cannot read, plus a copy of the session key that you can. The TGT is encrypted with the KDC’s own key, so it is opaque to you; you hold it and hand it back. Everything after this uses the session key rather than your password, which is why you type a password once and then reach a dozen services without typing it again. Single sign-on is a consequence of the design rather than a feature bolted on.

Step five is where the value lands. The file server decrypts your service ticket with its own long-term key, from a keytab on its disk. It does not ask the KDC anything, has never seen your password, and cannot replay your ticket against another service, because a service ticket names exactly one service. A compromised file server does not become a compromised password.

The vocabulary, once, because it is the part that gets examined:

The client-side commands are three: kinit to get a TGT, klist to list what you hold, kdestroy to throw it away. Run klist on a machine holding no tickets and it does not go quiet:

# openSUSE Leap 16.0, x86_64
$ klist; echo "exit=$?"
klist: Invalid UID in persistent keyring name while resolving ccache
exit=1

That is not the message most people expect, and it is the more instructive one, because it is complaining about where the cache would be rather than about what is in it. The configuration explains why:

# openSUSE Leap 16.0, x86_64
$ grep -vE "^\s*(#|$)" /etc/krb5.conf; echo ---; ls /etc/krb5.conf.d
includedir  /etc/krb5.conf.d
[libdefaults]
    dns_canonicalize_hostname = false
    rdns = false
	default_ccache_name = KEYRING:persistent:%{uid}
[realms]
[logging]
    kdc = FILE:/var/log/krb5/krb5kdc.log
    admin_server = FILE:/var/log/krb5/kadmind.log
    default = SYSLOG:NOTICE:DAEMON
---
crypto-policies
enable_sssd_conf_dir

default_ccache_name = KEYRING:persistent:%{uid} is the answer. Tickets go into the kernel keyring, keyed by uid, and this container has no persistent keyring for uid 0 to attach to, so the library cannot even name the cache. MIT’s own compiled-in default is a file, FILE:/tmp/krb5cc_%{uid}, and every distribution in this topic overrides it, which is why documentation telling you to look in /tmp is describing a machine you probably do not have. Name a file cache explicitly and the message becomes the one worth memorising:

# openSUSE Leap 16.0, x86_64
$ export KRB5CCNAME=FILE:/tmp/krb5cc_0; klist; echo "exit=$?"
klist: No credentials cache found (filename: /tmp/krb5cc_0)
exit=1

That is what “not logged in to Kerberos” looks like, no credential cache, exit status 1. Two failures, two different messages, and neither means a password was rejected. KRB5CCNAME overrides the configured default for one process and its children, which makes it the fastest way to test without disturbing anything. The keyring has an operational tail, too: because the cache is keyed by uid, sudo to another user and your tickets do not come with you, and a daemon running under its own account cannot borrow yours. That is deliberate, and it is why services get keytabs instead of ticket caches.

[realms] is empty here, because nothing has joined this machine to anything. Ask for a ticket anyway and the client says precisely what it failed to find:

# openSUSE Leap 16.0, x86_64
$ export KRB5CCNAME=FILE:/tmp/krb5cc_0; kinit jsmith@EXAMPLE.COM; echo "exit=$?"
kinit: Cannot find KDC for realm "EXAMPLE.COM" while getting initial credentials
exit=1

Cannot find KDC for realm is a discovery failure, not an authentication failure. It never reached the point of asking for a password. On a configured client the [realms] section names the realm’s KDCs, or is left empty on purpose so the client finds them from DNS SRV records: _kerberos._udp.EXAMPLE.COM and friends, looked up because dns_lookup_kdc defaults to true. That is what Active Directory relies on, and it is why a client with the wrong DNS server presents as Kerberos being broken rather than as DNS being broken.

kinit: Clock skew too great while getting initial credentials

timedatectl
chronyc tracking

SSSD, the client that glues it all to NSS and PAM

Nothing described so far knows how to make ls -l print a username. SSSD is the piece that does. It runs as a daemon, it is configured per domain, and each domain names an identity provider and an authentication provider, separately, which is the same distinction the whole topic is built on.

# openSUSE Leap 16.0, x86_64
$ sssd --version; echo ---; klist -V
2.10.2
---
Kerberos 5 version 1.21.3

Worth checking on a machine you have inherited, because SSSD option names and defaults have moved between the 1.x that shipped with RHEL 7 and the 2.x everything ships now.

Here is a working configuration for a machine that gets its users from LDAP:

# openSUSE Leap 16.0, x86_64
$ cat /etc/sssd/sssd.conf
[sssd]
domains = example.com
services = nss

[domain/example.com]
id_provider = ldap
ldap_uri = ldap://localhost
ldap_search_base = dc=example,dc=com
ldap_id_use_start_tls = false
cache_credentials = true

Every line is load-bearing. domains = example.com names this stanza and is not a DNS lookup. It has to match the text after domain/ in the section header and nothing else. services = nss says which of SSSD’s interfaces to answer on. id_provider = ldap sets where identity comes from and ldap_search_base is the base DN from earlier.

The last two lines are the interesting ones, and one is a lie of omission. ldap_id_use_start_tls defaults to true, so setting it to false is an explicit downgrade, honest about what this is: a demonstration directory reached over localhost, where there is no network to eavesdrop on. On anything real that one line decides whether every identity lookup crosses the network in the clear. And cache_credentials = true, off by default, stores a hash of a successful authentication locally, for the panel below. In this file it does nothing at all, because nothing here authenticates, which is a good illustration of how a setting can be present, correct, and inert.

Notice what is missing: there is no auth_provider, and services does not list pam. This machine can look users up and cannot log them in. That is not a broken configuration, it is the split from the top of the topic made visible in six lines. Adding auth_provider = krb5 with a KDC, or auth_provider = ldap to bind against the directory, is what turns lookups into logins. With auth_provider absent, SSSD falls back to the identity provider where that provider can authenticate, which is why id_provider = ad on its own gets you both halves and id_provider = ldap on its own quietly gets you authentication by bind.

Then NSS has to be told to ask SSSD at all, in /etc/nsswitch.conf, a file which, on this particular distribution, had to be created first, for reasons the distributions section covers.

# openSUSE Leap 16.0, x86_64
$ grep -E "^(passwd|group)" /etc/nsswitch.conf; echo ---; getent passwd jsmith; echo ---; id jsmith
passwd:  compat sss
group:   compat sss
---
jsmith:*:10001:10001:Jane Smith:/home/jsmith:/bin/bash
---
uid=10001(jsmith) gid=10001 groups=10001

That is a /etc/passwd line for a user who is not in /etc/passwd. The fields came from the directory: uidNumber became the third field, gidNumber the fourth, cn the GECOS field, homeDirectory and loginShell the last two. The * in the password field is NSS declining to expose a hash, which is correct. That is authentication’s business, not identity’s.

id jsmith is the more revealing of the two. uid=10001(jsmith) carries a name in brackets and gid=10001 does not, because there is a user entry for 10001 in the directory and no group entry for it anywhere. A bare number with no name beside it means the lookup for that object failed, not that it has no name, the same tell appears in ls -l, which prints numbers instead of names for exactly this reason.

The supplementary list is empty because group membership is a separate set of entries with their own objectClass, and this directory has none. In the older RFC 2307 schema a posixGroup lists its members as memberUid: jsmith, plain usernames; in RFC 2307bis, which FreeIPA uses, a group lists member: with full DNs, and Active Directory’s own schema works the same way. SSSD is told which by ldap_schema, which defaults to rfc2307, and pointing it at a 2307bis directory without changing that produces users who log in perfectly and belong to nothing. “Their groups are missing” is the most common real complaint about a new SSSD deployment, and this is usually why.

getent is the right tool because it asks the same question the operating system asks, through the same NSS path, rather than asking LDAP directly:

# openSUSE Leap 16.0, x86_64
$ getent passwd root; echo ---; getent passwd nosuchuser; echo "exit=$?"
root:x:0:0:root:/root:/bin/bash
---
exit=2

Two useful facts in one transcript. root comes from /etc/passwd: compat is first on the line and it answered, so SSSD was never consulted, which is exactly what you want for the account that has to work when the network is down. And a user who exists nowhere produces no output and exit status 2, which is what you test in a script rather than grepping for an empty string.

sssctl config-check is the first command to run on a non-working daemon, which is unforgiving about its configuration file and vaguer than this in the journal:

# openSUSE Leap 16.0, x86_64
$ sssctl config-check
Issues identified by validators: 0

Messages generated during configuration merging: 0

Used configuration snippet files: 0

Then the cache, which is where the surprises live:

# openSUSE Leap 16.0, x86_64
$ getent passwd jsmith >/dev/null; sssctl user-show jsmith
Name: jsmith
Cache entry creation date: 08/08/26 17:51:25
Cache entry last update time: 08/08/26 17:51:25
Cache entry expiration time: 08/08/26 19:21:25
Initgroups expiration time: Initgroups were not yet performed
Cached in InfoPipe: No

Every entry has a creation time and an expiration time, ninety minutes apart here, which is the default entry_cache_timeout of 5400 seconds. Until it expires, changes made in the directory are invisible on this machine: a user renamed centrally, a shell changed, a group membership added. “I changed it in the directory and the server has not noticed” is very often exactly this.

Initgroups expiration time: Initgroups were not yet performed is the other line worth reading. Identity and group membership are fetched separately: looking a user up caches the user, and only an initgroups call, which is what id and a real login do, goes back for the list of groups they are in. A getent passwd that succeeds therefore proves nothing about whether groups will resolve.

The cache itself is a few files, and the mode on them is not decoration:

# openSUSE Leap 16.0, x86_64
$ ls -l /var/lib/sss/db
total 3768
-rw-------. 1 root root 1286144 Aug  8 17:52 cache_example.com.ldb
-rw-------. 1 root root 1286144 Aug  8 17:52 config.ldb
-rw-------. 1 root root 1286144 Aug  8 17:52 timestamps_example.com.ldb

0600, owned by root, one cache_<domain>.ldb per configured domain. It holds the identity data of everybody who has logged in to this machine and, with cache_credentials on, hashes of their passwords. A backup that sweeps up /var/lib/sss/db is a backup carrying credentials, and so is a disk image of a decommissioned machine.

sudo sss_cache -u jsmith        # one user
sudo sss_cache -g developers    # one group
sudo sss_cache -E               # everything, but only marks it expired

Active Directory, realm join, and where Winbind fits

Active Directory is an LDAP directory and a Kerberos KDC in one product, plus DNS and a proprietary replication mechanism. Everything above applies: the tree, the DNs, the base DN, the tickets. Two things differ enough to matter.

Its schema is not the POSIX one. An AD user has sAMAccountName where a posixAccount has uid, and by default has no uidNumber at all. Something has to produce a numeric uid, and there are two answers: store POSIX attributes in AD and read them, or have the client derive uids algorithmically from the user’s security identifier.

ldap_id_mapping is the switch, and its default depends on which provider you are using, which is exactly the sort of thing that gets assumed rather than checked. The AD provider sets it to true and maps from objectSID; under the generic id_provider = ldap it is false and SSSD expects real uidNumber attributes. Turning it off on an AD domain is how you say “we populated POSIX attributes, use those”.

The algorithm is deterministic rather than random (it hashes the domain SID to pick a slice, then derives the uid from the relative identifier within it) but determinism only holds if every client agrees on the arithmetic. The inputs are ldap_idmap_range_min (200000), ldap_idmap_range_max (2000200000), and ldap_idmap_range_size (200000). Change the range size on one machine and every uid on that machine changes, so files written before the change are owned by nobody after it. Mixing algorithmic mapping with POSIX attributes across an estate, or mixing range settings, reproduces the NFS ownership problem from the top of this topic on purpose. The check is one command on two hosts: id someuser should return the same number on both, and if it does not, nothing else about the deployment is worth debugging yet.

Joining is a real operation, not a configuration file. A domain member has an identity of its own, a computer account with a password, and joining creates it:

realm discover ad.example.com
sudo realm join --user=Administrator ad.example.com
realm list

realm discover is a DNS SRV lookup plus a root DSE query, and it is worth the ten seconds: it reports the realm name, the server software, and which client packages are needed, before you have changed anything. realm join then gets a Kerberos ticket as the administrator you named, creates the computer object, writes /etc/krb5.keytab, generates /etc/sssd/sssd.conf, and rewires PAM and NSS through whichever tool the family uses: authselect on RHEL, pam-auth-update on Debian.

The two client stacks, which the exam does ask you to distinguish:

SSSD is the default and Winbind is the exception. Use SSSD unless this machine is a Samba file server presenting shares to domain users, in which case smbd needs Winbind’s mapping of Windows security identifiers to Unix uids, because that mapping has to be the same one the file server uses when it writes ownership onto the disk. Running both at once is possible and is a well-known way to end up with two different uids for one person.

sudo klist -k /etc/krb5.keytab
sudo klist -Kte /etc/krb5.keytab

Samba is two different jobs with one name

This trips people up in conversation as much as in configuration, so state it plainly: Samba the file service and Samba the domain member are separate things, and either can exist without the other.

A standalone Samba server with security = user keeps its own account database with smbpasswd, shares a directory, and knows nothing about LDAP or Kerberos. That is what most people mean by “a Samba server”. Change one line to security = ads and the same daemon is a domain member: users authenticate with domain credentials, their tickets are Kerberos tickets, and smbd needs a way to turn a Windows security identifier into a Unix uid so it can set ownership on the files it writes. That is the Winbind job from the table above, and it is the one case where Winbind rather than SSSD is the answer.

The third row is the one people forget exists. samba-tool domain provision stands up an AD-compatible directory, KDC, and DNS, and Windows clients join it as they would join a Microsoft one. That makes Samba a peer of Active Directory rather than a client of it, and it is why “Samba” is an unhelpfully broad answer to “how are you doing identity”.

Across distributions

The protocols are identical everywhere. The package names, the file locations, and the tool that rewires PAM are not.

Two rows in that table are traps. The first is the LDAP server: Red Hat deprecated openldap-servers in RHEL 7.4 and dropped it in RHEL 8, so on RHEL, AlmaLinux, and Rocky the client tools are present and the server package simply is not there, 389 Directory Server is the supported one. The slapd behind the Debian captures in this topic has no RHEL-family equivalent to install.

The second is pam_mkhomedir. There is no libpam-mkhomedir package, on Debian or anywhere else; the module ships inside libpam-modules and is switched on with pam-auth-update --enable mkhomedir. On the RHEL family the same job is usually done by oddjob-mkhomedir, which creates the directory through a privileged helper rather than inside the PAM stack, and is turned on as an authselect feature: authselect select sssd with-mkhomedir. Either way, a central user logging in to a machine with neither enabled lands in a home directory that does not exist, gets dropped into / with a shell that cannot write its history, and reports it as “the account is broken”.

authselect is the row worth knowing for the exam and for your fingers. It manages /etc/pam.d and /etc/nsswitch.conf as a versioned profile rather than as files you edit, so authselect select sssd writes a known-good stack and authselect current tells you what is in force. Editing /etc/pam.d/system-auth by hand on such a system produces a warning and a change the next authselect run discards. It replaced authconfig, the name in older documentation. What it writes carries three decisions:

passwd:     files sss systemd
shadow:     files systemd
group:      files [SUCCESS=merge] sss [SUCCESS=merge] systemd

files comes before sss, so root and the other local accounts still resolve when the directory is unreachable. shadow names no sss because there is nothing to name: libnss_sss exports getpwnam, getgrnam, gethostbyname and their relatives and no shadow entry points at all, so sss on a shadow line is inert wherever it appears, and it does appear, because Debian’s sssd-common package writes it in.

[SUCCESS=merge] on the group line is the third decision, and the one to recognise. NSS normally stops at the first source that answers, so a group defined both in /etc/group and in the directory would return only its local members: a user’s central membership of wheel or docker vanishes silently, with no error anywhere. [SUCCESS=merge] means “having found a match, carry on and combine the member lists”. The authselect sssd profile ships it; Debian’s nsswitch.conf after installing SSSD does not, and a hand-written line will not have it unless you put it there. If a group is defined in exactly one place you will never notice; the moment somebody adds a local group with a name the directory already uses, you will.

SUSE adds a wrinkle that will confuse anybody arriving from the other two families, and it is why the SSSD captures earlier could grep a file this section says does not exist:

# openSUSE Leap 16.0, x86_64
$ ls -l /etc/nsswitch.conf /usr/etc/nsswitch.conf 2>&1; grep -E "^(passwd|group)" /usr/etc/nsswitch.conf
ls: cannot access '/etc/nsswitch.conf': No such file or directory
-rw-r--r--. 1 root root 2222 Aug 26  2025 /usr/etc/nsswitch.conf
passwd:		compat systemd
group:		compat [SUCCESS=merge] systemd

There is no /etc/nsswitch.conf at all until something creates one. The distribution’s default lives in /usr/etc, and /etc is reserved for local changes, you copy the file down and edit the copy, which is exactly what happened to the machine behind the SSSD captures above. Note what the shipped default says: compat systemd, with no sss anywhere. Nothing on this distribution adds it for you, and until it is added, SSSD can be installed, configured, running, and answering nothing. The failure mode is silent in both directions: grep in /etc finds no file and tells you nothing about what the machine is doing, and editing /usr/etc instead is real until the next package update overwrites it. getent passwd is the question that does not care which file won.

Prove it

# Is the directory reachable, and what is its base DN
ldapsearch -x -H ldap://dc1.example.com -s base -b "" namingContexts

# Can I read the entry for a user, and does it have POSIX attributes
ldapsearch -x -LLL -H ldap://dc1.example.com -b dc=example,dc=com "(uid=jsmith)"

# Does the operating system agree, which is a different question
getent passwd jsmith
id jsmith

# Is SSSD's configuration valid, and what is cached
sudo sssctl config-check
sudo sssctl user-show jsmith
sudo sssctl domain-status example.com

# Do I hold a Kerberos ticket, and until when
klist

# Do the clocks agree, which Kerberos requires
timedatectl

ldapsearch working and getent not working is the single most informative result in that list. It says the directory, the network, and the credentials are fine, and that the problem is on this machine between NSS and SSSD. Running both before forming a theory takes ten seconds and eliminates half the possible causes.

What trips people up

1. The bind DN is not a username

-D jsmith fails. The bind DN is a full distinguished name, uid=jsmith,ou=people,dc=example,dc=com. Active Directory additionally accepts jsmith@example.com and EXAMPLE\jsmith, which is convenient and teaches the wrong lesson to anybody who then tries it against OpenLDAP. The error is ldap_bind: Invalid credentials (49) either way and the server will not say whether it was the name or the password, so check the DN with a search first.

2. ldapsearch works and getent does not

They ask different things through different code. ldapsearch speaks to the server; getent goes through NSS, which reads /etc/nsswitch.conf, which has to name sss, and then SSSD has to be running with a valid configuration. So: sss missing, SSSD not running, a filter in sssd.conf that excludes the entry, or a search base that does not contain it. sssctl config-check plus systemctl status sssd narrows four candidates in two commands.

3. Clock skew too great

Kerberos rejects messages whose timestamps differ from the receiver’s clock by more than five minutes, because those timestamps are what stop a captured authenticator being replayed. Check the clock on the client, the KDC, and the service, since any pair can be the offending one. System clock synchronized: no from timedatectl is the finding, and the fix is NTP rather than anything in krb5.conf.

4. Everything works until the cache expires

Change something in the directory, watch the machine ignore it for ninety minutes, conclude the change did not save. sssctl user-show prints the expiration time and sss_cache -u <user> marks the entry stale. The inverse trips people harder: an account disabled centrally still works on the machines where it is cached, and with cache_credentials on it works offline too. Offboarding is not complete when the directory entry changes.

5. ldaps:// on 636 versus StartTLS on 389

636 with implicit TLS was never standardised; StartTLS on 389 is the specified mechanism. Both work in practice, and the trap is a client that attempts StartTLS and silently continues in the clear when it fails. Make it mandatory, and check certificates rather than setting ldap_tls_reqcert = never.

6. Assuming LDAP is doing the authentication

It may be, by bind, and that sends the password to the server. On an Active Directory or FreeIPA deployment the password is checked by Kerberos and the directory is only answering identity questions. Which one is in play decides whether your TLS configuration is a serious matter or a merely important one, and it is written in sssd.conf as auth_provider.

Work it through

A new machine was joined to the domain this morning. id jsmith prints her uid, gid, home directory, and shell correctly. ssh jsmith@newhost rejects the password. Her password is definitely right, because she is logged in to an older server with it right now. Reason it out before reading on.

Name which of the two questions is failing. id works, so identity lookup is fine: NSS is finding sss, SSSD is running, the search base is right, and the machine can reach the directory. The whole left-hand side of the diagram is healthy and the failure is on the right, which is a completely different set of causes.

Ask what is answering the authentication question.

grep -E 'id_provider|auth_provider' /etc/sssd/sssd.conf

If auth_provider is missing, SSSD uses the identity provider, which for id_provider = ldap means a bind, so the client needs to reach port 389 or 636 and needs TLS configured. If it says krb5 or the domain is ad, Kerberos is answering and the causes are Kerberos causes.

Test that path directly, as her:

kinit jsmith@EXAMPLE.COM
klist

This is the sharpest test available, because it removes SSSD, PAM, and sshd from the picture entirely. Three outcomes and three different lessons:

• Clock skew too great, the new machine’s clock. timedatectl on it and on a working server. This is the classic new-machine failure, and virtual machines cloned from a template are especially prone to it.
• Preauthentication failed. The password really is wrong for this realm, or she is in a different realm than the one the machine defaults to.
• kinit succeeds and ssh still fails, Kerberos is fine and PAM is not wired up. On the RHEL family, authselect current; a profile without SSSD means the join wrote sssd.conf and nothing rewired PAM.

Now change one detail. Suppose id jsmith had also failed, on a machine where ldapsearch against the same server returns her entry. Then nothing on either side is reaching SSSD, and the candidates collapse to nsswitch.conf missing sss or the daemon not running. Same symptom class, entirely different half of the system. And one more: suppose everything works for her and fails for a colleague who has never logged in to any machine. That is the cache, and it means the directory is unreachable and you have been reading cached answers for everybody who used the host recently.

The point worth extracting: identity and authentication fail separately, and the first command should be the one that tells you which. id answers the identity question without touching authentication; kinit answers the authentication question without touching NSS, PAM, or the service you were actually trying to use. Two commands, thirty seconds, and the search space is halved before you have read a log file.

Try it

Optional, and it needs no directory server of your own. A container is enough.

1. Start a container and install a directory: on Debian, apt-get install slapd ldap-utils, answering the prompts with a domain of example.com.
2. ldapsearch -x -H ldap://localhost -s base -b "" namingContexts. Confirm the base DN it reports matches the domain you gave.
3. ldapsearch -x -H ldap://localhost -b dc=example,dc=com. Read the objectClass lines and say which attribute each one is requiring.
4. Write an LDIF file with an ou=people container and one user carrying both inetOrgPerson and posixAccount, then load it with ldapadd -x -D cn=admin,dc=example,dc=com -W -f yourfile.ldif.
5. Find your user with a filter: "(uid=yourname)", then "(&(objectClass=posixAccount)(uidNumber>=10000))".
6. ldapwhoami -x -H ldap://localhost with no -D, then with -D and the admin DN. Then deliberately get the password wrong and read the result code.
7. klist on any machine. Read the message and say precisely what it is telling you about your Kerberos state.
8. On a machine with SSSD, sssctl config-check, then getent passwd <a central user>, then sssctl user-show that user and find the cache expiry.

Verification step. You have it when you can be handed a hostname and a domain and say, without looking anything up, which command tells you the base DN, which tells you whether the machine’s identity lookups work, and which tells you whether authentication works, and why those are three different commands.

Check yourself

ldapsearch -x -H ldap://dc1.example.com -s base -b "" namingContexts

References

• RFC 4511: Lightweight Directory Access Protocol (LDAP): The Protocol - IETF. Accessed 2026-08-08.
• RFC 4120: The Kerberos Network Authentication Service (V5) - IETF. Accessed 2026-08-08.
• ldapsearch(1) - Debian. Accessed 2026-08-08.
• nsswitch.conf(5) - Linux man-pages project. Accessed 2026-08-08.
• sssd.conf(5) - Debian. Accessed 2026-08-08.
• realm(8) - Debian. Accessed 2026-08-08.
• krb5.conf(5) - MIT Kerberos. Accessed 2026-08-08.
• smb.conf(5) - Samba Team. Accessed 2026-08-08.

Captured output came from two containers. The LDAP transcripts are a real OpenLDAP server installed in a Debian 13 container, with the directory built by the LDIF shown in the topic. The SSSD and Kerberos transcripts came from an openSUSE Leap 16 container running its own directory with SSSD pointed at it, which is why klist reports no credential cache: there is no KDC in these captures, and the Kerberos exchange itself is described from RFC 4120 and the MIT documentation rather than shown. Blocks without a distribution and architecture header are illustrative or quoted from documentation.