[Samba] Slow directory listing after adding new trusted domain to current forest

[Jeff Hodge]

We have an ubuntu 14.04 server running samba 4.1.6 that is a member of our
OLDDOMAIN. We recently added a new trusted domain to our forest. We
noticed any new directories created by users in that new domain take a long
time for the directory to list in the command line.

This does not seem to happen when listing the directories with only
user/group ID:

***@d101:/home/OLDDOMAIN/test/data/Production# time ls -lan
total 36
drwxrwxrwx 6 0 0 4096 Jul 29 12:50 .
drwxrwxrwx 3 0 0 4096 Jul 28 10:24 ..
drwxrwxr-x+ 2 590 1319 4096 Jul 28 16:16 NEWDOMAIN-Jeff
drwxrwxr-x+ 2 500 504 4096 Jul 28 14:46 NEWDOMAIN-Jeff-acl
drwxrwxr-x+ 2 500 504 4096 Jul 29 12:50 Jeff

real 0m0.002s
user 0m0.002s
sys 0m0.000s

When trying to list the directory with the user/group mappings, it is much
slower:

***@d101:/home/OLDDOMAIN/test/data/Production# time ls -la
total 36
drwxrwxrwx 6 root root 4096 Jul 29 12:50 .
drwxrwxrwx 3 root root 4096 Jul 28 10:24 ..
drwxrwxr-x+ 2 NEWDOMAIN\jhodge NEWDOMAIN\domain users 4096 Jul 28 16:16
NEWDOMAIN-Jeff
drwxrwxr-x+ 2 OLDDOMAIN\jhodge OLDDOMAIN\domain users 4096 Jul
28 14:46 NEWDOMAIN-Jeff-acl
drwxrwxr-x+ 2 OLDDOMAIN\jhodge OLDDOMAIN\domain users 4096 Jul
29 12:50 Jeff

real 0m19.727s
user 0m0.005s
sys 0m0.000s

It does seem to cache the information, because another listing a few
moments later is normal. However if you wait another 10 minutes, it will
take 10-20 seconds to list the directory.

***@d101:/home/OLDDOMAIN/test/data/Production# time ls -la
total 36
drwxrwxrwx 6 root root 4096 Jul 29 12:50 .
drwxrwxrwx 3 root root 4096 Jul 28 10:24 ..
drwxrwxr-x+ 2 NEWDOMAIN\jhodge NEWDOMAIN\domain users 4096 Jul 28 16:16
NEWDOMAIN-Jeff
drwxrwxr-x+ 2 OLDDOMAIN\jhodge OLDDOMAIN\domain users 4096 Jul
28 14:46 NEWDOMAIN-Jeff-acl
drwxrwxr-x+ 2 OLDDOMAIN\jhodge OLDDOMAIN\domain users 4096 Jul
29 12:50 Jeff

real 0m0.010s
user 0m0.000s
sys 0m0.006s

It would seem this is a problem with the mapping of the new domain
user/group ID's. This behavior was never seen before we added the new
domain. I noticed the group mapping/account polices have not been updated
since the server was added to the domain, does this need to be updated?
The winbindd_cache does seem to update and idmap table have updated
recently.

***@d101:/var/lib/samba# ll
total 7656
drwxr-xr-x 6 root root 4096 Apr 13 15:26 .
drwxr-xr-x 43 root root 4096 Feb 4 2015 ..
-rw------- 1 root root 421888 Apr 29 2014 account_policy.tdb
-rw------- 1 root root 425984 Apr 29 2014 group_mapping.tdb
drwxr-xr-x 10 root root 4096 Apr 29 2014 printers
drwxr-xr-x 3 root root 4096 Mar 4 2015 private
-rw------- 1 root root 528384 Apr 29 2014 registry.tdb
-rw------- 1 root root 421888 Jul 29 13:04 share_info.tdb
drwxrwx--T 2 root sambashare 4096 Jul 28 09:56 usershares
-rw------- 1 root root 5353472 Aug 1 10:36 winbindd_cache.tdb
-rw-r--r-- 1 root root 663552 Jul 29 12:53 winbindd_idmap.tdb
drwxr-x--- 2 root root 4096 Apr 13 15:26 winbindd_privileged

Any idea why this slowness would happen and how it can be resolved?
Rejoining the domain? Can you force samba to recreate the group
mappings/idmaps?

Thank you for your time.