[Samba] NT_STATUS_IO_TIMEOUT at open_socket_out_send due to firewall

[Mike Ely]

Hi,

We've got a few layers of firewall between our DCs and a domain guest
providing winbind services. What I've noticed is that on first run
winbind tries to use the high ports to set up part of the communication.
During that process, the winbindd process runs at 100% CPU, and it takes
about two minutes to time out. After that time things settle down and
winbind works perfectly.

For various reasons we're not eager to open a wide range of ports across
all the firewalls, and would like to know if there's a way in smb.conf
to skip the step that's hanging.

I've validated that opening the port in the below log (49155) allows
winbind to start cleanly and work without hanging for two minutes, but
this was only for testing purposes and the port had to be closed back up
after.

Here's loglevel 10 from where the thing happens on a Centos7 box running
Samba 4.2.10:

[2016/06/15 11:23:34.554465, 3, pid=6383, effective(0, 0), real(0, 0)]
../source3/lib/util_sock.c:636(open_socket_out_send)
Connecting to [redacted IP of DC on different segment] at port 49155
[2016/06/15 11:25:41.772517, 10, pid=6383, effective(0, 0), real(0, 0),
class=winbind] ../source3/winbindd/winbindd_cm.c:2698(cm_connect_lsa_tcp)
cli_rpc_pipe_open_schannel_with_key failed: NT_STATUS_IO_TIMEOUT



smb:conf:
[global]
workgroup = TEST
realm = TEST.EXAMPLE.COM
security = ads
idmap config * : range = 16777216-33554431
template shell = /bin/false
kerberos method = secrets and keytab
winbind use default domain = true
winbind offline logon = false
log file = /var/log/samba/%m.log
log level = 10
netbios name = TESTWB
server string = WB server
invalid users = root
socket options = TCP_NODELAY
winbind enum users = yes
winbind enum groups = yes
winbind max domain connections = 5
winbind max clients = 1000

[Mike Ely]

Bump.

Post by Mike Ely
Hi,
We've got a few layers of firewall between our DCs and a domain guest
providing winbind services. What I've noticed is that on first run
winbind tries to use the high ports to set up part of the
communication. During that process, the winbindd process runs at 100%
CPU, and it takes about two minutes to time out. After that time
things settle down and winbind works perfectly.
For various reasons we're not eager to open a wide range of ports
across all the firewalls, and would like to know if there's a way in
smb.conf to skip the step that's hanging.
I've validated that opening the port in the below log (49155) allows
winbind to start cleanly and work without hanging for two minutes, but
this was only for testing purposes and the port had to be closed back
up after.
Here's loglevel 10 from where the thing happens on a Centos7 box
[2016/06/15 11:23:34.554465, 3, pid=6383, effective(0, 0), real(0,
0)] ../source3/lib/util_sock.c:636(open_socket_out_send)
Connecting to [redacted IP of DC on different segment] at port 49155
[2016/06/15 11:25:41.772517, 10, pid=6383, effective(0, 0), real(0,
0), class=winbind]
../source3/winbindd/winbindd_cm.c:2698(cm_connect_lsa_tcp)
cli_rpc_pipe_open_schannel_with_key failed: NT_STATUS_IO_TIMEOUT
[global]
workgroup = TEST
realm = TEST.EXAMPLE.COM
security = ads
idmap config * : range = 16777216-33554431
template shell = /bin/false
kerberos method = secrets and keytab
winbind use default domain = true
winbind offline logon = false
log file = /var/log/samba/%m.log
log level = 10
netbios name = TESTWB
server string = WB server
invalid users = root
socket options = TCP_NODELAY
winbind enum users = yes
winbind enum groups = yes
winbind max domain connections = 5
winbind max clients = 1000

[Jason Waters]

Have you tried using IPTABLES to REJECT that port. It will send something
back and then maybe the timeout will fail immediately instead of waiting
the timeout period? So on the local machine(that is running winbind) try

iptables -A OUTPUT -p tcp --dport 49155 -j REJECT
iptables -A INPUT -p tcp --dport 49155 -j REJECT

Might not need it in both, but since we are checking...

Jason

Post by Mike Ely
Bump.

Post by Mike Ely
Hi,
We've got a few layers of firewall between our DCs and a domain guest
providing winbind services. What I've noticed is that on first run winbind
tries to use the high ports to set up part of the communication. During
that process, the winbindd process runs at 100% CPU, and it takes about two
minutes to time out. After that time things settle down and winbind works
perfectly.
For various reasons we're not eager to open a wide range of ports across
all the firewalls, and would like to know if there's a way in smb.conf to
skip the step that's hanging.
I've validated that opening the port in the below log (49155) allows
winbind to start cleanly and work without hanging for two minutes, but this
was only for testing purposes and the port had to be closed back up after.
Here's loglevel 10 from where the thing happens on a Centos7 box running
[2016/06/15 11:23:34.554465, 3, pid=6383, effective(0, 0), real(0, 0)]
../source3/lib/util_sock.c:636(open_socket_out_send)
Connecting to [redacted IP of DC on different segment] at port 49155
[2016/06/15 11:25:41.772517, 10, pid=6383, effective(0, 0), real(0, 0),
class=winbind] ../source3/winbindd/winbindd_cm.c:2698(cm_connect_lsa_tcp)
cli_rpc_pipe_open_schannel_with_key failed: NT_STATUS_IO_TIMEOUT
[global]
workgroup = TEST
realm = TEST.EXAMPLE.COM
security = ads
idmap config * : range = 16777216-33554431
template shell = /bin/false
kerberos method = secrets and keytab
winbind use default domain = true
winbind offline logon = false
log file = /var/log/samba/%m.log
log level = 10
netbios name = TESTWB
server string = WB server
invalid users = root
socket options = TCP_NODELAY
winbind enum users = yes
winbind enum groups = yes
winbind max domain connections = 5
winbind max clients = 1000

--
To unsubscribe from this list go to the following URL and read the
instructions: https://lists.samba.org/mailman/options/samba

[Mike Ely]

Have you tried using IPTABLES to REJECT that port. It will send
something back and then maybe the timeout will fail immediately
instead of waiting the timeout period? So on the local machine(that
is running winbind) try
iptables -A OUTPUT -p tcp --dport 49155 -j REJECT
iptables -A INPUT -p tcp --dport 49155 -j REJECT
Might not need it in both, but since we are checking...

Good idea. The only problem is that port 49155 is one of those RPC
highports, so there's no guarantee that specific port will always be
used, and I wouldn't necessarily want to add an OUTPUT filter against
all highports.

[Jason Waters]

True, but would be interesting enough to test. You could edit the startup
script(winbind) to run the iptables command before it starts, and then
remove it after it is started?

Have you tried using IPTABLES to REJECT that port. It will send something
back and then maybe the timeout will fail immediately instead of waiting
the timeout period? So on the local machine(that is running winbind) try
iptables -A OUTPUT -p tcp --dport 49155 -j REJECT
iptables -A INPUT -p tcp --dport 49155 -j REJECT
Might not need it in both, but since we are checking...
Good idea. The only problem is that port 49155 is one of those RPC

highports, so there's no guarantee that specific port will always be used,
and I wouldn't necessarily want to add an OUTPUT filter against all
highports.
--
To unsubscribe from this list go to the following URL and read the
instructions: https://lists.samba.org/mailman/options/samba