[Samba] Winbind process stuck at 100% after changing use_mmap to no

[Alex Crow]

Hi,

We've had a strange issue after following the recommendations at
https://wiki.samba.org/index.php/Ping_pong, particularly the part about
mmap coherence. We are running CTDB/Samba over a MooseFS clustered FS,
and we'd not done the ping-pong before. After finding that the mmap
coherence test did not pass, we changed "use mmap" to "no" in smb.conf.

This morning as users began to access their drives and profiles,
performance tanked to such a degree that most people could not complete
loading profiles and even where successful, windows drive mappings were
taking minutes to complete per share.

When looking at the active CTDB servers, on each a single Winbindd
process was taking up 100% of CPU. After reverting the "use_mmap" to
yes, performance returned completely to normal.

We found this very odd as this is a recommended setting according to the
page above.

Here are package versions:

Installed Packages
Name : samba
Arch : x86_64
Version : 4.2.10
Release : 6.el7_2
Size : 1.8 M
Repo : installed
From repo : updates
Summary : Server and Client software to interoperate with Windows
machines
URL : http://www.samba.org/
Licence : GPLv3+ and LGPLv3+
Description : Samba is the standard Windows interoperability suite of
programs for Linux and Unix.

Installed Packages
Name : ctdb
Arch : x86_64
Version : 4.2.10
Release : 6.el7_2
Size : 1.2 M
Repo : installed
From repo : updates
Summary : A Clustered Database based on Samba's Trivial Database (TDB)
URL : http://www.samba.org/
Licence : GPLv3+ and LGPLv3+
Description : CTDB is a cluster implementation of the TDB database used
by Samba and other
: projects to store temporary data. If an application is
already using TDB for
: temporary data it is very easy to convert that
application to be cluster aware
: and use CTDB instead.

Linux zearing.ifa.net 3.10.0-327.18.2.el7.x86_64 #1 SMP Thu May 12
11:03:55 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Best regards

Alex
--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.

"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: [phone] Fax: [phone].
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).

[Alex Crow]

Post by Alex Crow
Hi,
We've had a strange issue after following the recommendations at
https://wiki.samba.org/index.php/Ping_pong, particularly the part
about mmap coherence. We are running CTDB/Samba over a MooseFS
clustered FS, and we'd not done the ping-pong before. After finding
that the mmap coherence test did not pass, we changed "use mmap" to
"no" in smb.conf.
This morning as users began to access their drives and profiles,
performance tanked to such a degree that most people could not
complete loading profiles and even where successful, windows drive
mappings were taking minutes to complete per share.
When looking at the active CTDB servers, on each a single Winbindd
process was taking up 100% of CPU. After reverting the "use_mmap" to
yes, performance returned completely to normal.
We found this very odd as this is a recommended setting according to
the page above.

Hmmm. I know very little about MooseFS. Did you get an strace
on the winbindd process to try and figure out what it was doing ?

I'm afraid not, the priority was to get people working and I was fixing
it remotely.

I may be able to try it on a test setup.

Thanks

Alex
--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.

"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: [phone] Fax: [phone].
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).

[Alex Crow]

Post by Alex Crow

Post by Alex Crow
Hi,
We've had a strange issue after following the recommendations at
https://wiki.samba.org/index.php/Ping_pong, particularly the part
about mmap coherence. We are running CTDB/Samba over a MooseFS
clustered FS, and we'd not done the ping-pong before. After finding
that the mmap coherence test did not pass, we changed "use mmap" to
"no" in smb.conf.
This morning as users began to access their drives and profiles,
performance tanked to such a degree that most people could not
complete loading profiles and even where successful, windows drive
mappings were taking minutes to complete per share.
When looking at the active CTDB servers, on each a single Winbindd
process was taking up 100% of CPU. After reverting the "use_mmap" to
yes, performance returned completely to normal.
We found this very odd as this is a recommended setting according to
the page above.

Hmmm. I know very little about MooseFS. Did you get an strace
on the winbindd process to try and figure out what it was doing ?

I'm afraid not, the priority was to get people working and I was fixing
it remotely.
I may be able to try it on a test setup.
Thanks
Alex

Hi,

I get endless preads on an strace, stalling occasionally:

Thanks

Alex

pread(29, "\0\0\0\0", 4, 5104) = 4
pread(29, "\0\0\0\0", 4, 5108) = 4
pread(29, "\0\0\0\0", 4, 5112) = 4
pread(29, "\0\0\0\0", 4, 5116) = 4
pread(29, "\0\0\0\0", 4, 5120) = 4
pread(29, "\0\0\0\0", 4, 5124) = 4
pread(29, "\0\0\0\0", 4, 5128) = 4
pread(29, "\0\0\0\0", 4, 5132) = 4
pread(29, "\0\0\0\0", 4, 5136) = 4
pread(29, "\0\0\0\0", 4, 5140) = 4
pread(29, "\0\0\0\0", 4, 5144) = 4
pread(29, "\0\0\0\0", 4, 5148) = 4
pread(29, "\0\0\0\0", 4, 5152) = 4
pread(29, "\0\0\0\0", 4, 5156) = 4
pread(29, "\0\0\0\0", 4, 5160) = 4
pread(29, "\0\0\0\0", 4, 5164) = 4
pread(29, "\0\0\0\0", 4, 5168) = 4
pread(29, "\0\0\0\0", 4, 5172) = 4
pread(29, "\0\0\0\0", 4, 5176) = 4
pread(29, "\0\0\0\0", 4, 5180) = 4
pread(29, "\0\0\0\0", 4, 5184) = 4
pread(29, "\0\0\0\0", 4, 5188) = 4
pread(29, "\0\0\0\0", 4, 5192) = 4
pread(29, "\0\0\0\0", 4, 5196) = 4
pread(29, "\0\0\0\0", 4, 5200) = 4
pread(29, "\0\0\0\0", 4, 5204) = 4
pread(29, "\0\0\0\0", 4, 5208) = 4
pread(29, "\0\0\0\0", 4, 5212) = 4
pread(29, "\0\0\0\0", 4, 5216) = 4
pread(29, "\0\0\0\0", 4, 5220) = 4
pread(29, "\0\0\0\0", 4, 5224) = 4
pread(29, "\0\0\0\0", 4, 5228) = 4
pread(29, "\0\0\0\0", 4, 5232) = 4
pread(29, "\0\0\0\0", 4, 5236) = 4
pread(29, "\0\0\0\0", 4, 5240) = 4
pread(29, "\0\0\0\0", 4, 5244) = 4
pread(29, "\0\0\0\0", 4, 5248) = 4
pread(29, "\0\0\0\0", 4, 5252) = 4
pread(29, "\0\0\0\0", 4, 5256) = 4
pread(29, "\0\0\0\0", 4, 5260) = 4
pread(29, "\0\0\0\0", 4, 5264) = 4
pread(29, "\0\0\0\0", 4, 5268) = 4
pread(29, "\0\0\0\0", 4, 5272) = 4
pread(29, "\0\0\0\0", 4, 5276) = 4
pread(29, "\0\0\0\0", 4, 5280) = 4
pread(29, "\0\0\0\0", 4, 5284) = 4
pread(29, "\0\0\0\0", 4, 5288) = 4
pread(29, "\0\0\0\0", 4, 5292) = 4
pread(29, "\0\0\0\0", 4, 5296) = 4
pread(29, "\0\0\0\0", 4, 5300) = 4
pread(29, "\0\0\0\0", 4, 5304) = 4
pread(29, "\0\0\0\0", 4, 5308) = 4
pread(29, "\0\0\0\0", 4, 5312) = 4
pread(29, "\0\0\0\0", 4, 5316) = 4
pread(29, "\0\0\0\0", 4, 5320) = 4
pread(29, "\0\0\0\0", 4, 5324) = 4
pread(29, "\0\0\0\0", 4, 5328) = 4
pread(29, "\0\0\0\0", 4, 5332) = 4
pread(29, "\0\0\0\0", 4, 5336) = 4
pread(29, "\0\0\0\0", 4, 5340) = 4
pread(29, "\0\0\0\0", 4, 5344) = 4
pread(29, "\0\0\0\0", 4, 5348) = 4
pread(29, "\0\0\0\0", 4, 5352) = 4
pread(29, "\0\0\0\0", 4, 5356) = 4
pread(29, "\0\0\0\0", 4, 5360) = 4
pread(29, "\0\0\0\0", 4, 5364) = 4
pread(29, "\0\0\0\0", 4, 5368) = 4
pread(29, "\0\0\0\0", 4, 5372) = 4
pread(29, "\0\0\0\0", 4, 5376) = 4
pread(29, "\0\0\0\0", 4, 5380) = 4
pread(29, "\0\0\0\0", 4, 5384) = 4
pread(29, "\0\0\0\0", 4, 5388) = 4
pread(29, "\0\0\0\0", 4, 5392) = 4
pread(29, "\0\0\0\0", 4, 5396) = 4
pread(29, "\0\0\0\0", 4, 5400) = 4
pread(29, "\0\0\0\0", 4, 5404) = 4
pread(29, "\0\0\0\0", 4, 5408) = 4
pread(29, "\0\0\0\0", 4, 5412) = 4
pread(29, "\0\0\0\0", 4, 5416) = 4
pread(29, "\0\0\0\0", 4, 5420) = 4
pread(29, "\0\0\0\0", 4, 5424) = 4
pread(29, "\0\0\0\0", 4, 5428) = 4
pread(29, "\0\0\0\0", 4, 5432) = 4
pread(29, "\0\0\0\0", 4, 5436) = 4
pread(29, "\0\0\0\0", 4, 5440) = 4
pread(29, "\0\0\0\0", 4, 5444) = 4
pread(29, "\0\0\0\0", 4, 5448) = 4
pread(29, "\0\0\0\0", 4, 5452) = 4
pread(29, "\0\0\0\0", 4, 5456) = 4
pread(29, "\0\0\0\0", 4, 5460) = 4
pread(29, "\0\0\0\0", 4, 5464) = 4
pread(29, "\0\0\0\0", 4, 5468) = 4
pread(29, "\0\0\0\0", 4, 5472) = 4
pread(29, "\0\0\0\0", 4, 5476) = 4
pread(29, "\0\0\0\0", 4, 5480) = 4
pread(29, "\0\0\0\0", 4, 5484) = 4
pread(29, "\0\0\0\0", 4, 5488) = 4
pread(29, "\0\0\0\0", 4, 5492) = 4
pread(29, "\0\0\0\0", 4, 5496) = 4
pread(29, "\0\0\0\0", 4, 5500) = 4
pread(29, "\0\0\0\0", 4, 5504) = 4
pread(29, "\0\0\0\0", 4, 5508) = 4
pread(29, "\0\0\0\0", 4, 5512) = 4
pread(29, "\0\0\0\0", 4, 5516) = 4
pread(29, "\0\0\0\0", 4, 5520) = 4
pread(29, "\0\0\0\0", 4, 5524) = 4
pread(29, "\0\0\0\0", 4, 5528) = 4
pread(29, "\0\0\0\0", 4, 5532) = 4
pread(29, "\0\0\0\0", 4, 5536) = 4
pread(29, "\0\0\0\0", 4, 5540) = 4
pread(29, "\0\0\0\0", 4, 5544) = 4
pread(29, "\0\0\0\0", 4, 5548) = 4
pread(29, "\0\0\0\0", 4, 5552) = 4
pread(29, "\0\0\0\0", 4, 5556) = 4
pread(29, "\0\0\0\0", 4, 5560) = 4
pread(29, "\0\0\0\0", 4, 5564) = 4
pread(29, "\0\0\0\0", 4, 5568) = 4
pread(29, "\0\0\0\0", 4, 5572) = 4
pread(29, "\0\0\0\0", 4, 5576) = 4
pread(29, "\0\0\0\0", 4, 5580) = 4
pread(29, "\0\0\0\0", 4, 5584) = 4
pread(29, "\0\0\0\0", 4, 5588) = 4
pread(29, "\0\0\0\0", 4, 5592) = 4
pread(29, "\0\0\0\0", 4, 5596) = 4
pread(29, "\0\0\0\0", 4, 5600) = 4
pread(29, "\0\0\0\0", 4, 5604) = 4
pread(29, "\0\0\0\0", 4, 5608) = 4
pread(29, "\0\0\0\0", 4, 5612) = 4
pread(29, "\0\0\0\0", 4, 5616) = 4
pread(29, "\0\0\0\0", 4, 5620) = 4
pread(29, "\0\0\0\0", 4, 5624) = 4
pread(29, "\0\0\0\0", 4, 5628) = 4
pread(29, "\0\0\0\0", 4, 5632) = 4
pread(29, "\0\0\0\0", 4, 5636) = 4
pread(29, "\0\0\0\0", 4, 5640) = 4
pread(29, "\0\0\0\0", 4, 5644) = 4
pread(29, "\0\0\0\0", 4, 5648) = 4
pread(29, "\0\0\0\0", 4, 5652) = 4
pread(29, "\0\0\0\0", 4, 5656) = 4
pread(29, "\0\0\0\0", 4, 5660) = 4
pread(29, "\0\0\0\0", 4, 5664) = 4
pread(29, "\0\0\0\0", 4, 5668) = 4
pread(29, "\0\0\0\0", 4, 5672) = 4
pread(29, "\0\0\0\0", 4, 5676) = 4
pread(29, "\0\0\0\0", 4, 5680) = 4
pread(29, "\0\0\0\0", 4, 5684) = 4
pread(29, "\0\0\0\0", 4, 5688) = 4
pread(29, "\0\0\0\0", 4, 5692) = 4
pread(29, "\0\0\0\0", 4, 5696) = 4
pread(29, "\0\0\0\0", 4, 5700) = 4
pread(29, "\0\0\0\0", 4, 5704) = 4
pread(29, "\0\0\0\0", 4, 5708) = 4
pread(29, "\0\0\0\0", 4, 5712) = 4
pread(29, "\0\0\0\0", 4, 5716) = 4
pread(29, "\0\0\0\0", 4, 5720) = 4
pread(29, "\0\0\0\0", 4, 5724) = 4
pread(29, "\0\0\0\0", 4, 5728) = 4
pread(29, "\0\0\0\0", 4, 5732) = 4
pread(29, "\0\0\0\0", 4, 5736) = 4
pread(29, "\0\0\0\0", 4, 5740) = 4
pread(29, "\0\0\0\0", 4, 5744) = 4
pread(29, "\0\0\0\0", 4, 5748) = 4
pread(29, "\0\0\0\0", 4, 5752) = 4
pread(29, "\0\0\0\0", 4, 5756) = 4
pread(29, "\0\0\0\0", 4, 5760) = 4
pread(29, "\0\0\0\0", 4, 5764) = 4
pread(29, "\0\0\0\0", 4, 5768) = 4
pread(29, "\0\0\0\0", 4, 5772) = 4
pread(29, "\0\0\0\0", 4, 5776) = 4
pread(29, "\0\0\0\0", 4, 5780) = 4
pread(29, "\0\0\0\0", 4, 5784) = 4
pread(29, "\0\0\0\0", 4, 5788) = 4
pread(29, "\0\0\0\0", 4, 5792) = 4
pread(29, "\0\0\0\0", 4, 5796) = 4
pread(29, "\0\0\0\0", 4, 5800) = 4
pread(29, "\0\0\0\0", 4, 5804) = 4
pread(29, "\0\0\0\0", 4, 5808) = 4
pread(29, "\0\0\0\0", 4, 5812) = 4
pread(29, "\0\0\0\0", 4, 5816) = 4
pread(29, "\0\0\0\0", 4, 5820) = 4
pread(29, "\0\0\0\0", 4, 5824) = 4
pread(29, "\0\0\0\0", 4, 5828) = 4
pread(29, "\0\0\0\0", 4, 5832) = 4
pread(29, "\0\0\0\0", 4, 5836) = 4
pread(29, "\0\0\0\0", 4, 5840) = 4
pread(29, "\0\0\0\0", 4, 5844) = 4
pread(29, "\0\0\0\0", 4, 5848) = 4
pread(29, "\0\0\0\0", 4, 5852) = 4
pread(29, "\0\0\0\0", 4, 5856) = 4
pread(29, "\0\0\0\0", 4, 5860) = 4
pread(29, "\0\0\0\0", 4, 5864) = 4
pread(29, "\0\0\0\0", 4, 5868) = 4
pread(29, "\0\0\0\0", 4, 5872) = 4
pread(29, "\0\0\0\0", 4, 5876) = 4
pread(29, "\0\0\0\0", 4, 5880) = 4
pread(29, "\0\0\0\0", 4, 5884) = 4
pread(29, "\0\0\0\0", 4, 5888) = 4
pread(29, "\0\0\0\0", 4, 5892) = 4
pread(29, "\0\0\0\0", 4, 5896) = 4
pread(29, "\0\0\0\0", 4, 5900) = 4
pread(29, "\0\0\0\0", 4, 5904) = 4
pread(29, "\0\0\0\0", 4, 5908) = 4
pread(29, "\0\0\0\0", 4, 5912) = 4
pread(29, "\0\0\0\0", 4, 5916) = 4
pread(29, "\0\0\0\0", 4, 5920) = 4
pread(29, "\0\0\0\0", 4, 5924) = 4
pread(29, "\0\0\0\0", 4, 5928) = 4
pread(29, "\0\0\0\0", 4, 5932) = 4
pread(29, "\0\0\0\0", 4, 5936) = 4
pread(29, "\0\0\0\0", 4, 5940) = 4
pread(29, "\0\0\0\0", 4, 5944) = 4
pread(29, "\0\0\0\0", 4, 5948) = 4
pread(29, "\0\0\0\0", 4, 5952) = 4
pread(29, "\0\0\0\0", 4, 5956) = 4
pread(29, "\0\0\0\0", 4, 5960) = 4
pread(29, "\0\0\0\0", 4, 5964) = 4
pread(29, "\0\0\0\0", 4, 5968) = 4
pread(29, "\0\0\0\0", 4, 5972) = 4
pread(29, "\0\0\0\0", 4, 5976) = 4
pread(29, "\0\0\0\0", 4, 5980) = 4
pread(29, "\0\0\0\0", 4, 5984) = 4
pread(29, "\0\0\0\0", 4, 5988) = 4
pread(29, "\0\0\0\0", 4, 5992) = 4
pread(29, "\0\0\0\0", 4, 5996) = 4
pread(29, "\0\0\0\0", 4, 6000) = 4
pread(29, "\0\0\0\0", 4, 6004) = 4
pread(29, "\0\0\0\0", 4, 6008) = 4
pread(29, "\0\0\0\0", 4, 6012) = 4
pread(29, "\0\0\0\0", 4, 6016) = 4
pread(29, "\0\0\0\0", 4, 6020) = 4
pread(29, "\0\0\0\0", 4, 6024) = 4
pread(29, "\0\0\0\0", 4, 6028) = 4
pread(29, "\0\0\0\0", 4, 6032) = 4
pread(29, "\0\0\0\0", 4, 6036) = 4
pread(29, "\0\0\0\0", 4, 6040) = 4
pread(29, "\0\0\0\0", 4, 6044) = 4
pread(29, "\0\0\0\0", 4, 6048) = 4
pread(29, "\0\0\0\0", 4, 6052) = 4
pread(29, "\0\0\0\0", 4, 6056) = 4
pread(29, "\0\0\0\0", 4, 6060) = 4
pread(29, "\0\0\0\0", 4, 6064) = 4
pread(29, "\0\0\0\0", 4, 6068) = 4
pread(29, "\0\0\0\0", 4, 6072) = 4
pread(29, "\0\0\0\0", 4, 6076) = 4
pread(29, "\0\0\0\0", 4, 6080) = 4
pread(29, "\0\0\0\0", 4, 6084) = 4
pread(29, "\0\0\0\0", 4, 6088) = 4
pread(29, "\0\0\0\0", 4, 6092) = 4
pread(29, "\0\0\0\0", 4, 6096) = 4
pread(29, "\0\0\0\0", 4, 6100) = 4
pread(29, "\0\0\0\0", 4, 6104) = 4
pread(29, "\0\0\0\0", 4, 6108) = 4
pread(29, "\0\0\0\0", 4, 6112) = 4
pread(29, "\0\0\0\0", 4, 6116) = 4
pread(29, "\0\0\0\0", 4, 6120) = 4
pread(29, "\0\0\0\0", 4, 6124) = 4
pread(29, "\0\0\0\0", 4, 6128) = 4
pread(29, "\0\0\0\0", 4, 6132) = 4
pread(29, "\0\0\0\0", 4, 6136) = 4
pread(29, "\0\0\0\0", 4, 6140) = 4
pread(29, "\0\0\0\0", 4, 6144) = 4
pread(29, "\0\0\0\0", 4, 6148) = 4
pread(29, "\0\0\0\0", 4, 6152) = 4
pread(29, "\0\0\0\0", 4, 6156) = 4
pread(29, "\0\0\0\0", 4, 6160) = 4
pread(29, "\0\0\0\0", 4, 6164) = 4
pread(29, "\0\0\0\0", 4, 6168) = 4
pread(29, "\0\0\0\0", 4, 6172) = 4
pread(29, "\0\0\0\0", 4, 6176) = 4
pread(29, "\0\0\0\0", 4, 6180) = 4
pread(29, "\0\0\0\0", 4, 6184) = 4
pread(29, "\0\0\0\0", 4, 6188) = 4
pread(29, "\0\0\0\0", 4, 6192) = 4
pread(29, "\0\0\0\0", 4, 6196) = 4
pread(29, "\0\0\0\0", 4, 6200) = 4
pread(29, "\0\0\0\0", 4, 6204) = 4
pread(29, "\0\0\0\0", 4, 6208) = 4
pread(29, "\0\0\0\0", 4, 6212) = 4
pread(29, "\0\0\0\0", 4, 6216) = 4
pread(29, "\0\0\0\0", 4, 6220) = 4
pread(29, "\0\0\0\0", 4, 6224) = 4
pread(29, "\0\0\0\0", 4, 6228) = 4
pread(29, "\0\0\0\0", 4, 6232) = 4
pread(29, "\0\0\0\0", 4, 6236) = 4
pread(29, "\0\0\0\0", 4, 6240) = 4
pread(29, "\0\0\0\0", 4, 6244) = 4
pread(29, "\0\0\0\0", 4, 6248) = 4
pread(29, "\0\0\0\0", 4, 6252) = 4
pread(29, "\0\0\0\0", 4, 6256) = 4
pread(29, "\0\0\0\0", 4, 6260) = 4
pread(29, "\0\0\0\0", 4, 6264) = 4
pread(29, "\0\0\0\0", 4, 6268) = 4
pread(29, "\0\0\0\0", 4, 6272) = 4
pread(29, "\0\0\0\0", 4, 6276) = 4
pread(29, "\0\0\0\0", 4, 6280) = 4
pread(29, "\0\0\0\0", 4, 6284) = 4
pread(29, "\0\0\0\0", 4, 6288) = 4
pread(29, "\0\0\0\0", 4, 6292) = 4
pread(29, "\0\0\0\0", 4, 6296) = 4
pread(29, "\0\0\0\0", 4, 6300) = 4
pread(29, "\0\0\0\0", 4, 6304) = 4
pread(29, "\0\0\0\0", 4, 6308) = 4
pread(29, "\0\0\0\0", 4, 6312) = 4
pread(29, "\0\0\0\0", 4, 6316) = 4
pread(29, "\0\0\0\0", 4, 6320) = 4
pread(29, "\0\0\0\0", 4, 6324) = 4
pread(29, "\0\0\0\0", 4, 6328) = 4
pread(29, "\0\0\0\0", 4, 6332) = 4
pread(29, "\0\0\0\0", 4, 6336) = 4
pread(29, "\0\0\0\0", 4, 6340) = 4
pread(29, "\0\0\0\0", 4, 6344) = 4
pread(29, "\0\0\0\0", 4, 6348) = 4
pread(29, "\0\0\0\0", 4, 6352) = 4
pread(29, "\0\0\0\0", 4, 6356) = 4
pread(29, "\0\0\0\0", 4, 6360) = 4
pread(29, "\0\0\0\0", 4, 6364) = 4
pread(29, "\0\0\0\0", 4, 6368) = 4
pread(29, "\0\0\0\0", 4, 6372) = 4
pread(29, "\0\0\0\0", 4, 6376) = 4
pread(29, "\0\0\0\0", 4, 6380) = 4
pread(29, "\0\0\0\0", 4, 6384) = 4
pread(29, "\0\0\0\0", 4, 6388) = 4
pread(29, "\0\0\0\0", 4, 6392) = 4
pread(29, "\0\0\0\0", 4, 6396) = 4
pread(29, "\0\0\0\0", 4, 6400) = 4
pread(29, "\0\0\0\0", 4, 6404) = 4
pread(29, "\0\0\0\0", 4, 6408) = 4
pread(29, "\0\0\0\0", 4, 6412) = 4
pread(29, "\0\0\0\0", 4, 6416) = 4
pread(29, "\0\0\0\0", 4, 6420) = 4
pread(29, "\0\0\0\0", 4, 6424) = 4
pread(29, "\0\0\0\0", 4, 6428) = 4
pread(29, "\0\0\0\0", 4, 6432) = 4
pread(29, "\0\0\0\0", 4, 6436) = 4
pread(29, "\0\0\0\0", 4, 6440) = 4
pread(29, "\0\0\0\0", 4, 6444) = 4
pread(29, "\0\0\0\0", 4, 6448) = 4
pread(29, "\0\0\0\0", 4, 6452) = 4
pread(29, "\0\0\0\0", 4, 6456) = 4
pread(29, "\0\0\0\0", 4, 6460) = 4
pread(29, "\0\0\0\0", 4, 6464) = 4
pread(29, "\0\0\0\0", 4, 6468) = 4
pread(29, "\0\0\0\0", 4, 6472) = 4
pread(29, "\0\0\0\0", 4, 6476) = 4
pread(29, "\0\0\0\0", 4, 6480) = 4
pread(29, "\0\0\0\0", 4, 6484) = 4
pread(29, "\0\0\0\0", 4, 6488) = 4
pread(29, "\0\0\0\0", 4, 6492) = 4
pread(29, "\0\0\0\0", 4, 6496) = 4
pread(29, "\0\0\0\0", 4, 6500) = 4
pread(29, "\0\0\0\0", 4, 6504) = 4
pread(29, "\0\0\0\0", 4, 6508) = 4
pread(29, "\0\0\0\0", 4, 6512) = 4
pread(29, "\0\0\0\0", 4, 6516) = 4
pread(29, "\0\0\0\0", 4, 6520) = 4
pread(29, "\0\0\0\0", 4, 6524) = 4
pread(29, "\0\0\0\0", 4, 6528) = 4
pread(29, "\0\0\0\0", 4, 6532) = 4
pread(29, "\0\0\0\0", 4, 6536) = 4
pread(29, "\0\0\0\0", 4, 6540) = 4
pread(29, "\0\0\0\0", 4, 6544) = 4
pread(29, "\0\0\0\0", 4, 6548) = 4
pread(29, "\0\0\0\0", 4, 6552) = 4
pread(29, "\0\0\0\0", 4, 6556) = 4
pread(29, "\0\0\0\0", 4, 6560) = 4
pread(29, "\0\0\0\0", 4, 6564) = 4
pread(29, "\0\0\0\0", 4, 6568) = 4
pread(29, "\0\0\0\0", 4, 6572) = 4
pread(29, "\0\0\0\0", 4, 6576) = 4
pread(29, "\0\0\0\0", 4, 6580) = 4
pread(29, "\0\0\0\0", 4, 6584) = 4
pread(29, "\0\0\0\0", 4, 6588) = 4
pread(29, "\0\0\0\0", 4, 6592) = 4
pread(29, "\0\0\0\0", 4, 6596) = 4
pread(29, "\0\0\0\0", 4, 6600) = 4
pread(29, "\0\0\0\0", 4, 6604) = 4
pread(29, "\0\0\0\0", 4, 6608) = 4
pread(29, "\0\0\0\0", 4, 6612) = 4
pread(29, "\0\0\0\0", 4, 6616) = 4
pread(29, "\0\0\0\0", 4, 6620) = 4
pread(29, "\0\0\0\0", 4, 6624) = 4
pread(29, "\0\0\0\0", 4, 6628) = 4
pread(29, "\0\0\0\0", 4, 6632) = 4
pread(29, "\0\0\0\0", 4, 6636) = 4
pread(29, "\0\0\0\0", 4, 6640) = 4
pread(29, "\0\0\0\0", 4, 6644) = 4
pread(29, "\0\0\0\0", 4, 6648) = 4
pread(29, "\0\0\0\0", 4, 6652) = 4
pread(29, "\0\0\0\0", 4, 6656) = 4
pread(29, "\0\0\0\0", 4, 6660) = 4
pread(29, "\0\0\0\0", 4, 6664) = 4
pread(29, "\0\0\0\0", 4, 6668) = 4
pread(29, "\0\0\0\0", 4, 6672) = 4
pread(29, "\0\0\0\0", 4, 6676) = 4
pread(29, "\0\0\0\0", 4, 6680) = 4
pread(29, "\0\0\0\0", 4, 6684) = 4
pread(29, "\0\0\0\0", 4, 6688) = 4
pread(29, "\0\0\0\0", 4, 6692) = 4
pread(29, "\0\0\0\0", 4, 6696) = 4
pread(29, "\0\0\0\0", 4, 6700) = 4
pread(29, "\0\0\0\0", 4, 6704) = 4
pread(29, "\0\0\0\0", 4, 6708) = 4
pread(29, "\0\0\0\0", 4, 6712) = 4
pread(29, "\0\0\0\0", 4, 6716) = 4
pread(29, "\0\0\0\0", 4, 6720) = 4
pread(29, "\0\0\0\0", 4, 6724) = 4
pread(29, "\0\0\0\0", 4, 6728) = 4
pread(29, "\0\0\0\0", 4, 6732) = 4
pread(29, "\0\0\0\0", 4, 6736) = 4
pread(29, "\0\0\0\0", 4, 6740) = 4
pread(29, "\0\0\0\0", 4, 6744) = 4
pread(29, "\0\0\0\0", 4, 6748) = 4
pread(29, "\0\0\0\0", 4, 6752) = 4
pread(29, "\0\0\0\0", 4, 6756) = 4
pread(29, "\0\0\0\0", 4, 6760) = 4
pread(29, "\0\0\0\0", 4, 6764) = 4
pread(29, "\0\0\0\0", 4, 6768) = 4
pread(29, "\0\0\0\0", 4, 6772) = 4
pread(29, "\0\0\0\0", 4, 6776) = 4
pread(29, "\0\0\0\0", 4, 6780) = 4
pread(29, "\0\0\0\0", 4, 6784) = 4
pread(29, "\0\0\0\0", 4, 6788) = 4
pread(29, "\0\0\0\0", 4, 6792) = 4
pread(29, "\0\0\0\0", 4, 6796) = 4
pread(29, "\0\0\0\0", 4, 6800) = 4
pread(29, "\0\0\0\0", 4, 6804) = 4
pread(29, "\0\0\0\0", 4, 6808) = 4
pread(29, "\0\0\0\0", 4, 6812) = 4
pread(29, "\0\0\0\0", 4, 6816) = 4
pread(29, "\0\0\0\0", 4, 6820) = 4
pread(29, "\0\0\0\0", 4, 6824) = 4
pread(29, "\0\0\0\0", 4, 6828) = 4
pread(29, "\0\0\0\0", 4, 6832) = 4
pread(29, "\0\0\0\0", 4, 6836) = 4
pread(29, "\0\0\0\0", 4, 6840) = 4
pread(29, "\0\0\0\0", 4, 6844) = 4
pread(29, "\0\0\0\0", 4, 6848) = 4
pread(29, "\0\0\0\0", 4, 6852) = 4
pread(29, "\0\0\0\0", 4, 6856) = 4
pread(29, "\0\0\0\0", 4, 6860) = 4
pread(29, "\0\0\0\0", 4, 6864) = 4
pread(29, "\0\0\0\0", 4, 6868) = 4
pread(29, "\0\0\0\0", 4, 6872) = 4
pread(29, "\0\0\0\0", 4, 6876) = 4
pread(29, "\0\0\0\0", 4, 6880) = 4
pread(29, "\0\0\0\0", 4, 6884) = 4
pread(29, "\0\0\0\0", 4, 6888) = 4
pread(29, "\0\0\0\0", 4, 6892) = 4
pread(29, "\0\0\0\0", 4, 6896) = 4
pread(29, "\0\0\0\0", 4, 6900) = 4
pread(29, "\0\0\0\0", 4, 6904) = 4
pread(29, "\0\0\0\0", 4, 6908) = 4
pread(29, "\0\0\0\0", 4, 6912) = 4
pread(29, "\0\0\0\0", 4, 6916) = 4
pread(29, "\0\0\0\0", 4, 6920) = 4
pread(29, "\0\0\0\0", 4, 6924) = 4
pread(29, "\0\0\0\0", 4, 6928) = 4
pread(29, "\0\0\0\0", 4, 6932) = 4
pread(29, "\0\0\0\0", 4, 6936) = 4
pread(29, "\0\0\0\0", 4, 6940) = 4
pread(29, "\0\0\0\0", 4, 6944) = 4
pread(29, "\0\0\0\0", 4, 6948) = 4
pread(29, "\0\0\0\0", 4, 6952) = 4
pread(29, "\0\0\0\0", 4, 6956) = 4
pread(29, "\0\0\0\0", 4, 6960) = 4
pread(29, "\0\0\0\0", 4, 6964) = 4
pread(29, "\0\0\0\0", 4, 6968) = 4
pread(29, "\0\0\0\0", 4, 6972) = 4
pread(29, "\0\0\0\0", 4, 6976) = 4
pread(29, "\0\0\0\0", 4, 6980) = 4
pread(29, "\0\0\0\0", 4, 6984) = 4
pread(29, "\0\0\0\0", 4, 6988) = 4
pread(29, "\0\0\0\0", 4, 6992) = 4
pread(29, "\0\0\0\0", 4, 6996) = 4
pread(29, "\0\0\0\0", 4, 7000) = 4
pread(29, "\0\0\0\0", 4, 7004) = 4
pread(29, "\0\0\0\0", 4, 7008) = 4
pread(29, "\0\0\0\0", 4, 7012) = 4
pread(29, "\0\0\0\0", 4, 7016) = 4
pread(29, "\0\0\0\0", 4, 7020) = 4
pread(29, "\0\0\0\0", 4, 7024) = 4
pread(29, "\0\0\0\0", 4, 7028) = 4
pread(29, "\0\0\0\0", 4, 7032) = 4
pread(29, "\0\0\0\0", 4, 7036) = 4
pread(29, "\0\0\0\0", 4, 7040) = 4
pread(29, "\0\0\0\0", 4, 7044) = 4
pread(29, "\0\0\0\0", 4, 7048) = 4
pread(29, "\0\0\0\0", 4, 7052) = 4
pread(29, "\0\0\0\0", 4, 7056) = 4
pread(29, "\0\0\0\0", 4, 7060) = 4
pread(29, "\0\0\0\0", 4, 7064) = 4
pread(29, "\0\0\0\0", 4, 7068) = 4
pread(29, "\0\0\0\0", 4, 7072) = 4
pread(29, "\0\0\0\0", 4, 7076) = 4
pread(29, "\0\0\0\0", 4, 7080) = 4
pread(29, "\0\0\0\0", 4, 7084) = 4
pread(29, "\0\0\0\0", 4, 7088) = 4
pread(29, "\0\0\0\0", 4, 7092) = 4
pread(29, "\0\0\0\0", 4, 7096) = 4
pread(29, "\0\0\0\0", 4, 7100) = 4
pread(29, "\0\0\0\0", 4, 7104) = 4
pread(29, "\0\0\0\0", 4, 7108) = 4
pread(29, "\0\0\0\0", 4, 7112) = 4
pread(29, "\0\0\0\0", 4, 7116) = 4
pread(29, "\0\0\0\0", 4, 7120) = 4
pread(29, "\0\0\0\0", 4, 7124) = 4
pread(29, "\0\0\0\0", 4, 7128) = 4
pread(29, "\0\0\0\0", 4, 7132) = 4
pread(29, "\0\0\0\0", 4, 7136) = 4
pread(29, "\0\0\0\0", 4, 7140) = 4
pread(29, "\0\0\0\0", 4, 7144) = 4
pread(29, "\0\0\0\0", 4, 7148) = 4
pread(29, "\0\0\0\0", 4, 7152) = 4
pread(29, "\0\0\0\0", 4, 7156) = 4
pread(29, "\0\0\0\0", 4, 7160) = 4
pread(29, "\0\0\0\0", 4, 7164) = 4
pread(29, "\0\0\0\0", 4, 7168) = 4
pread(29, "\0\0\0\0", 4, 7172) = 4
pread(29, "\0\0\0\0", 4, 7176) = 4
pread(29, "\0\0\0\0", 4, 7180) = 4
pread(29, "\0\0\0\0", 4, 7184) = 4
pread(29, "\0\0\0\0", 4, 7188) = 4
pread(29, "\0\0\0\0", 4, 7192) = 4
pread(29, "\0\0\0\0", 4, 7196) = 4
pread(29, "\0\0\0\0", 4, 7200) = 4
pread(29, "\0\0\0\0", 4, 7204) = 4
pread(29, "\0\0\0\0", 4, 7208) = 4
pread(29, "\0\0\0\0", 4, 7212) = 4
pread(29, "\0\0\0\0", 4, 7216) = 4
pread(29, "\0\0\0\0", 4, 7220) = 4
pread(29, "\0\0\0\0", 4, 7224) = 4
pread(29, "\0\0\0\0", 4, 7228) = 4
pread(29, "\0\0\0\0", 4, 7232) = 4
pread(29, "\0\0\0\0", 4, 7236) = 4
pread(29, "\0\0\0\0", 4, 7240) = 4
pread(29, "\0\0\0\0", 4, 7244) = 4
pread(29, "\0\0\0\0", 4, 7248) = 4
pread(29, "\0\0\0\0", 4, 7252) = 4
pread(29, "\0\0\0\0", 4, 7256) = 4
pread(29, "\0\0\0\0", 4, 7260) = 4
pread(29, "\0\0\0\0", 4, 7264) = 4
pread(29, "\0\0\0\0", 4, 7268) = 4
pread(29, "\0\0\0\0", 4, 7272) = 4
pread(29, "\0\0\0\0", 4, 7276) = 4
pread(29, "\0\0\0\0", 4, 7280) = 4
pread(29, "\0\0\0\0", 4, 7284) = 4
pread(29, "\0\0\0\0", 4, 7288) = 4
pread(29, "\0\0\0\0", 4, 7292) = 4
pread(29, "\0\0\0\0", 4, 7296) = 4
pread(29, "\0\0\0\0", 4, 7300) = 4
pread(29, "\0\0\0\0", 4, 7304) = 4
pread(29, "\0\0\0\0", 4, 7308) = 4
pread(29, "\0\0\0\0", 4, 7312) = 4
pread(29, "\0\0\0\0", 4, 7316) = 4
pread(29, "\0\0\0\0", 4, 7320) = 4
pread(29, "\0\0\0\0", 4, 7324) = 4
pread(29, "\0\0\0\0", 4, 7328) = 4
pread(29, "\0\0\0\0", 4, 7332) = 4
pread(29, "\0\0\0\0", 4, 7336) = 4
pread(29, "\0\0\0\0", 4, 7340) = 4
pread(29, "\0\0\0\0", 4, 7344) = 4
pread(29, "\0\0\0\0", 4, 7348) = 4
pread(29, "\0\0\0\0", 4, 7352) = 4
pread(29, "\0\0\0\0", 4, 7356) = 4
pread(29, "\0\0\0\0", 4, 7360) = 4
pread(29, "\0\0\0\0", 4, 7364) = 4
pread(29, "\0\0\0\0", 4, 7368) = 4
pread(29, "\0\0\0\0", 4, 7372) = 4
pread(29, "\0\0\0\0", 4, 7376) = 4
pread(29, "\0\0\0\0", 4, 7380) = 4
pread(29, "\0\0\0\0", 4, 7384) = 4
pread(29, "\0\0\0\0", 4, 7388) = 4
pread(29, "\0\0\0\0", 4, 7392) = 4
pread(29, "\0\0\0\0", 4, 7396) = 4
pread(29, "\0\0\0\0", 4, 7400) = 4
pread(29, "\0\0\0\0", 4, 7404) = 4
pread(29, "\0\0\0\0", 4, 7408) = 4
pread(29, "\0\0\0\0", 4, 7412) = 4
pread(29, "\0\0\0\0", 4, 7416) = 4
pread(29, "\0\0\0\0", 4, 7420) = 4
pread(29, "\0\0\0\0", 4, 7424) = 4
pread(29, "\0\0\0\0", 4, 7428) = 4
pread(29, "\0\0\0\0", 4, 7432) = 4
pread(29, "\0\0\0\0", 4, 7436) = 4
pread(29, "\0\0\0\0", 4, 7440) = 4
pread(29, "\0\0\0\0", 4, 7444) = 4
pread(29, "\0\0\0\0", 4, 7448) = 4
pread(29, "\0\0\0\0", 4, 7452) = 4
pread(29, "\0\0\0\0", 4, 7456) = 4
pread(29, "\0\0\0\0", 4, 7460) = 4
pread(29, "\0\0\0\0", 4, 7464) = 4
pread(29, "\0\0\0\0", 4, 7468) = 4
pread(29, "\0\0\0\0", 4, 7472) = 4
pread(29, "\0\0\0\0", 4, 7476) = 4
pread(29, "\0\0\0\0", 4, 7480) = 4
pread(29, "\0\0\0\0", 4, 7484) = 4
pread(29, "\0\0\0\0", 4, 7488) = 4
pread(29, "\0\0\0\0", 4, 7492) = 4
pread(29, "\0\0\0\0", 4, 7496) = 4
pread(29, "\0\0\0\0", 4, 7500) = 4
pread(29, "\0\0\0\0", 4, 7504) = 4
pread(29, "\0\0\0\0", 4, 7508) = 4
pread(29, "\0\0\0\0", 4, 7512) = 4
pread(29, "\0\0\0\0", 4, 7516) = 4
pread(29, "\0\0\0\0", 4, 7520) = 4
pread(29, "\0\0\0\0", 4, 7524) = 4
pread(29, "\0\0\0\0", 4, 7528) = 4
pread(29, "\0\0\0\0", 4, 7532) = 4
pread(29, "\0\0\0\0", 4, 7536) = 4
pread(29, "\0\0\0\0", 4, 7540) = 4
pread(29, "\0\0\0\0", 4, 7544) = 4
pread(29, "\0\0\0\0", 4, 7548) = 4
pread(29, "\0\0\0\0", 4, 7552) = 4
pread(29, "\0\0\0\0", 4, 7556) = 4
pread(29, "\0\0\0\0", 4, 7560) = 4
pread(29, "\0\0\0\0", 4, 7564) = 4
pread(29, "\0\0\0\0", 4, 7568) = 4
pread(29, "\0\0\0\0", 4, 7572) = 4
pread(29, "\0\0\0\0", 4, 7576) = 4
pread(29, "\0\0\0\0", 4, 7580) = 4
pread(29, "\0\0\0\0", 4, 7584) = 4
pread(29, "\0\0\0\0", 4, 7588) = 4
pread(29, "\0\0\0\0", 4, 7592) = 4
pread(29, "\0\0\0\0", 4, 7596) = 4
pread(29, "\0\0\0\0", 4, 7600) = 4
pread(29, "\0\0\0\0", 4, 7604) = 4
pread(29, "\0\0\0\0", 4, 7608) = 4
pread(29, "\0\0\0\0", 4, 7612) = 4
pread(29, "\0\0\0\0", 4, 7616) = 4
pread(29, "\0\0\0\0", 4, 7620) = 4
pread(29, "\0\0\0\0", 4, 7624) = 4
pread(29, "\0\0\0\0", 4, 7628) = 4
pread(29, "\0\0\0\0", 4, 7632) = 4
pread(29, "\0\0\0\0", 4, 7636) = 4
pread(29, "\0\0\0\0", 4, 7640) = 4
pread(29, "\0\0\0\0", 4, 7644) = 4
pread(29, "\0\0\0\0", 4, 7648) = 4
pread(29, "\0\0\0\0", 4, 7652) = 4
pread(29, "\0\0\0\0", 4, 7656) = 4
pread(29, "\0\0\0\0", 4, 7660) = 4
pread(29, "\0\0\0\0", 4, 7664) = 4
pread(29, "\0\0\0\0", 4, 7668) = 4
pread(29, "\0\0\0\0", 4, 7672) = 4
pread(29, "\0\0\0\0", 4, 7676) = 4
pread(29, "\0\0\0\0", 4, 7680) = 4
pread(29, "\0\0\0\0", 4, 7684) = 4
pread(29, "\0\0\0\0", 4, 7688) = 4
pread(29, "\0\0\0\0", 4, 7692) = 4
pread(29, "\0\0\0\0", 4, 7696) = 4
pread(29, "\0\0\0\0", 4, 7700) = 4
pread(29, "\0\0\0\0", 4, 7704) = 4
pread(29, "\0\0\0\0", 4, 7708) = 4
pread(29, "\0\0\0\0", 4, 7712) = 4
pread(29, "\0\0\0\0", 4, 7716) = 4
pread(29, "\0\0\0\0", 4, 7720) = 4
pread(29, "\0\0\0\0", 4, 7724) = 4
pread(29, "\0\0\0\0", 4, 7728) = 4
pread(29, "\0\0\0\0", 4, 7732) = 4
pread(29, "\0\0\0\0", 4, 7736) = 4
pread(29, "\0\0\0\0", 4, 7740) = 4
pread(29, "\0\0\0\0", 4, 7744) = 4
pread(29, "\0\0\0\0", 4, 7748) = 4
pread(29, "\0\0\0\0", 4, 7752) = 4
pread(29, "\0\0\0\0", 4, 7756) = 4
pread(29, "\0\0\0\0", 4, 7760) = 4
pread(29, "\0\0\0\0", 4, 7764) = 4
pread(29, "\0\0\0\0", 4, 7768) = 4
pread(29, "\0\0\0\0", 4, 7772) = 4
pread(29, "\0\0\0\0", 4, 7776) = 4
pread(29, "\0\0\0\0", 4, 7780) = 4
pread(29, "\0\0\0\0", 4, 7784) = 4
pread(29, "\0\0\0\0", 4, 7788) = 4
pread(29, "\0\0\0\0", 4, 7792) = 4
pread(29, "\0\0\0\0", 4, 7796) = 4
pread(29, "\0\0\0\0", 4, 7800) = 4
pread(29, "\0\0\0\0", 4, 7804) = 4
pread(29, "\0\0\0\0", 4, 7808) = 4
pread(29, "\0\0\0\0", 4, 7812) = 4
pread(29, "\0\0\0\0", 4, 7816) = 4
pread(29, "\0\0\0\0", 4, 7820) = 4
pread(29, "\0\0\0\0", 4, 7824) = 4
pread(29, "\0\0\0\0", 4, 7828) = 4
pread(29, "\0\0\0\0", 4, 7832) = 4
pread(29, "\0\0\0\0", 4, 7836) = 4
pread(29, "\0\0\0\0", 4, 7840) = 4
pread(29, "\0\0\0\0", 4, 7844) = 4
pread(29, "\0\0\0\0", 4, 7848) = 4
pread(29, "\0\0\0\0", 4, 7852) = 4
pread(29, "\0\0\0\0", 4, 7856) = 4
pread(29, "\0\0\0\0", 4, 7860) = 4
pread(29, "\0\0\0\0", 4, 7864) = 4
pread(29, "\0\0\0\0", 4, 7868) = 4
pread(29, "\0\0\0\0", 4, 7872) = 4
pread(29, "\0\0\0\0", 4, 7876) = 4
pread(29, "\0\0\0\0", 4, 7880) = 4
pread(29, "\0\0\0\0", 4, 7884) = 4
pread(29, "\0\0\0\0", 4, 7888) = 4
pread(29, "\0\0\0\0", 4, 7892) = 4
pread(29, "\0\0\0\0", 4, 7896) = 4
pread(29, "\0\0\0\0", 4, 7900) = 4
pread(29, "\0\0\0\0", 4, 7904) = 4
pread(29, "\0\0\0\0", 4, 7908) = 4
pread(29, "\0\0\0\0", 4, 7912) = 4
pread(29, "\0\0\0\0", 4, 7916) = 4
pread(29, "\0\0\0\0", 4, 7920) = 4
pread(29, "\0\0\0\0", 4, 7924) = 4
pread(29, "\0\0\0\0", 4, 7928) = 4
pread(29, "\0\0\0\0", 4, 7932) = 4
pread(29, "\0\0\0\0", 4, 7936) = 4
pread(29, "\0\0\0\0", 4, 7940) = 4
pread(29, "\0\0\0\0", 4, 7944) = 4
pread(29, "\0\0\0\0", 4, 7948) = 4
pread(29, "\0\0\0\0", 4, 7952) = 4
pread(29, "\0\0\0\0", 4, 7956) = 4
pread(29, "\0\0\0\0", 4, 7960) = 4
pread(29, "\0\0\0\0", 4, 7964) = 4
pread(29, "\0\0\0\0", 4, 7968) = 4
pread(29, "\0\0\0\0", 4, 7972) = 4
pread(29, "\0\0\0\0", 4, 7976) = 4
pread(29, "\0\0\0\0", 4, 7980) = 4
pread(29, "\0\0\0\0", 4, 7984) = 4
pread(29, "\0\0\0\0", 4, 7988) = 4
pread(29, "\0\0\0\0", 4, 7992) = 4
pread(29, "\0\0\0\0", 4, 7996) = 4
pread(29, "\0\0\0\0", 4, 8000) = 4
pread(29, "\0\0\0\0", 4, 8004) = 4
pread(29, "\0\0\0\0", 4, 8008) = 4
pread(29, "\0\0\0\0", 4, 8012) = 4
pread(29, "\0\0\0\0", 4, 8016) = 4
pread(29, "\0\0\0\0", 4, 8020) = 4
pread(29, "\0\0\0\0", 4, 8024) = 4
pread(29, "\0\0\0\0", 4, 8028) = 4
pread(29, "\0\0\0\0", 4, 8032) = 4
pread(29, "\0\0\0\0", 4, 8036) = 4
pread(29, "\0\0\0\0", 4, 8040) = 4
pread(29, "\0\0\0\0", 4, 8044) = 4
pread(29, "\0\0\0\0", 4, 8048) = 4
pread(29, "\0\0\0\0", 4, 8052) = 4
pread(29, "\0\0\0\0", 4, 8056) = 4
pread(29, "\0\0\0\0", 4, 8060) = 4
pread(29, "\0\0\0\0", 4, 8064) = 4
pread(29, "\0\0\0\0", 4, 8068) = 4
pread(29, "\0\0\0\0", 4, 8072) = 4
pread(29, "\0\0\0\0", 4, 8076) = 4
pread(29, "\0\0\0\0", 4, 8080) = 4
pread(29, "\0\0\0\0", 4, 8084) = 4
pread(29, "\0\0\0\0", 4, 8088) = 4
pread(29, "\0\0\0\0", 4, 8092) = 4
pread(29, "\0\0\0\0", 4, 8096) = 4
pread(29, "\0\0\0\0", 4, 8100) = 4
pread(29, "\0\0\0\0", 4, 8104) = 4
pread(29, "\0\0\0\0", 4, 8108) = 4
pread(29, "\0\0\0\0", 4, 8112) = 4
pread(29, "\0\0\0\0", 4, 8116) = 4
pread(29, "\0\0\0\0", 4, 8120) = 4
pread(29, "\0\0\0\0", 4, 8124) = 4
pread(29, "\0\0\0\0", 4, 8128) = 4
pread(29, "\0\0\0\0", 4, 8132) = 4
pread(29, "\0\0\0\0", 4, 8136) = 4
pread(29, "\0\0\0\0", 4, 8140) = 4
pread(29, "\0\0\0\0", 4, 8144) = 4
pread(29, "\0\0\0\0", 4, 8148) = 4
pread(29, "\0\0\0\0", 4, 8152) = 4
pread(29, "\0\0\0\0", 4, 8156) = 4
pread(29, "\0\0\0\0", 4, 8160) = 4
pread(29, "\0\0\0\0", 4, 8164) = 4
pread(29, "\0\0\0\0", 4, 8168) = 4
pread(29, "\0\0\0\0", 4, 8172) = 4
pread(29, "\0\0\0\0", 4, 8176) = 4
pread(29, "\0\0\0\0", 4, 8180) = 4
pread(29, "\0\0\0\0", 4, 8184) = 4
pread(29, "\0\0\0\0", 4, 8188) = 4
pread(29, "\0\0\0\0", 4, 8192) = 4
pread(29, "\0\0\0\0", 4, 8196) = 4
pread(29, "\0\0\0\0", 4, 8200) = 4
pread(29, "\0\0\0\0", 4, 8204) = 4
pread(29, "\0\0\0\0", 4, 8208) = 4
pread(29, "\0\0\0\0", 4, 8212) = 4
pread(29, "\0\0\0\0", 4, 8216) = 4
pread(29, "\0\0\0\0", 4, 8220) = 4
pread(29, "\0\0\0\0", 4, 8224) = 4
pread(29, "\0\0\0\0", 4, 8228) = 4
pread(29, "\0\0\0\0", 4, 8232) = 4
pread(29, "\0\0\0\0", 4, 8236) = 4
pread(29, "\0\0\0\0", 4, 8240) = 4
pread(29, "\0\0\0\0", 4, 8244) = 4
pread(29, "\0\0\0\0", 4, 8248) = 4
pread(29, "\0\0\0\0", 4, 8252) = 4
pread(29, "\0\0\0\0", 4, 8256) = 4
pread(29, "\0\0\0\0", 4, 8260) = 4
pread(29, "\0\0\0\0", 4, 8264) = 4
pread(29, "\0\0\0\0", 4, 8268) = 4
pread(29, "\0\0\0\0", 4, 8272) = 4
pread(29, "\0\0\0\0", 4, 8276) = 4
pread(29, "\0\0\0\0", 4, 8280) = 4
pread(29, "\0\0\0\0", 4, 8284) = 4
pread(29, "\0\0\0\0", 4, 8288) = 4
pread(29, "\0\0\0\0", 4, 8292) = 4
pread(29, "\0\0\0\0", 4, 8296) = 4
pread(29, "\0\0\0\0", 4, 8300) = 4
pread(29, "\0\0\0\0", 4, 8304) = 4
pread(29, "\0\0\0\0", 4, 8308) = 4
pread(29, "\0\0\0\0", 4, 8312) = 4
pread(29, "\0\0\0\0", 4, 8316) = 4
pread(29, "\0\0\0\0", 4, 8320) = 4
pread(29, "\0\0\0\0", 4, 8324) = 4
pread(29, "\0\0\0\0", 4, 8328) = 4
pread(29, "\0\0\0\0", 4, 8332) = 4
pread(29, "\0\0\0\0", 4, 8336) = 4
pread(29, "\0\0\0\0", 4, 8340) = 4
pread(29, "\0\0\0\0", 4, 8344) = 4
pread(29, "\0\0\0\0", 4, 8348) = 4
pread(29, "\0\0\0\0", 4, 8352) = 4
pread(29, "\0\0\0\0", 4, 8356) = 4
pread(29, "\0\0\0\0", 4, 8360) = 4
pread(29, "\0\0\0\0", 4, 8364) = 4
pread(29, "\0\0\0\0", 4, 8368) = 4
pread(29, "\0\0\0\0", 4, 8372) = 4
pread(29, "\0\0\0\0", 4, 8376) = 4
pread(29, "\0\0\0\0", 4, 8380) = 4
pread(29, "\0\0\0\0", 4, 8384) = 4
pread(29, "\0\0\0\0", 4, 8388) = 4
pread(29, "\0\0\0\0", 4, 8392) = 4
pread(29, "\0\0\0\0", 4, 8396) = 4
pread(29, "\0\0\0\0", 4, 8400) = 4
pread(29, "\0\0\0\0", 4, 8404) = 4
pread(29, "\0\0\0\0", 4, 8408) = 4
pread(29, "\0\0\0\0", 4, 8412) = 4
pread(29, "\0\0\0\0", 4, 8416) = 4
pread(29, "\0\0\0\0", 4, 8420) = 4
pread(29, "\0\0\0\0", 4, 8424) = 4
pread(29, "\0\0\0\0", 4, 8428) = 4
pread(29, "\0\0\0\0", 4, 8432) = 4
pread(29, "\0\0\0\0", 4, 8436) = 4
pread(29, "\0\0\0\0", 4, 8440) = 4
pread(29, "\0\0\0\0", 4, 8444) = 4
pread(29, "\0\0\0\0", 4, 8448) = 4
pread(29, "\0\0\0\0", 4, 8452) = 4
pread(29, "\0\0\0\0", 4, 8456) = 4
pread(29, "\0\0\0\0", 4, 8460) = 4
pread(29, "\0\0\0\0", 4, 8464) = 4
pread(29, "\0\0\0\0", 4, 8468) = 4
pread(29, "\0\0\0\0", 4, 8472) = 4
pread(29, "\0\0\0\0", 4, 8476) = 4
pread(29, "\0\0\0\0", 4, 8480) = 4
pread(29, "\0\0\0\0", 4, 8484) = 4
pread(29, "\0\0\0\0", 4, 8488) = 4
pread(29, "\0\0\0\0", 4, 8492) = 4
pread(29, "\0\0\0\0", 4, 8496) = 4
pread(29, "\0\0\0\0", 4, 8500) = 4
pread(29, "\0\0\0\0", 4, 8504) = 4
pread(29, "\0\0\0\0", 4, 8508) = 4
pread(29, "\0\0\0\0", 4, 8512) = 4
pread(29, "\0\0\0\0", 4, 8516) = 4
pread(29, "\0\0\0\0", 4, 8520) = 4
pread(29, "\0\0\0\0", 4, 8524) = 4
pread(29, "\0\0\0\0", 4, 8528) = 4
pread(29, "\0\0\0\0", 4, 8532) = 4
pread(29, "\0\0\0\0", 4, 8536) = 4
pread(29, "\0\0\0\0", 4, 8540) = 4
pread(29, "\0\0\0\0", 4, 8544) = 4
pread(29, "\0\0\0\0", 4, 8548) = 4
pread(29, "\0\0\0\0", 4, 8552) = 4
pread(29, "\0\0\0\0", 4, 8556) = 4
pread(29, "\0\0\0\0", 4, 8560) = 4
pread(29, "\0\0\0\0", 4, 8564) = 4
pread(29, "\0\0\0\0", 4, 8568) = 4
pread(29, "\0\0\0\0", 4, 8572) = 4
pread(29, "\0\0\0\0", 4, 8576) = 4
pread(29, "\0\0\0\0", 4, 8580) = 4
pread(29, "\0\0\0\0", 4, 8584) = 4
pread(29, "\0\0\0\0", 4, 8588) = 4
pread(29, "\0\0\0\0", 4, 8592) = 4
pread(29, "\0\0\0\0", 4, 8596) = 4
pread(29, "\0\0\0\0", 4, 8600) = 4
pread(29, "\0\0\0\0", 4, 8604) = 4
pread(29, "\0\0\0\0", 4, 8608) = 4
pread(29, "\0\0\0\0", 4, 8612) = 4
pread(29, "\0\0\0\0", 4, 8616) = 4
pread(29, "\0\0\0\0", 4, 8620) = 4
pread(29, "\0\0\0\0", 4, 8624) = 4
pread(29, "\0\0\0\0", 4, 8628) = 4
pread(29, "\0\0\0\0", 4, 8632) = 4
pread(29, "\0\0\0\0", 4, 8636) = 4
pread(29, "\0\0\0\0", 4, 8640) = 4
pread(29, "\0\0\0\0", 4, 8644) = 4
pread(29, "\0\0\0\0", 4, 8648) = 4
pread(29, "\0\0\0\0", 4, 8652) = 4
pread(29, "\0\0\0\0", 4, 8656) = 4
pread(29, "\0\0\0\0", 4, 8660) = 4
pread(29, "\0\0\0\0", 4, 8664) = 4
pread(29, "\0\0\0\0", 4, 8668) = 4
pread(29, "\0\0\0\0", 4, 8672) = 4
pread(29, "\0\0\0\0", 4, 8676) = 4
pread(29, "\0\0\0\0", 4, 8680) = 4
pread(29, "\0\0\0\0", 4, 8684) = 4
pread(29, "\0\0\0\0", 4, 8688) = 4
pread(29, "\0\0\0\0", 4, 8692) = 4
pread(29, "\0\0\0\0", 4, 8696) = 4
pread(29, "\0\0\0\0", 4, 8700) = 4
pread(29, "\0\0\0\0", 4, 8704) = 4
pread(29, "\0\0\0\0", 4, 8708) = 4
pread(29, "\0\0\0\0", 4, 8712) = 4
pread(29, "\0\0\0\0", 4, 8716) = 4
pread(29, "\0\0\0\0", 4, 8720) = 4
pread(29, "\0\0\0\0", 4, 8724) = 4
pread(29, "\0\0\0\0", 4, 8728) = 4
pread(29, "\0\0\0\0", 4, 8732) = 4
pread(29, "\0\0\0\0", 4, 8736) = 4
pread(29, "\0\0\0\0", 4, 8740) = 4
pread(29, "\0\0\0\0", 4, 8744) = 4
pread(29, "\0\0\0\0", 4, 8748) = 4
pread(29, "\0\0\0\0", 4, 8752) = 4
pread(29, "\0\0\0\0", 4, 8756) = 4
pread(29, "\0\0\0\0", 4, 8760) = 4
pread(29, "\0\0\0\0", 4, 8764) = 4
pread(29, "\0\0\0\0", 4, 8768) = 4
pread(29, "\0\0\0\0", 4, 8772) = 4
pread(29, "\0\0\0\0", 4, 8776) = 4
pread(29, "\0\0\0\0", 4, 8780) = 4
pread(29, "\0\0\0\0", 4, 8784) = 4
pread(29, "\0\0\0\0", 4, 8788) = 4
pread(29, "\0\0\0\0", 4, 8792) = 4
pread(29, "\0\0\0\0", 4, 8796) = 4
pread(29, "\0\0\0\0", 4, 8800) = 4
pread(29, "\0\0\0\0", 4, 8804) = 4
pread(29, "\0\0\0\0", 4, 8808) = 4
pread(29, "\0\0\0\0", 4, 8812) = 4
pread(29, "\0\0\0\0", 4, 8816) = 4
pread(29, "\0\0\0\0", 4, 8820) = 4
pread(29, "\0\0\0\0", 4, 8824) = 4
pread(29, "\0\0\0\0", 4, 8828) = 4
pread(29, "\0\0\0\0", 4, 8832) = 4
pread(29, "\0\0\0\0", 4, 8836) = 4
pread(29, "\0\0\0\0", 4, 8840) = 4
pread(29, "\0\0\0\0", 4, 8844) = 4
pread(29, "\0\0\0\0", 4, 8848) = 4
pread(29, "\0\0\0\0", 4, 8852) = 4
pread(29, "\0\0\0\0", 4, 8856) = 4
pread(29, "\0\0\0\0", 4, 8860) = 4
pread(29, "\0\0\0\0", 4, 8864) = 4
pread(29, "\0\0\0\0", 4, 8868) = 4
pread(29, "\0\0\0\0", 4, 8872) = 4
pread(29, "\0\0\0\0", 4, 8876) = 4
pread(29, "\0\0\0\0", 4, 8880) = 4
pread(29, "\0\0\0\0", 4, 8884) = 4
pread(29, "\0\0\0\0", 4, 8888) = 4
pread(29, "\0\0\0\0", 4, 8892) = 4
pread(29, "\0\0\0\0", 4, 8896) = 4
pread(29, "\0\0\0\0", 4, 8900) = 4
pread(29, "\0\0\0\0", 4, 8904) = 4
pread(29, "\0\0\0\0", 4, 8908) = 4
pread(29, "\0\0\0\0", 4, 8912) = 4
pread(29, "\0\0\0\0", 4, 8916) = 4
pread(29, "\0\0\0\0", 4, 8920) = 4
pread(29, "\0\0\0\0", 4, 8924) = 4
pread(29, "\0\0\0\0", 4, 8928) = 4
pread(29, "\0\0\0\0", 4, 8932) = 4
pread(29, "\0\0\0\0", 4, 8936) = 4
pread(29, "\0\0\0\0", 4, 8940) = 4
pread(29, "\0\0\0\0", 4, 8944) = 4
pread(29, "\0\0\0\0", 4, 8948) = 4
pread(29, "\0\0\0\0", 4, 8952) = 4
pread(29, "\0\0\0\0", 4, 8956) = 4
pread(29, "\0\0\0\0", 4, 8960) = 4
pread(29, "\0\0\0\0", 4, 8964) = 4
pread(29, "\0\0\0\0", 4, 8968) = 4
pread(29, "\0\0\0\0", 4, 8972) = 4
pread(29, "\0\0\0\0", 4, 8976) = 4
pread(29, "\0\0\0\0", 4, 8980) = 4
pread(29, "\0\0\0\0", 4, 8984) = 4
pread(29, "\0\0\0\0", 4, 8988) = 4
pread(29, "\0\0\0\0", 4, 8992) = 4
pread(29, "\0\0\0\0", 4, 8996) = 4
pread(29, "\0\0\0\0", 4, 9000) = 4
pread(29, "\0\0\0\0", 4, 9004) = 4
pread(29, "\0\0\0\0", 4, 9008) = 4
pread(29, "\0\0\0\0", 4, 9012) = 4
pread(29, "\0\0\0\0", 4, 9016) = 4
pread(29, "\0\0\0\0", 4, 9020) = 4
pread(29, "\0\0\0\0", 4, 9024) = 4
pread(29, "\0\0\0\0", 4, 9028) = 4
pread(29, "\0\0\0\0", 4, 9032) = 4
pread(29, "\0\0\0\0", 4, 9036) = 4
pread(29, "\0\0\0\0", 4, 9040) = 4
pread(29, "\0\0\0\0", 4, 9044) = 4
pread(29, "\0\0\0\0", 4, 9048) = 4
pread(29, "\0\0\0\0", 4, 9052) = 4
pread(29, "\0\0\0\0", 4, 9056) = 4
pread(29, "\0\0\0\0", 4, 9060) = 4
pread(29, "\0\0\0\0", 4, 9064) = 4
pread(29, "\0\0\0\0", 4, 9068) = 4
pread(29, "\0\0\0\0", 4, 9072) = 4
pread(29, "\0\0\0\0", 4, 9076) = 4
pread(29, "\0\0\0\0", 4, 9080) = 4
pread(29, "\0\0\0\0", 4, 9084) = 4
pread(29, "\0\0\0\0", 4, 9088) = 4
pread(29, "\0\0\0\0", 4, 9092) = 4
pread(29, "\0\0\0\0", 4, 9096) = 4
pread(29, "\0\0\0\0", 4, 9100) = 4
pread(29, "\0\0\0\0", 4, 9104) = 4
pread(29, "\0\0\0\0", 4, 9108) = 4
pread(29, "\0\0\0\0", 4, 9112) = 4
pread(29, "\0\0\0\0", 4, 9116) = 4
pread(29, "\0\0\0\0", 4, 9120) = 4
pread(29, "\0\0\0\0", 4, 9124) = 4
pread(29, "\0\0\0\0", 4, 9128) = 4
pread(29, "\0\0\0\0", 4, 9132) = 4
pread(29, "\0\0\0\0", 4, 9136) = 4
pread(29, "\0\0\0\0", 4, 9140) = 4
pread(29, "\0\0\0\0", 4, 9144) = 4
pread(29, "\0\0\0\0", 4, 9148) = 4
pread(29, "\0\0\0\0", 4, 9152) = 4
pread(29, "\0\0\0\0", 4, 9156) = 4
pread(29, "\0\0\0\0", 4, 9160) = 4
pread(29, "\0\0\0\0", 4, 9164) = 4
pread(29, "\0\0\0\0", 4, 9168) = 4
pread(29, "\0\0\0\0", 4, 9172) = 4
pread(29, "\0\0\0\0", 4, 9176) = 4
pread(29, "\0\0\0\0", 4, 9180) = 4
pread(29, "\0\0\0\0", 4, 9184) = 4
pread(29, "\0\0\0\0", 4, 9188) = 4
pread(29, "\0\0\0\0", 4, 9192) = 4
pread(29, "\0\0\0\0", 4, 9196) = 4
pread(29, "\0\0\0\0", 4, 9200) = 4
pread(29, "\0\0\0\0", 4, 9204) = 4
pread(29, "\0\0\0\0", 4, 9208) = 4
pread(29, "\0\0\0\0", 4, 9212) = 4
pread(29, "\0\0\0\0", 4, 9216) = 4
pread(29, "\0\0\0\0", 4, 9220) = 4
pread(29, "\0\0\0\0", 4, 9224) = 4
pread(29, "\0\0\0\0", 4, 9228) = 4
pread(29, "\0\0\0\0", 4, 9232) = 4
pread(29, "\0\0\0\0", 4, 9236) =

Post by Alex Crow
--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.
"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: [phone] Fax: [phone].
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).

--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.

"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: [phone] Fax: [phone].
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).

[Alex Crow]

Post by Alex Crow

Post by Alex Crow
Hi,
We've had a strange issue after following the recommendations at
https://wiki.samba.org/index.php/Ping_pong, particularly the part
about mmap coherence. We are running CTDB/Samba over a MooseFS
clustered FS, and we'd not done the ping-pong before. After finding
that the mmap coherence test did not pass, we changed "use mmap" to
"no" in smb.conf.
This morning as users began to access their drives and profiles,
performance tanked to such a degree that most people could not
complete loading profiles and even where successful, windows drive
mappings were taking minutes to complete per share.
When looking at the active CTDB servers, on each a single Winbindd
process was taking up 100% of CPU. After reverting the "use_mmap" to
yes, performance returned completely to normal.
We found this very odd as this is a recommended setting according to
the page above.

Hmmm. I know very little about MooseFS. Did you get an strace
on the winbindd process to try and figure out what it was doing ?

I'm afraid not, the priority was to get people working and I was fixing
it remotely.
I may be able to try it on a test setup.
Thanks
Alex

Same with smbd:

pread(32, "\0\0\0\0", 4, 34956) = 4
pread(32, "\0\0\0\0", 4, 34960) = 4
pread(32, "\0\0\0\0", 4, 34964) = 4
pread(32, "\0\0\0\0", 4, 34968) = 4
pread(32, "\0\0\0\0", 4, 34972) = 4
pread(32, "\0\0\0\0", 4, 34976) = 4
pread(32, "\0\0\0\0", 4, 34980) = 4
pread(32, "\0\0\0\0", 4, 34984) = 4
pread(32, "\0\0\0\0", 4, 34988) = 4
pread(32, "\0\0\0\0", 4, 34992) = 4
pread(32, "\0\0\0\0", 4, 34996) = 4
pread(32, "\0\0\0\0", 4, 35000) = 4
pread(32, "\0\0\0\0", 4, 35004) = 4
pread(32, "\0\0\0\0", 4, 35008) = 4
pread(32, "\0\0\0\0", 4, 35012) = 4
pread(32, "\0\0\0\0", 4, 35016) = 4
pread(32, "\0\0\0\0", 4, 35020) = 4
pread(32, "\0\0\0\0", 4, 35024) = 4
pread(32, "\0\0\0\0", 4, 35028) = 4
pread(32, "\0\0\0\0", 4, 35032) = 4
pread(32, "\0\0\0\0", 4, 35036) = 4
pread(32, "\0\0\0\0", 4, 35040) = 4
pread(32, "\0\0\0\0", 4, 35044) = 4
pread(32, "\0\0\0\0", 4, 35048) = 4
pread(32, "\0\0\0\0", 4, 35052) = 4
pread(32, "\0\0\0\0", 4, 35056) = 4
pread(32, "\0\0\0\0", 4, 35060) = 4
pread(32, "\0\0\0\0", 4, 35064) = 4
pread(32, "\0\0\0\0", 4, 35068) = 4
pread(32, "\0\0\0\0", 4, 35072) = 4
pread(32, "\0\0\0\0", 4, 35076) = 4
pread(32, "\0\0\0\0", 4, 35080) = 4
pread(32, "\0\0\0\0", 4, 35084) = 4

--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.

"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: [phone] Fax: [phone].
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).

[Alex Crow]

Post by Alex Crow

Post by Alex Crow
Hi,
We've had a strange issue after following the recommendations at
https://wiki.samba.org/index.php/Ping_pong, particularly the part
about mmap coherence. We are running CTDB/Samba over a MooseFS
clustered FS, and we'd not done the ping-pong before. After finding
that the mmap coherence test did not pass, we changed "use mmap" to
"no" in smb.conf.
This morning as users began to access their drives and profiles,
performance tanked to such a degree that most people could not
complete loading profiles and even where successful, windows drive
mappings were taking minutes to complete per share.
When looking at the active CTDB servers, on each a single Winbindd
process was taking up 100% of CPU. After reverting the "use_mmap" to
yes, performance returned completely to normal.
We found this very odd as this is a recommended setting according to
the page above.

Hmmm. I know very little about MooseFS. Did you get an strace
on the winbindd process to try and figure out what it was doing ?

I'm afraid not, the priority was to get people working and I was fixing
it remotely.
I may be able to try it on a test setup.
Thanks
Alex

Here's the end of the winbindd process, this time on logoff:

fcntl(25, F_SETLKW, {type=F_UNLCK, whence=SEEK_SET, start=8, len=1}) = 0
open("/etc/group", O_RDONLY|O_CLOEXEC) = 34
fstat(34, {st_mode=S_IFREG|0644, st_size=963, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0x7f6e2ca09000
read(34, "root:x:0:\nbin:x:1:\ndaemon:x:2:\ns"..., 4096) = 963
read(34, "", 4096) = 0
close(34) = 0
munmap(0x7f6e2ca09000, 4096) = 0
socket(PF_LOCAL, SOCK_STREAM, 0) = 34
fcntl(34, F_GETFL) = 0x2 (flags O_RDWR)
fcntl(34, F_SETFD, FD_CLOEXEC|0x2) = 0
connect(34, {sa_family=AF_LOCAL, sun_path="/var/run/nslcd/socket"}, 23) = 0
poll([{fd=34, events=POLLOUT}], 1, 10000) = 1 ([{fd=34, revents=POLLOUT}])
sendto(34, "\1\0\0\0\212\23\0\0\257&\0\0", 12, MSG_NOSIGNAL, NULL, 0) = 12
poll([{fd=34, events=POLLIN}], 1, 60000) = 1 ([{fd=34,
revents=POLLIN|POLLHUP}])
read(34, "\1\0\0\0\212\23\0\0\3\0\0\0", 1024) = 12
close(34) = 0
fcntl(28, F_SETLKW, {type=F_RDLCK, whence=SEEK_SET, start=232, len=1}) = 0
fcntl(28, F_SETLKW, {type=F_UNLCK, whence=SEEK_SET, start=232, len=1}) = 0
fcntl(32, F_SETLKW, {type=F_WRLCK, whence=SEEK_SET, start=48, len=1}) = 0
fcntl(32, F_SETLKW, {type=F_UNLCK, whence=SEEK_SET, start=48, len=1}) = 0
writev(33,
[{"\264\r\0\0\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"...,
3496}, {"\0\0\0\0\0\0\0\0s\0\0\300", 12}], 2) = -1 EPIPE (Broken pipe)
--- SIGPIPE {si_signo=SIGPIPE, si_code=SI_USER, si_pid=31971, si_uid=0} ---
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=3661, ...}) = 0
geteuid() = 0
write(16, "[2016/07/02 15:12:16.209453, 0]"..., 95) = 95
geteuid() = 0
write(16, " Could not write result\n", 25) = 25
close(11) = 0
close(35) = 0
exit_group(1) = ?


--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.

"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: [phone] Fax: [phone].
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).

[Alex Crow]

Post by Alex Crow
We've had a strange issue after following the recommendations at
https://wiki.samba.org/index.php/Ping_pong, particularly the part
about mmap coherence. We are running CTDB/Samba over a MooseFS
clustered FS, and we'd not done the ping-pong before. After finding
that the mmap coherence test did not pass, we changed "use mmap" to
"no" in smb.conf.

"use mmap" only affects tdbs, which should not sit on the clustered
fs, they should *always* be on a local file system like ext4, assuming
you're using Linux. The wiki page above states that it's not
absolutely essential to provide mmap coherence, so you should not
bother too much.
Or are you putting tdb files on moosefs?
BTW, the straces you're seeing are probably not real spins, that's
traverses. With 10000 hash chains, that will be 10000 preads. That's
just taking time. If the tdbs sit on moosefs, it might actually be
true that it does not like that. But as I said: Never put tdbs on a
cluster file system. If cluster file systems did that access pattern
fine, ctdb would not exist :-)
Volker

Hi Volker,

I've only put the "private dir" onto MooseFS, as instructed in the CTDB
docs.

So, in that case, I'm assuming from your comments that it is no worry
that the mmap test does not pass on the MooseFS mounts?

The "private dir" contains these files:

-rw-------+ 1 root root 24576 Jun 23 19:48 netlogon_creds_cli.tdb
-rw-------+ 1 root root 421888 Jun 23 19:48 passdb.tdb
-rw-------+ 1 root root 430080 Jun 23 19:47 secrets.tdb
drwxr-xr-x+ 3 root root 1001200 Jun 23 19:48 smbd.tmp

It just strikes me as odd that "use mmap = no" causes such a slowdown in
the the case I only have these files on my clustered FS. The timestamp
of all those files is the last time I joined one of the member servers
to the domain.

More worrying is that the rw ping-pong (no mmap) does not work properly
even when I mount my MooseFS with -o mfscachemode=none, which is not
supposed to do any local page/file cache and yet still I get
inconsistent results when adding and removing ping_pong tests on other
nodes (ie increment count does not always match running test nodes). I
have raised this with the MooseFS chaps and am waiting for a response.

Many thanks

Alex



--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.

"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: [phone] Fax: [phone].
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).

[Alex Crow]

Post by Alex Crow
I've only put the "private dir" onto MooseFS, as instructed in the CTDB
docs.

Can you quote these docs, so that we can correct them?

Post by Alex Crow
So, in that case, I'm assuming from your comments that it is no worry
that the mmap test does not pass on the MooseFS mounts?

Yes, it is a high worry. If MooseFS is not mmap coherent, you need
"use mmap = no", and we need to fix Samba and/or MooseFS to support
tdbs better.
But as said previously, ctdb should make this unnecessary.

Post by Alex Crow
-rw-------+ 1 root root 24576 Jun 23 19:48 netlogon_creds_cli.tdb
-rw-------+ 1 root root 421888 Jun 23 19:48 passdb.tdb
-rw-------+ 1 root root 430080 Jun 23 19:47 secrets.tdb
drwxr-xr-x+ 3 root root 1001200 Jun 23 19:48 smbd.tmp

This is weird. ctdb-controlled files should have the node-number
appended. Are you sure that you are running ctdb?
Volker

100% sure I'm using it! :

[***@zearing ~]# ctdb status
Number of nodes:6
pnn:0 172.16.40.13 OK
pnn:1 172.16.40.14 OK (THIS NODE)
pnn:2 172.16.40.15 OK
pnn:3 192.168.21.24 OK
pnn:4 192.168.21.25 OK
pnn:5 192.168.21.27 OK
Generation:1417726848
Size:6
hash:0 lmaster:0
hash:1 lmaster:1
hash:2 lmaster:2
hash:3 lmaster:3
hash:4 lmaster:4
hash:5 lmaster:5
Recovery mode:NORMAL (0)
Recovery master:4

smb.conf:

[global]

max protocol = SMB2
host msdfs = yes
workgroup = IFA_NET
netbios name = LIZARDCLUSTER
private dir = /mfs/ctdb/private
clustering = yes
use mmap = yes

/etc/sysconfig/ctdb:

# Options to ctdbd, read by ctdbd_wrapper(1)
#
# See ctdbd.conf(5) for more information about CTDB configuration variables.

# Shared recovery lock file to avoid split brain. No default.
#
# Do NOT run CTDB without a recovery lock file unless you know exactly
# what you are doing.
CTDB_RECOVERY_LOCK=/mfs/ctdb/recovery.lock

# List of nodes in the cluster. Default is below.
CTDB_NODES=/mfs/ctdb/nodes

# List of public addresses for providing NAS services. No default.
CTDB_PUBLIC_ADDRESSES=/mfs/ctdb/public_addresses_cl

# What services should CTDB manage? Default is none.
CTDB_MANAGES_SAMBA=yes
CTDB_MANAGES_WINBIND=yes

# CTDB_MANAGES_NFS=yes

# Raise the file descriptor limit for CTDB?
# ulimit -n 10000

# Default is to use the log file below instead of syslog.
# CTDB_LOGFILE=/var/log/log.ctdb
# CTDB_SYSLOG=no

# Default log level is ERR. NOTICE is a little more verbose.
CTDB_DEBUGLEVEL=NOTICE

# Set some CTDB tunable variables during CTDB startup?
# CTDB_SET_TraverseTimeout=60
CTDB_SAMBA_SKIP_SHARE_CHECK=yes

--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.

"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: [phone] Fax: [phone].
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).

[Alex Crow]

And you do have "clustering = yes" in your smb.conf?
Volker

Yes, I posted that in my last reply.

I think those files were in /mfs/ctdb/private because I tried to rejoin
the domain with "clustering = yes" commented out but "private dir" still
set and then realised my mistake.

Thanks,

Alex
--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.

"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: [phone] Fax: [phone].
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).

[Alex Crow]

Post by Alex Crow
I've only put the "private dir" onto MooseFS, as instructed in the CTDB
docs.

Can you quote these docs, so that we can correct them?

Here, under the lustre section. I applied the same config at it's a
similar FS (ie distributed with a central metadata server):

https://wiki.samba.org/index.php/CTDB_Setup

"

In comparison of Lustre configurating, setting up CTDB/Samba on the two
different versions keeps the same way. The following settings are
recommended:

clustering = yes
idmap backend = tdb2
private dir=/mnt/lustre/ctdb
fileid:mapping = fsname
use mmap = no
nt acl support = yes
ea support = yes


"


Alex
--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.

"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: [phone] Fax: [phone].
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).

[Alex Crow]

Post by Alex Crow
I've only put the "private dir" onto MooseFS, as instructed in the CTDB
docs.

Can you quote these docs, so that we can correct them?

Also Gluster have the same tip here:
http://download.gluster.org/pub/gluster/glusterfs/doc/Gluster_CTDB_setup.v1.pdf

• Add the following lines to the [global] section of your Samba
configuration
clustering = yes
idmap backend = tdb2
private dir = /gluster/lock

Can you confirm that "private dir" does *not* have to be set in smb.conf
to point to a shared FS? If not, would that solve the original problem?

Thanks

Alex
--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.

"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: [phone] Fax: [phone].
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).

[Alex Crow]

Post by Alex Crow
• Add the following lines to the [global] section of your Samba
configuration
clustering = yes
idmap backend = tdb2
private dir = /gluster/lock
Can you confirm that "private dir" does *not* have to be set in smb.conf to
point to a shared FS? If not, would that solve the original problem?

No, it does not have to be set. In fact, it is very likely harmful.
ctdb determines where the clustered dbs are located, not samba.
Volker

HI Volker,

I've set up the "DR" side of my cluster to "use mmap = no" and with
"private dir" removed from the smb.conf.

However I still have huge CPU utilisation from winbindd and smbd just
fromtrying to long in a single test XP machine.

I have the MooseFS guys on the case as well. Should I put them in touch
with you or vice/versa?

Many thanks for your gracious assistance,

Alex

--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.

"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: [phone] Fax: [phone].
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).

[Alex Crow]

Post by Alex Crow
I've set up the "DR" side of my cluster to "use mmap = no" and with
"private dir" removed from the smb.conf.

Why do you set "use mmap = no"?

Post by Alex Crow
I have the MooseFS guys on the case as well. Should I put them in touch
with you or vice/versa?

MooseFS should not have any business at all in tdb files. If you still
find tdb files used on MooseFS, you should investigate your
configuration and modify it such that tdb files only exist on local
file systems.
Volker

Hi Volker,

There are *no* more tdb files on MooseFS, that's what I'm trying to test.

From the ping_pong wiki page it's instructed to use "use_mmap = no" if
the ping_pong test does not work correctly on "ping_ping -m -rw
test.file", which it did, completely.

Text here:

"


Testing mmap coherence

If you add the -m switch to ping_pong along with -rw then it will do the
I/O coherence test via mmap. It isn't absolutely essential that a
cluster filesystem supports coherent mmap for CTDB/Samba, but it's nice
for bragging points over other cluster filesystems. If your cluster
filesystem doesn't pass this test then just use the "use mmap = no"
option in smb.conf. Even if it does pass this test that option may be a
good idea on most cluster filesystems."


So is this documentation wrong as well? It seems to suggest setting it
anyway, which is clearly not working for us (and now for sure there is
no involvement with TDB files as I've deleted the "private dir" in smb.conf.


The only reason I was asking if you'd like to talk to the MooseFS chaps
is because of the failure of the -rw test, which as far as the
documentation goes, seems to refer to the shared FS rather than any
places where tdb files live.

I hope you can see what I'm getting at here!

Many thanks

Alex




--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.

"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: [phone] Fax: [phone].
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).

[Alex Crow]

FYI, by "it did, completely" I meant it failed completely.

Even if the only file we have to have on the cluster FS (which now seems
to be down solely to the ctdb lock file), I'm still worried what these
failures mean:

1) does the -rw (without -m) test suggest any problems with the MooseFS
FS I'm serving files to users from?
2) does the mmap test failure mean I should not even have my CTDB lock
file on MooseFS?
3) Do the inconsistent reports from a ping_pong -rw test show there is a
major problem with MooseFS as the underlying FS for Samba exports and
something needs to be done with the underlying FS?

I'm a paying licensee of MooseFS Pro, so I hope you can understand my
wishes to get this problem sorted. If contracting some of the Sernet
chaps is required, we've done it before and might be prepared to do so
in this case.

Best regards

Alex

Post by Alex Crow

Post by Alex Crow
I've set up the "DR" side of my cluster to "use mmap = no" and with
"private dir" removed from the smb.conf.

Why do you set "use mmap = no"?

Post by Alex Crow
I have the MooseFS guys on the case as well. Should I put them in touch
with you or vice/versa?

MooseFS should not have any business at all in tdb files. If you still
find tdb files used on MooseFS, you should investigate your
configuration and modify it such that tdb files only exist on local
file systems.
Volker

Hi Volker,
There are *no* more tdb files on MooseFS, that's what I'm trying to test.
From the ping_pong wiki page it's instructed to use "use_mmap = no" if
the ping_pong test does not work correctly on "ping_ping -m -rw
test.file", which it did, completely.
"
Testing mmap coherence
If you add the -m switch to ping_pong along with -rw then it will do
the I/O coherence test via mmap. It isn't absolutely essential that a
cluster filesystem supports coherent mmap for CTDB/Samba, but it's
nice for bragging points over other cluster filesystems. If your
cluster filesystem doesn't pass this test then just use the "use mmap
= no" option in smb.conf. Even if it does pass this test that option
may be a good idea on most cluster filesystems."
So is this documentation wrong as well? It seems to suggest setting it
anyway, which is clearly not working for us (and now for sure there is
no involvement with TDB files as I've deleted the "private dir" in smb.conf.
The only reason I was asking if you'd like to talk to the MooseFS
chaps is because of the failure of the -rw test, which as far as the
documentation goes, seems to refer to the shared FS rather than any
places where tdb files live.
I hope you can see what I'm getting at here!
Many thanks
Alex

--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.

"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: [phone] Fax: [phone].
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).

[Alex Crow]

Post by Alex Crow
FYI, by "it did, completely" I meant it failed completely.
Even if the only file we have to have on the cluster FS (which now
seems to be down solely to the ctdb lock file), I'm still worried
1) does the -rw (without -m) test suggest any problems with the
MooseFS FS I'm serving files to users from?
2) does the mmap test failure mean I should not even have my CTDB
lock file on MooseFS?
3) Do the inconsistent reports from a ping_pong -rw test show there
is a major problem with MooseFS as the underlying FS for Samba
exports and something needs to be done with the underlying FS?
I'm a paying licensee of MooseFS Pro, so I hope you can understand my
wishes to get this problem sorted. If contracting some of the Sernet
chaps is required, we've done it before and might be prepared to do
so in this case.

ROFL ROFL
Do you know who Volker is ???
Please listen to him, do what he advises and don't bother contacting
Sernet, because you already have.
Rowland

I know who Volker is and I've met Jeremy Allison may years ago at the
now defunct London Linux expo..

What am I supposed to be listening to? I'm just suggesting some way I
can contribute to MooseFS by getting it to work consistently with
Samba/CTDB. Volker suggests just dropping the FS which is not a very
good option for our organisation right now. The wiki is unclear and
contradictory, and I don't have the facts to update it without risking
it being equally broken afterwards, and misleading others, so that's why
I'm trying to provide clear information, performing tests, reporting
back, and asking for advice to find out how it really *does* work.

I feel a little offended by the tone of this message, as I've been on
this list for over 15 years and feel I've helped out a lot of people
with other queries I've already experienced.

I am prepared to pay Volker via Sernet as this is of great importance to
us and will help develop another open source product. Is that so hard to
understand?

Best regards

Alex

--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.

"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: [phone] Fax: [phone].
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).

[Alex Crow]

.

Sorry if I offended you, but Volker was telling you what you needed to
do. You were coming over as if you didn't want to do what you were
being advised to do.

No, that is just not the case. I just wanted to be sure of the
implications of what I was doing and what areas of the operation of
Samba and CTDB might be affected given the confusing results of my
ping_pong tests.

Post by Alex Crow
I am prepared to pay Volker via Sernet as this is of great importance
to us and will help develop another open source product. Is that so
hard to understand?

When you put it that way, no, but the way to put it previously, yes.

I did not "put it" like anything. I just saw an problem in my setup,
read some documents on the samba wiki, followed some advice and saw some
unexpected behaviour. Perhaps I imagined the wiki to be more moderated
that it really is, so my trust was misplaced. I don't blame anyone, all
I'm trying to do is solve a problem.

how should I put this?
MooseFS sounds like it isn't useful with CTDB, you have paid for
MooseFS, it doesn't work, so you are now blaming Samba. You need to do
what Volker says and get MooseFS to work with mmap.

AFAIK I don't need to get it to work with mmap, just get MooseFS to fix
the non-mmap I/O coherence as, as Volker says, mmap is only important
for TDB files and we don't store those on MooseFS.

Again, no "blame" was ever implied. I saw Winbindd using 100% CPU after
changing the "use mmap" parameter to "no" and was rather perplexed as to
how that could happen. The first natural place to ask would be this
list, no? I also don't recall using any accusatory or inflammatory
language, just stating observations, posting logs and actually at one or
two points saying I'm also talking to the MooesFS devs.

As a busy sysadmin I rely on a number of sources to design and set up
infrastructure, and sometimes it's easy to get mislead. At no point have
I ever questioned the knowledge of people like Jeremy, Tridge, or
Volker. I'm just dismayed as a professional at getting "ROFLS" in a
reply to what I feel are perfectly sensible questions stemming from a
misguided source of documentation.

Apologies for clutter.

Alex




--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.

"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: [phone] Fax: [phone].
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).

[Alex Crow]

Sorry for that Alex. Rowland, please be nicer to professional
people just asking for help.
I'm just back from a weeks vacation and trying to catch up.
What is the current state of your problem Alex ? As I understand
it you are requesting the MooseFS developers fix the mmap/pwrite
coherence issues.
Is there anything else you need from Samba ? What issues are
outstanding for you ?
Cheers,
Jeremy.

Hi Jeremy,

We are working through a couple of niggles, one of which is the
inconsistent results from the ping_pong test, the other being the
occasional odd permission issue when a file shows with the correct
permissions on the FS but users are getting "Access Denied" errors.
Simply "re-applying" the permissions seems to fix it, even though
getfacl showed them correctly in the first place.

At the moment we're pretty happy with the Samba side of things. I did
wonder if there was any help that you kind chaps might be able to give
to the MooseFS guys if they need it (they've not asked yet but I've
suggested they might be able to). Overall it's working well, and much
better than when we tried to move Samba shares to GlusterFS (bad
memories of that one, lots of small files were not a good match).

Many thanks for your message,

Alex
--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.

"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: [phone] Fax: [phone].
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).

[Alex Crow]

Post by Alex Crow
At the moment we're pretty happy with the Samba side of things. I
did wonder if there was any help that you kind chaps might be able
to give to the MooseFS guys if they need it (they've not asked yet
but I've suggested they might be able to). Overall it's working

The main hint I would like to give to the MooseFS developers is to get
ping_pong -rw working. If you need any assistance in setting that up
and getting arguments on why this is important, let us know!
With best regards,
Volker

Hi Volker,

Any arguments that I can pass on would be much appreciated.

I've also had a try with RozoFS, which did not work as expected as it
only supports block-level locks (4096 bytes) so the first lock prevent
further locks in that block range, which naturally causes ping_pong to
stall. However I am told by their devs that this should not be an issue
for CTDB's recovery lock or Samba as they should not need to lock ranges
of less than a 4k, and they in fact integrate CTDB for their customers.

Best regards

Alex

--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.

"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: [phone] Fax: [phone].
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).

[Alex Crow]

Post by Alex Crow

The main hint I would like to give to the MooseFS developers is to get
ping_pong -rw working. If you need any assistance in setting that up
and getting arguments on why this is important, let us know!
With best regards,
Volker

Any arguments that I can pass on would be much appreciated.

The main argument is that Windows applications tend to coordinate access
to files via byte-range locks. They expect data coherency under those
byte range locks, meaning that if a value is written under a brlock,
the brlock is released, and another process reads the value under the
brlock, the value written must be visible. If this is not guaranteed,
then data corruption will likely happen. The main question to the
MooseFS developers would be under which circumstances a value written
via pwrite on node A is visible via pread on node B. ping_pong checks
this sequentiality with fcntl byte range locks, which are the Posix
way of serializing access to ranges of a file. If MooseFS gives other
guarantees, we would like to know about them.

Post by Alex Crow
I've also had a try with RozoFS, which did not work as expected as
it only supports block-level locks (4096 bytes) so the first lock
prevent further locks in that block range, which naturally causes
ping_pong to stall. However I am told by their devs that this should
not be an issue for CTDB's recovery lock or Samba as they should not
need to lock ranges of less than a 4k, and they in fact integrate
CTDB for their customers.

That's right. For the recovery lock this does not matter. But for plain
file access it will. On RozoFS you should set "posix locking = no",
but then you depend on external serialization via ctdb: With "strict
locking = yes" (which is default), Samba does check byte range locks and
this way serializes access to user files on RozoFS. This serialization
however is not coordinated with the cluster file system in question,
so the RozoFS developers might have to tell us about their approach to
data coherency too.

So perhaps posix locking=no might be worth trying on MooseFS too?

I will pass on your comments in my support ticket with the MooseFS staff...

Thanks again and best regards

Alex

--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.

"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: [phone] Fax: [phone].
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).

[Alex Crow]

Hi Volker,

Post by Alex Crow

Post by Alex Crow
I've set up the "DR" side of my cluster to "use mmap = no" and with
"private dir" removed from the smb.conf.

Why do you set "use mmap = no"?

Post by Alex Crow
I have the MooseFS guys on the case as well. Should I put them in touch
with you or vice/versa?

MooseFS should not have any business at all in tdb files. If you still
find tdb files used on MooseFS, you should investigate your
configuration and modify it such that tdb files only exist on local
file systems.
Volker

There are *no* more tdb files on MooseFS, that's what I'm trying to test.

Why are you still using "use mmap = no" then?

Because the wiki article does not explicitly state that mmap coherence
is only needed for areas that store tdb files. I'd perhaps incorrectly
deduced that as you had explicitly said that "private dir" on the
cluster FS was a bad idea, that disabling that would stop all tdb files
being written to the MooseFS FS, and that would be the solution.

My understanding was the "use mmap" affected general FS operations from
clients, not just tdb files.

Post by Alex Crow
From the ping_pong wiki page it's instructed to use "use_mmap = no" if

As I sad in an earlier mail: The wiki page is wrong. If you don't trust
one of the primary developers of Samba and to a lesser extent ctdb
(myself, sorry for bragging) and trust the wiki more, then I am a bit
lost, sorry.

I am completely aware of who you and Jeremy are and I have the utmost
respect and thanks for the amazing software you have provided. We're
planning our migration to Samba 4 AD (from classic) right now.

Post by Alex Crow
"
Testing mmap coherence
If you add the -m switch to ping_pong along with -rw then it will do the
I/O coherence test via mmap. It isn't absolutely essential that a

^^^^^^^^^^^^^^^^^^^^^^^^^^
This is the key piece. It is not essential.

Again, I was not clear if it meant that "it's not essential, so if it
fails, you *must* disable mmap, if it doesn't fail, no worries but
perhaps do it just in case". However there is no mention anywhere,
including in the smb.conf man page, that it may lead to huge perfomance
impact.

Post by Alex Crow
cluster filesystem supports coherent mmap for CTDB/Samba, but it's nice
for bragging points over other cluster filesystems. If your cluster
filesystem doesn't pass this test then just use the "use mmap = no"
option in smb.conf. Even if it does pass this test that option may be a
good idea on most cluster filesystems."

This piece is wrong. The wiki should give you hints who wrote this.
Please contact the author of those lines, refer to the mail thread
here and ask these lines to be changed or even better -- change these
lines yourself.

I see what you mean here, but I'd not want to edit a wiki when I have
less idea of the facts that the person that wrote the original text! I'd
not want to update it without knowing the precise facts, hence my
reaching out to the Samba list and yourself.

Post by Alex Crow
So is this documentation wrong as well? It seems to suggest setting it

Yes, the documentation is wrong.

Post by Alex Crow
anyway, which is clearly not working for us (and now for sure there is
no involvement with TDB files as I've deleted the "private dir" in smb.conf.
The only reason I was asking if you'd like to talk to the MooseFS chaps
is because of the failure of the -rw test, which as far as the
documentation goes, seems to refer to the shared FS rather than any
places where tdb files live.

If MooseFS does not even survive -rw without asking for mmap coherence,
just don't use it for Samba. This will eat your data for breakfast.
Gluster was there loooong ago, and they fixed it before becoming a
viable storage solution for Samba.
Volker

MooseFS does not ask for mmap coherence. It has inconsistent results on
a plain -rw test. I think the chaps over there are interested in gaining
some input from your considerable experience. If they have to pay fees
via Sernet or we do, I'm not bothered. I already have them going over my
setup and checking everything, so I'm hoping it's a misconfiguration on
my part. The other choice was GlusterFS but its small-file performance
was almost an order of magnitude below MooseFS.

Many thanks

Alex
--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.

"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: [phone] Fax: [phone].
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).

[Alex Crow]

Post by Alex Crow

Why are you still using "use mmap = no" then?

Because the wiki article does not explicitly state that mmap
coherence is only needed for areas that store tdb files. I'd perhaps

Maybe that is the omission in the docs. I thought I had mentioned that
"use mmap" is only relevant for tdb files, not for plain exported
shares.

Perhaps I did not read one of your replies carefully enough, so I
apologise if that is the case.

Post by Alex Crow
incorrectly deduced that as you had explicitly said that "private
dir" on the cluster FS was a bad idea, that disabling that would
stop all tdb files being written to the MooseFS FS, and that would
be the solution.

The solution is to use ctdb with the database files on local
directories. That's what ctdb is for.

Yes, I'm clear on that now - I'm just surprised that the "private dir"
stuff is all over the internet, not just on the Samba wiki. Sometimes
this global sharing community does go a bit wrong!

Post by Alex Crow
My understanding was the "use mmap" affected general FS operations
from clients, not just tdb files.

It just affects tdb files.

Thanks for the confirmation, I was already suspecting this was the case.

Post by Alex Crow

This piece is wrong. The wiki should give you hints who wrote this.
Please contact the author of those lines, refer to the mail thread
here and ask these lines to be changed or even better -- change these
lines yourself.

I see what you mean here, but I'd not want to edit a wiki when I
have less idea of the facts that the person that wrote the original
text! I'd not want to update it without knowing the precise facts,
hence my reaching out to the Samba list and yourself.

Well, now you should have a better understanding of the issue :-)

I do indeed and shall edit the page as much as I feel confident to do so.

Post by Alex Crow

If MooseFS does not even survive -rw without asking for mmap coherence,
just don't use it for Samba. This will eat your data for breakfast.
Gluster was there loooong ago, and they fixed it before becoming a
viable storage solution for Samba.
Volker

MooseFS does not ask for mmap coherence. It has inconsistent results
on a plain -rw test.

MooseFS needs to fix that. ping_pong with -rw is pretty basic Posix
stuff. Admittedly it seems to be very hard for cluster file systems to
implement, but it is essential if you want to provide Posix semantics.
Samba and SMB clients depend on this. SMB clients coordinate via byte
range locks, and data coherence coordinated by those locks just basically
assumed.

As I said on another email that has crossed this, I have the devs
probing the problem right now. As it uses an central in-memory metadata
server for all locking, I'm hoping it will be an easy fix for them.

ping_pong was explicitly designed by Andrew Tridgell as a torture test
case for cluster file systems developers. It does not come with the
huge SMB baggage that is difficult to set up for cluster file systems
developers. The calls it makes are simple enough to be understandable
in file system traces.

Yes, I looked at the code and it is indeed terse but brutal!

Many thanks

Alex


--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.

"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: [phone] Fax: [phone].
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).