[Samba] The sad state of samba 4 adaption for home/small business routers.

[Andy Walsh]

Hi,

i would like to inspire a discussion on why samba4 has nearly no adaption on
home/small business routers firmware.

I recently switched from my old Asus-N16 running tomato to a WRT-1200AC and
was researching potential firmware's that i could run. We have a new
synology disk-station at work that runs Samba 4.x and it works great. So i
was surprised to notice that 3 years after the 4.0 release i could not find
a single home/small business oriented router firmware that was supporting
samba 4.x? I delved deeper into this and after several weeks toying around
with the tomato/dd-wrt/openWRT/buildroot/openembedded/optware2/entware
toolchains and the samba 4.3 source + waf, i understand why.


Some problems i encountered, while trying to build a samba 4 package for any
of the most popular router firmwares.

1) Waf's cross-compile approach/requirements are simply not feasible,
clearly understandable for most package maintainers.

The problem is that most maintainers can not test/create the required
qemu/crossanswer files for all the targets, that Waf relies on. As far as i
know samba4 is the only package in openWRT/buildroot that requires such a
special cross compile handling. So while i have a arm32 version running on
my AC1200, thats based on a fork of openWRT. We can not create/submit a
general package, since we lack the confidence to create crossanswer files
for all the different target platforms and creating a valid qemu host
package per target is also problematic, since no package did ever require
such a workflow.


2) The size of the final waf bin/lib install, even if stripped down to the
bare minimum of smbd/nmbd/smbpasswd is unsuitable for most older routers
that have just 8MB nvram and problematic for 16/32 MB more recent routers,
depending on what other packages are added to the firmware.

The smallest samba 4.3 package we could manage to build is around 9-12MB or
4.9MB lzma compressed using this makefile + some multicall/static linking
hacks/patches:
https://github.com/wongsyrone/openwrt-1/tree/master/package/external
/samba43related/samba43

In contrast Samba 3.6 can be stripped down to around 900kb (lzma), which is
not a problem for 8MB nvram targets.

I tried to base my build on the buildroot example and create a normal shared
lib package, since i don't like to maintain all the multicall patches and
the size gain over the shared version is just about 20%, which don't really
brings us this much closer.

So is there a way to further strip down samba4 to create a home environment
based "minimal fileserver" package? What most home users want is just a
working smb3 fileserver (s3fs) to reach windows8/10 clients and netbios so
they can browse, find the file-server. Most will not even go through the
hassle and setup user profiles/restrictions and use guest mode.
I personally also found it very intriguing to have the btrfs vfs module and
the associated transparent compression option.


Here is the current status of the big firmwares regarding samba 4:
Tomato: don't even try to get this working, since they are based on a old
kernel and lack the toolchain to even build samba 4
dd-wrt: follows openWRT
openWRT: no official package exists and the versions that where created in
forks can not be submitted, since only arm target crossanswers are used in
the makefile.
Buildroot: made some progress in the recent weeks to add more cross answer
files, but its basically a full install that is too big.
openembedded: seems to haven given up after the 4.0 experimental support
optware2/entware-ng2: no real efforts are made


So i welcome a discussion on how to help fixing this situation, so also
home/sbu can enjoy a up-to-date samba 4 version.

thx
Andy

[Reindl Harald]

Post by Andy Walsh
i would like to inspire a discussion on why samba4 has nearly no adaption on
home/small business routers firmware.
I recently switched from my old Asus-N16 running tomato to a WRT-1200AC and
was researching potential firmware's that i could run. We have a new
synology disk-station at work that runs Samba 4.x and it works great. So i
was surprised to notice that 3 years after the 4.0 release i could not find
a single home/small business oriented router firmware that was supporting
samba 4.x? I delved deeper into this and after several weeks toying around
with the tomato/dd-wrt/openWRT/buildroot/openembedded/optware2/entware
toolchains and the samba 4.3 source + waf, i understand why

most likely because there is no serious market

if i want a NAS i buy a NAS
if i want a router i buy a router

both are bad things if it comes to cheap consumer crap and combine two
bad things makes it really a terrible frankenstein box

in my case i buy a box, install linux with some iptables-rules and
whatever other services i need and that includes even wlan-accesspoints
for the last 6 years

no struggle with updates and security holes
no struggle with software versions
no struggle with "i can have this and this but not combined with that"
no struggle with "cool features but terrible slow" or otherside round
no bloatware

[Andy Walsh]

Post by Reindl Harald
most likely because there is no serious market
if i want a NAS i buy a NAS
if i want a router i buy a router
no struggle with updates and security holes
no struggle with software versions
no struggle with "i can have this and this but not combined with that"
no struggle with "cool features but terrible slow" or otherside round
no bloatware

Thats a strange argument, since up until recently home/sbu routers did not
come with 256/512MB ram, had USB3/sata3 ports and dual core arm cpu's.

All the above points are basically what i have now on my WRT-1200AC running
openWRT.
I use the latest 4.4.4 kernel + btrfs and can max out the GB ports using
smb3.1, while the system is stable and i can pick exactly what runs on the
system.

All this runs at 3-5 watts energy in a small form factor, at a very
affordable price.


bye
Andy


PS: I'm also curious what consumer router/nas combo would you buy to meet
your own requirements? There are security problems on almost all routers,
thats the reason why so many switch to openWRT/tomato/dd-wrt based
firmwares. That was also the main reason why the Asus N-16 and WRT54G became
famous.

[Reindl Harald]

Post by Andy Walsh

Post by Reindl Harald
most likely because there is no serious market
if i want a NAS i buy a NAS
if i want a router i buy a router
no struggle with updates and security holes
no struggle with software versions
no struggle with "i can have this and this but not combined with that"
no struggle with "cool features but terrible slow" or otherside round
no bloatware

Thats a strange argument, since up until recently home/sbu routers did not
come with 256/512MB ram, had USB3/sata3 ports and dual core arm cpu's.

bloatware means unsecure, uncomfortable webinterfaces with limited
functionality compared what iptables alone offers you with some knowledge

Post by Andy Walsh
All the above points are basically what i have now on my WRT-1200AC running
openWRT.
I use the latest 4.4.4 kernel + btrfs and can max out the GB ports using
smb3.1, while the system is stable and i can pick exactly what runs on the
system.
All this runs at 3-5 watts energy in a small form factor, at a very
affordable price.

and than there is a switch, some other hardware and so on

Post by Andy Walsh
PS: I'm also curious what consumer router/nas combo would you buy to meet
your own requirements? There are security problems on almost all routers,
thats the reason why so many switch to openWRT/tomato/dd-wrt based
firmwares. That was also the main reason why the Asus N-16 and WRT54G became
famous

none at all - no consumer hardware for me again in this life

my PC from 2011 plays router, switch (4 ports), 2 WLAN accesspoints with
hostapd and a single wireless card with a second fake MAC, hosts 4
permanent running virtual machines, two of them with a public IP, plays
music all day long, has 4 TB usable RAID10 storage, webserver,
fileserver, a connected SIP phone, permanent connection to 4 VPN
networks, is at the same time a mailserver and also capable to run a
full featured KDE desktop with 3D effects

with the monitor powered off the whole IT including a SIP phone that way
eats 45 watts and has horespower none of all that "embedded devices"
ever can offer, there is one ethernet cable to the modem and the whole
IT get's it's public IP addresses via DHCP inside a single box

in other words: i never ever will buy in the future any external device

and frankly if someone thinks 45 watts are to much you can achieve most
of ot with a single HP microserver with a XEON CPU with 17 watts TDP and
with hardware from 2014/2015 you end likely around 30-35 watts idle
while have the same horsepower to not need any second device at all

recently built a NAS system for Samba/NFS with such a box to have a
cheap, large storage used also as shared storage for VMware vSphere

15 TB traffic in the first week with a maximum of 15% system load

all that embedded crap is for people which needs handholding and have
fun to own a dozen of halfbaken devices instead just one real box

[Andy Walsh]

Post by Reindl Harald
bloatware means unsecure, uncomfortable webinterfaces with limited
functionality compared what iptables alone offers you with some knowledge
all that embedded crap is for people which needs handholding and have
fun to own a dozen of halfbaken devices instead just one real box

It seems you still have some misconception on what openWRT actually is and
can or can't do. You also quite underestimate what modern arm based embedded
devices can do.

I get it that you are happy and comfortable building/configuring your own
boxes by hand. In the meantime 99% of the normal home/sbu are not and hence
projects like openwrt/dd-wrt/tomato/buildroot try to bring some of your hand
crafted "magic" to the masses.

Also keep in mind that samba3.6 was successfully adopted on any device that
had a usb2.0 port in the past. So slapping a usbstick/sdcard/hdd on a cheap
router is already a common scenario.

bye
Andy

[Luca Olivetti]

Well yes, but these are usually a mass market product and will no doubt,
in the fullness of time, get to use a version of Samba 4 seeing as how
3.6 has been EOL since March 2015.

You'd be surprised to see what old shi^H^H^H "carefully crafted with
backported fixes packages" router vendors put in their products.
In a sense I'm glad they do, since due to a samba misconfiguration on
their part I could unlock a heavily locked router ;-)

Bye

[Luca Olivetti]

Post by Luca Olivetti

Well yes, but these are usually a mass market product and will no doubt,
in the fullness of time, get to use a version of Samba 4 seeing as how
3.6 has been EOL since March 2015.

You'd be surprised to see what old shi^H^H^H "carefully crafted with
backported fixes packages" router vendors put in their products.
In a sense I'm glad they do, since due to a samba misconfiguration on their
part I could unlock a heavily locked router ;-)

You shouldn't have to depend on samba misconfigurations,
GPLv3 should guarentee that you can replace the Samba
on *any* device. That's the whole point of GPLv3 - no
DRM.

Yes, I know, but it's very difficult with a telco provided router
sourced by a chinese manufacturer. In this case it is a zte f680
supplied by jazztel (now orange) in Spain to ftth customers.

In fact it's not samba that's locked but the whole router (i.e. it has a
restricted shell, but thanks to samba I could access a full shell).

If you find a Samba device using any Samba 3.2 or
newer that doesn't allow you to replace it, please
let us know so we can start the compliance process.

Unfortunately it seems it is using a really old version of samba

$ telnet x.x.x.x
Trying x.x.x.x...
Connected to x.x.x.x.
Escape character is '^]'.
F680
Login: root
Password:

BusyBox v1.01 (2015.04.27-11:59+0000) Built-in shell (ash)
Enter 'help' for a list of built-in commands.

/ # uname -a
/bin/sh: uname: not found
/ # /mnt/usb1_1_1/busybox uname -a
Linux F680 3.4.69 #366 SMP Mon Apr 27 20:21:47 CST 2015 armv7l GNU/Linux
/ # smbd -v
smbd: invalid option -- v
Usage: smbd [-D] [-p port] [-d debuglevel] [-l log basename] [-s
services file]
Version 1.9.16p10
-D become a daemon
-p port listen on the specified port
-d debuglevel set the debuglevel
-l log basename. Basename for log/debug files
-s services file. Filename of services file
-P passive only
-a overwrite log file, don't append

/ #

Bye

[Luca Olivetti]

Post by Luca Olivetti
Unfortunately it seems it is using a really old version of samba

[...]

Post by Luca Olivetti
Version 1.9.16p10
-D become a daemon
-p port listen on the specified port
-d debuglevel set the debuglevel
-l log basename. Basename for log/debug files
-s services file. Filename of services file
-P passive only
-a overwrite log file, don't append

Oh, you're hosed then :-). It's amazing that
this still works at all with modern Windows
systems.

I don't do windows at home ;-) but I cheated and I accessed it with a
winxp and win7 virtual machine, since smbclient doesn't work with it :-(


Bye

[Luca Olivetti]

I don't do windows at home ;-) but I cheated and I accessed it with a winxp
and win7 virtual machine, since smbclient doesn't work with it :-(

Hmmm. That should be an interesting bug to fix...
Maybe I'll set up a 1.9.16p10 server and fix
smbclient to work against it :-).

Don't do it for me, I have no need to access it now that it's "unlocked" ;-)
Besides, I suspect that's not really a stock 1.9.16p10 samba....

Bye

[Sketch]

GPLv3 should guarentee that you can replace the Samba
on *any* device. That's the whole point of GPLv3 - no
DRM.
If you find a Samba device using any Samba 3.2 or
newer that doesn't allow you to replace it, please
let us know so we can start the compliance process.

I wonder how that will play out with the new US FCC rules causing some
manufacturers to disallow installing 3rd party firmware:

http://arstechnica.com/information-technology/2016/03/tp-link-blocks-open-source-router-firmware-to-comply-with-new-fcc-rule/

Maybe they will just not include samba, or other GPLv3 software?

[Reindl Harald]

Post by Sketch

GPLv3 should guarentee that you can replace the Samba
on *any* device. That's the whole point of GPLv3 - no
DRM.
If you find a Samba device using any Samba 3.2 or
newer that doesn't allow you to replace it, please
let us know so we can start the compliance process.

I wonder how that will play out with the new US FCC rules causing some
http://arstechnica.com/information-technology/2016/03/tp-link-blocks-open-source-router-firmware-to-comply-with-new-fcc-rule/
Maybe they will just not include samba, or other GPLv3 software?

independent of the FCC - the logical result of "start the compliance
process" against a hardware manufacturer which sells a router and add a
goodie like samba is keep out the goodies in the next version

[Reindl Harald]

Post by Reindl Harald

Post by Sketch

GPLv3 should guarentee that you can replace the Samba
on *any* device. That's the whole point of GPLv3 - no
DRM.
If you find a Samba device using any Samba 3.2 or
newer that doesn't allow you to replace it, please
let us know so we can start the compliance process.

I wonder how that will play out with the new US FCC rules causing some
http://arstechnica.com/information-technology/2016/03/tp-link-blocks-open-source-router-firmware-to-comply-with-new-fcc-rule/
Maybe they will just not include samba, or other GPLv3 software?

independent of the FCC - the logical result of "start the compliance
process" against a hardware manufacturer which sells a router and add a
goodie like samba is keep out the goodies in the next version

That's their choice. They need to comply with all
licenses of software included in their product.
This kind of attitude "well if you *insist* people
follow your license then..." is amazing to me.

i personally don't care what any soho-equipment manufacturer does or
not, so it's not abut "attitude" - it's simple logic - and maybe the
switch to GPLv3 is the reason for the whole subject "The sad state of
samba 4 adaption"

again: i don't care about such hardware at all

What do you think would happen if such products
included Microsoft or Oracle software ? Do you
think they'd overlook license violations to
"keep the goodies in the next version" ?

no, but how does it matter what others do?

[Reindl Harald]

i personally don't care what any soho-equipment manufacturer does or not, so
it's not abut "attitude" - it's simple logic - and maybe the switch to GPLv3
is the reason for the whole subject "The sad state of samba 4 adaption"

Utter bollocks, and I have the OEMs to prove it.
"simple logic" my foot. Go peddle your lies
elsewhere please.

what the hell - stating a opinion is "peddle lies" for you?

The *only* vendor that dropped Samba over GPLv3
was Apple, who have religious objections to *any*
GPL software, and are busily re-writing anything
containing it. As is their right of course.
I'm not letting these slimy insinuations go
unchallenged

i have enough of that thread

[Sketch]

i personally don't care what any soho-equipment manufacturer does or not, so
it's not abut "attitude" - it's simple logic - and maybe the switch to GPLv3
is the reason for the whole subject "The sad state of samba 4 adaption"

If that's the case, it would need to be "The sad state of samba 3.2+
adoption", since that's when the license was changed...4 years before
Samba 4 was released.

[Reindl Harald]

Post by Andy Walsh

Post by Reindl Harald
bloatware means unsecure, uncomfortable webinterfaces with limited
functionality compared what iptables alone offers you with some knowledge
all that embedded crap is for people which needs handholding and have
fun to own a dozen of halfbaken devices instead just one real box

It seems you still have some misconception on what openWRT actually is and
can or can't do. You also quite underestimate what modern arm based embedded
devices can do.
I get it that you are happy and comfortable building/configuring your own
boxes by hand. In the meantime 99% of the normal home/sbu are not and hence
projects like openwrt/dd-wrt/tomato/buildroot try to bring some of your hand
crafted "magic" to the masses

no, it's not for the masses, for the masses is what comes out-of-the-box

there is not much difference between replace the operating system on a
mass-product and just setup my own box from-scratch in the end

i know what modern arm based devices can do

the HP microserver comes with a
http://ark.intel.com/de/products/65735/Intel-Xeon-Processor-E3-1220L-v2-3M-Cache-2_30-GHz
which has a TDP of 17W, 2.3 GHz x86_64 with AES-NI and turbo with 3.5
GHz will blow away any embedded device and so finally that machine can
do so much more in one box with horsepower, intergated large and fast
storage and you only need to decide if you use the PCI-Express Port for
a 4-Port-Switch or as WLAN-AP while you most likely can chose the
4-Port-NIC which gives you combined with the interated Dual-Port finally
a 5-Port-Switch with a WAN-Interface and theres surely some
USB3-WLAN-Adapter on the market with is capable to run a
WLAN-Accesspoint with "hostapd"

the operating system lives on a micro sd-card and with 8 GB RAM and 4
hard-drive bayes you can do a lot of things with that box

not to forget with 5 Ethernet cards that you can build up different
brdiges/routings and if you need 10 Gigabit, well, place a 10 Gigabit
4-Port-Card in the Slot and use the interal Gigabit NIC for the WAN

[Andy Walsh]

Post by Reindl Harald
no, it's not for the masses, for the masses is what comes out-of-the-box
there is not much difference between replace the operating system on a
mass-product and just setup my own box from-scratch in the end
i know what modern arm based devices can do
the HP microserver comes with a

http://ark.intel.com/de/products/65735/Intel-Xeon-Processor-E3-1220L-v2-3M-Cache-2_30-GHz

Post by Reindl Harald
which has a TDP of 17W, 2.3 GHz x86_64 with AES-NI and turbo with 3.5
GHz will blow away any embedded device and so finally that machine can
do so much more in one box with horsepower, intergated large and fast
storage and you only need to decide if you use the PCI-Express Port for
a 4-Port-Switch or as WLAN-AP while you most likely can chose the
4-Port-NIC which gives you combined with the interated Dual-Port finally
a 5-Port-Switch with a WAN-Interface and theres surely some
USB3-WLAN-Adapter on the market with is capable to run a
WLAN-Accesspoint with "hostapd"
the operating system lives on a micro sd-card and with 8 GB RAM and 4
hard-drive bayes you can do a lot of things with that box
not to forget with 5 Ethernet cards that you can build up different
brdiges/routings and if you need 10 Gigabit, well, place a 10 Gigabit
4-Port-Card in the Slot and use the interal Gigabit NIC for the WAN

I still think you completely miss the point here.
One of the main ideas here is to transform a cheap 50-100$ device into
something that can do as much as possible on the given hardware. So ofc your
setup that cost 3-10 times more can do more and is faster in some areas.

I specifically asked about home/sbu, yet you keep throwing in server class
hardware in this discussion.
So whats the point you trying to make here and how does this help actually
solving the problems i org. asked about?

thx
Andy

[Reindl Harald]

Post by Andy Walsh
I still think you completely miss the point here.
One of the main ideas here is to transform a cheap 50-100$ device into
something that can do as much as possible on the given hardware. So ofc your
setup that cost 3-10 times more can do more and is faster in some areas.
I specifically asked about home/sbu, yet you keep throwing in server class
hardware in this discussion.
So whats the point you trying to make here and how does this help actually
solving the problems i org. asked about?

than throw
http://www.amazon.de/Celeron-QuadCore-DDRIII-e-Mini-fanless/dp/B00JSA3R30/ref=sr_1_1?ie=UTF8&qid=1433334695&sr=8-1&keywords=mini+pc+2+lan
in the mix, buolt last year a small office with that device (router,
firewall, small samba server, vpn-gateway) and while it's near your 100$
class of hardware it has much more power and flexibility

the point i am trying to make here is that the usefulness of absue a
home-router as fileserver is overestimated

[Andy Walsh]

Post by Andy Walsh
Also keep in mind that samba3.6 was successfully adopted on any device that
had a usb2.0 port in the past. So slapping a usbstick/sdcard/hdd on a cheap
router is already a common scenario.

Yes, and I'd like to help you do the same for Samba4.x also.
Stop moaning about it, and start helping *do* it .

Oki, i just noticed that i have missed the work openembedded did in the last
months, since they seem to have solved the problem in a general way in there
trunk.
https://goo.gl/jixOqE

Its still the most complicated setup any cross compile package uses, but
this looks good. So i guess we now need to convert this to a openWRT (based
on buildroot) compatible setup.

If i understand the waf-samba.bbclass correctly they build the qemu for the
host system and configure it to use the staging_dir environment to generate
the answer files.

So first i need to create a qemu package for openWRT, since currently only a
qemu-guest package is available. I will try to get this running in the next
days and see what problems pop up adapting the openembedded logic.

Btw can you take a look and the config and check if there is anything
missing to reduce the size?
https://goo.gl/fMDbIX

thx
Andy

[Andy Walsh]

I'll take a look next week (technically it's my weekend and
I should be enjoying it, daylight savings time-change notwithstanding .
I'm not sure qemu should be required. I've worked with systems
getting S4 x-compiled without it - just correct choices in the
answer files.

The problem is that openWRT/buildroot needs to-be able to support as much
targets as possible in a single package/makefile.
https://git.openwrt.org/?p=openwrt.git;a=blob_plain;f=target/Config.in
https://git.openwrt.org/?p=openwrt.git;a=tree;f=target/linux

The way openWRT creates images for routers is to cross compile multiple
targets at once and than making the img + cross compiled optional packages
available for download.

New targets are also added frequently, so from a package maintainer point of
view we have no real clue what arch/kernel/basesystem combo we will get, so
preparing answer files seems problematic. Ofc atm we could manually create
files for the most used targets, but the point would be to make the whole S4
cross compile setup fool proof.

Atm you can grab the source from git, do a make menuconfig and all packages
should compile for all platforms/targets.


thx
Andy