Coova chilli throughput

[gareth41]

I'm on a 100mbit fiber optic line and have been testing throughput on coova-chilli 1.3.0 using different routers loaded with openwrt barrier-breaker. I've been using speedtest.net to get my results and cabling everything in, no tests over wifi. I'm getting around 45mbits on a TP-Link TL-WR1043ND v2, the CPU is 720Mhz with 64mb ram. I also tried on a Mikrotik RB951G-2hnd and got roughly the same. Ubiquiti Air-router gets less, about 30mbits. If I bypass chilli and use straight nat lan to wan on the routers with openwrt I get very close to my line speed of 100mbits. Is there anything that can be done to maximize throughput when using coova chilli? I also tried wifidog and there's no problems with throughput, I tested wifidog 1.2.1, the only problem is that it doesn't have all the features chilli does.

[sevan]

@gareth41 when you say coova-chilli 1.3.0, are you referring to the release from 2012 obtained from coova.org?
Have you tried building the latest code from a git checkout?

[gareth41]

Yes that's correct, using the 2012 version from coova.org. I'll try the latest code from git checkout tomorrow and report back

[gareth41]

I just tried to compile this for openwrt barrier_breaker, have added as a package with autoreconf in the makefile, but getting a stop error with no makefile found, am I missing something else that needs to be added?

Edit:
Managed to compile it, I hadn't run bootstrap and was also missing gengetopt. apt-get install gengetopt.

I have also edited redir.c to allow andriod/apple devices to stay connected to wifi without them bringing up the captive portal automatically on joining the network. Androids in particular will automatically disconnect from any public wifi hotspot after approximately 1 minute if the user doesn't log into the captive portal. Some users need longer than 1 minute to fill out register/payment forms on some public hotspots. The only caveat in doing this is the requirement for the user to then open their browser manually to bring up the captive portal. I can share this if anyone wants it.

[sevan]

Maybe push the changes to your fork on github & mention it on the mailing list, you'll get a clear indication of if should be included mainline.
I guess you're still benchmarking network performance.

[gareth41]

Yes been testing different configuration etc... with no change. Still can't get any more than around 45mbits on the tp-link. Also tried the good old linksys wrt54gl just out of interest, and with coova-chilli it struggles to reach 11mbits due to its slow processor, straight nat lan to wan on this is around 40mbits though.

Have also forked and made the changes to redir.c as per above.

[sevan]

Great, well done :)
I guess the next thing to test if you have the inclination is to run it on a faster box (x86?) and see if the throughput improves.

[gareth41]

Thanks, yes i've got a few spare boxes sitting around so will try this also.

Also found a bug relating to the md checksum sent in the query string from the http aaa proxy to auth server. Chilli seems to hash the auth server URL without the port number added to the URL, chilli then adds the port to the http "Host" header later. I grab the auth server address using $_SERVER['HTTP_HOST'] in my auth server php code and have to remove ":80". Looking at main-proxy.c, an easy fix if using the standard port 80 is to change the following to exclude the port number in the http request.

bassignformat(req->wbuf,
"GET %s HTTP/1.1\r\n"
"Host: %s:%d\r\n"
"User-Agent: CoovaChilli " VERSION "\r\n"
"Connection: close\r\n"
"\r\n",
req->url->data + uripos, hostname, port);

[bingyunxl]

recently, we test chilli(version 1.3.0 and master)

1. with CPU: QCA9531, Memory: 128M, without chilli, the router throughput is 96Mbps, CPU usage is 25%, run chilli, the throughput is 34Mbps, CPU usage is 100%.

2. with CPU: intel atom D525, Memory:1G, without chilli, the router throughput is 980Mbps, CPU usage is 10%, run chilli, the throughput is 543Mbps, CPU usage is 50%.

[AndrewMargarit]

@bingyunxl same results on both 1.3.0 and baster? or the master results are due to come?

[bingyunxl]

@AndrewMargarit Almost two results are same, so.......

[bingyunxl]

@AndrewMargarit maybe, use kmod-tun and copy packets is this performance issue, when our team test chilli throughput,we found the CPU's user and sys is high than normal

[gareth41]

Same here the CPU tops out at close to 100% with high throughput over the tun/tap device. Apparently OpenVPN suffers the same problem as it too uses a tun/tap device which brings me to my next question: Is encryption enabled on the chilli tun/tap device, and if so why not disable it? This would allow somewhat better throughput than what is currently achieved.

[nzamps]

Using xt_coova makes a significant difference.

[bingyunxl]

@nzamps we will try xt_coova, when we finish build and test this issue, I will post the result here

[gareth41]

ive just finished building it on openwrt trunk and testing on the tplink again (TL-WR1043ND v2), with no difference, still 45mbits. I'm assuming it compiled without problems, no errors were displayed after compiling. "--with-nfcoova" was specified in the Makefile and kmod-ipt-coova selected in make menuconfig, chilli shows compiled with HAVE_NETFILTER_COOVA when run in ssh console. kmod-ipt-coova is also installed. I'm not sure if i'm missing something?

[nzamps]

Firewall rules for xt_coova most likely. Unfortunately, xt_coova is not an option that generally works without some changes. See:

http://lists.coova.org/pipermail/chilli/2010-April/001239.html

[gareth41]

Thanks for the info, got it all loaded now however running into the same problem exactly as described here: http://lists.coova.org/pipermail/chilli/2014-April/002661.html

Client is redirected to chilli webserver using http 302 but connection is reset when trying to access the login script. If I manually authenticate client, everything works and the throughput is significantly faster!

There are also loads of ARP messages in the log:
dhcp.c: 5833: 0 (Debug) ARP: XX-XX-XX-XX-XX-XX asking about 10.1.0.12
dhcp.c: 5902: 0 (Debug) ARP: Did not ask for router address: 10.1.0.1
dhcp.c: 5903: 0 (Debug) ARP: Asked for target: 10.1.0.12
and this continues on, hundreds of lines.

The coovachilli version i'm using is from 28-11-2014, commit b93de20. I haven't tried to compile anything newer with xt_coova as there's terrible memory problems in new versions as documented in issue #85.

For now iv'e been sticking with the previous release 1.3.0 from 2012 for production as its rock solid for what we're using it for and has never experienced issues apart from the lack of throughput, hence this thread i'm posting in now.

[nzamps]

Might need to see your config and firewall rules. Ensure you have the dhcplisten IP on the LAN interface and the NAT'd address (e.g. 11.x) on the tun - on older versions the IP on the LAN interface was deleted on startup.

[gareth41]

Thanks, I checked the dhcplisten IP on LAN interface, it hasn't changed or been removed. Its set to 10.1.0.1. Also have 11.1.0.1 set on the tun.

br-coova Link encap:Ethernet HWaddr XX
inet addr:10.1.0.1 Bcast:10.1.255.255 Mask:255.255.0.0
inet6 addr: fe80::eade:27ff:fee3:fae3/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:408293 errors:0 dropped:0 overruns:0 frame:0
TX packets:482957 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:353279261 (336.9 MiB) TX bytes:558582372 (532.7 MiB)

tun0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
inet addr:11.1.0.1 P-t-P:11.1.0.1 Mask:255.255.0.0
UP POINTOPOINT RUNNING MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:100
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

I have br-coova as the chilli lan interface, bridged to port 1 so I can test cabled devices, the other ports are bridged to the default br-lan so I can retain ssh access to the router while testing chilli. I'm also using chilli in httpaaa mode.

Here is my config, i've only removed the info relating to the VPS where my auth server is located.
root@OpenWrt:/etc/chilli# cat main.conf

THIS FILE IS AUTOMATICALLY GENERATED

cmdsocket /var/run/chilli.br-coova.sock
unixipc chilli.br-coova.ipc
pidfile /var/run/chilli.br-coova.pid
net 10.1.0.0/16
uamlisten 11.1.0.1
uamport 3660
dhcpif br-coova
uamallowed "www.coova.org,xxxxxxxxx,11.1.0.1/32,10.1.0.1/32"
uamanydns
lease 3600
dhcplisten 10.1.0.1
dhcpstart 10
dynip 10.1.0.0/24
statip 10.1.1.0/24
nasip "0.0.0.0"
domain "lan"
dns1 "8.8.8.8"
dns2 "8.8.4.4"
uamhomepage http://11.1.0.1:3660/www/login.html
wwwdir /etc/chilli/www
wwwbin /etc/chilli/wwwsh
locationname "xxxxxx"
radiuslocationname "xxxxxx"
radiuslocationid "isocc=,cc=,ac=,network=xxxxxx,"

root@OpenWrt:/etc/chilli# cat hs.conf
radiusserver1 "localhost"
radiusserver2 "localhost"
radiussecret "testing123"
radiusauthport 1812
radiusacctport 1813
uamserver "http://xxxxxxxxxxxx/auth.php"
radiusnasid "66"
uamaaaurl "http://xxxxxxxxxxxxxx/aaa.php"
papalwaysok
macauth
acctupdate
uamaliasname "chilli"
adminupdatefile "/etc/chilli/local.conf"
uamsecret "testing123"

[gareth41]

Also I should have mentioned i've been running chilli with the --kname=on switch, without this everything works but all routing is still through chilli hence less throughput. with --kname=on I experience the issue described above

[nzamps]

Just had a quick look:

For bridges you'd probably need to use the 'usetap' option or try without the bridge.

[gareth41]

Thanks i'll give it a try.

I also added the following firewall rules:
iptables -I FORWARD -o br-wan --src 10.0.0.0/8 -m coova --name chilli -j ACCEPT
iptables -I FORWARD -i br-wan --dst 10.0.0.0/8 -m coova --name chilli --dest -j ACCEPT

iptables -I FORWARD --src 11.0.0.0/8 -j ACCEPT
iptables -I FORWARD --dst 11.0.0.0/8 -j ACCEPT

I have the wan in bridge mode too, i'll also try this without bridging

[bingyunxl]

using xt_coova, the "data flow control" function will be disabled, Is't not a good idea

[ankitherocker]

@gareth41 : Hello, may I have the changes you did to redir.c to disable captive portal on Android & iPhone & let them connect to hotspot via browser in couple of minutes? I was looking for it since a long time & found your comment here regarding the same.

Can you please help?
Thanks

[gareth41]

https://github.com/gareth41/coova-chilli/blob/master/src/redir.c#L1551

These are the changes for android, the only reason I made these changes was due to Androids only giving their users approx 60sec to complete captive portal login before disconnecting the wifi. Some people need longer than 60sec if they are required to signup for an account or fill out paypal/credit card forms etc... Apple seems ok, well atleast with the newer ios versions, no timeout i've noticed. I can give you the changes for apple if you still want them.

[ankitherocker]

Thanks a ton @gareth41 I need it because there is an unusual behaviour on some Android Phones. Some show captive window & some doesn't. So I have developed an App for the users to login but this captive portal windows pops up on some phones & deceits the App purpose. I want to keep it simple, no Captive Window only login through App.
Yeah would be great if I can get Apple changes as well. Would it work on Macs as well?

[gareth41]

Here's the code to trick ios devices, however I don't know about mac books etc as I don't have one to test. You could try running chilli in debug mode with -d -f switches and grab the redirect url from the log when you try to connect using a mac book.

if ((strcmp(userurl, "http://www.apple.com/library/test/success.html") == 0) ||
    (strcmp(userurl, "http://captive.apple.com") > 23) ||
    (strcmp(userurl, "http://www.ibook.info") > 20) ||
    (strcmp(userurl, "http://www.appleiphonecell.com") > 29) ||
    (strcmp(userurl, "http://www.airport.us") > 20) ||
    (strcmp(userurl, "http://www.thinkdifferent.us") > 27) ||
    (strcmp(userurl, "http://www.itools.info") > 21)) {
redir_http(buffer, "200 OK");
bcatcstr(buffer, "Content-Type: text/html; charset=UTF-8\r\n");
bbody = bfromcstralloc(512,
                       "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2//EN\">"
                       "<HTML>"
                       "<HEAD>"
                       "        <TITLE>Success</TITLE>"
                       "</HEAD>"
                       "<BODY>"
                       "Success"
                       "</BODY>"
                       "</HTML>\r\n");


} 

[mpangrazzi]

@ankitherocker @gareth41 Modifying redir.c in order to disable the captive portal behaviour on Android and Apple devices works, but it's not a smart solution.

It's way better to detect all URLs that those devices are calling when connected (e.g. captive.apple.com), then put them in HS_UAMALLOW configuration directive.

[gareth41]

Yes I agree with that, but for android devices when allowing the urls they check, it seems to open up Google.com and other Google sites like YouTube. This also brings me to another but unrelated issue I discovered today, with the change in DNS of coova.org recently to a Google owned ip, chili seems to allow Google.com and a handful of other Google sites through its firewall because coova.org is automatically added to the uam domains regardless of config. I've written a patch to fix this, just a simple edit of functions.in to remove coova.org from uam domains

[ankitherocker]

@gareth41 : +1: This is the exact issue I faced. Thanks for the patch & for now it seems to be the only way to handle it. I also see 1 more problem with this patch though, if Google or Apple (iOS6 had only 1 & iOS8 has 7 url checks) adds more domains to the list. It is going to be a miss & compiling chilli & upgrading the production systems with the patch would be tuff. What say?

[ankitherocker]

@gareth41 : Also, did you find out the reason for the disparity in chilli throughput? I recently tested the same on OpenWrt + CoovaChilli v/s DD-Wrt + Chillispot & the latter wins the throughput. Still trying to figure out why throughput drops on the first combo ~ OpenWrt + Coovachill & I see no reason to it for now.

[geohuz]

I deployed a large scale coovachilli application on our city's telecom ISP network, we have a 1g fiber link to the base backbone device, about 300 vlans on the nic which links to lay2 switch and then downlink to several AC network aggregation devices. I found the CPU usage and overall load was so high, and I have done several tweaks to avoid the chilli server to stop function (on the radius side I can see the login amounts sharp decline and I have to manually start the service several times in a day). I mainly change the tcp options in the sysctrl.conf, now the server can continue running without a restart at least for one or two days, but the load is still high, the performance is my big concern, I have also tried the newest version from the github, no obvious improvement, can you guys point out a direction to optimise the operating system or coovachilli? BTW, I have 16g memory and 64g disk space on the server.

[gareth41]

Our company is considering the option of hiring an experienced developer to solve these issues, but is going to cost us cost us thousands hence any developments/fixes unfortunately will have to be kept within the company. My knowledge of c/c++ is still quite limited as I only started learning this year, I've always been a php/mysql person. I'm happy to try and fix these issues with the help of people like David Bird and work together for the benefit of everyone, but unfortunately I kindly say that he doesn't seem to have the dedication and time to put into this project as he once did many years ago, he hasn't even commented in this thread regarding throughput, nor most other threads/issues people have raised. The old Coova forums and mailing lists used to be full of comments from David helping people.

[nzamps]

The issue is well known, the cost of copying the packets to userspace (tuntap) is a major performance hit - the same hit you seen with OpenVPN etc. A solution to pass the authenticated packets through the kernel (like how xt_coova works) is the only real viable solution that doesn't involve rewriting everything to not use a tuntap.

[ankitherocker]

If that's the case why would chillispot's performance seems to be better in terms of througput?

Regards,
Ankit Verma

Sent from my iPhone

On Sep 25, 2015, at 4:24 AM, Brian [email protected] wrote:

The issue is well known, the cost of copying the packets to userspace (tuntap) is a major performance hit - the same hit you seen with OpenVPN etc. A solution to pass the authenticated packets through the kernel (like how xt_coova works) is the only real viable solution that doesn't involve rewriting everything to not use a tuntap.

—
Reply to this email directly or view it on GitHub.

[nzamps]

How much difference do you see ? - from our testing it was fairly marginal compared with the difference when not running coovachilli or chillispot. Regardless, the code base's are quite different now so not easy to do a straight comparison.

[ankitherocker]

Well, I dint get a chance to test it on a fiber, would be doing it next week. But, on BB connection I got the same speed as my WAN running ddwrt+chillispot as compared to openwrt+chilli where I got ~30% drop in bandwidth. Seems strange to me as well.

Regards,
Ankit Verma

Sent from my iPhone

On Sep 25, 2015, at 5:41 AM, Brian [email protected] wrote:

How much difference do you see ? - from our testing it was fairly marginal compared with the difference when not running coovachilli or chillispot. Regardless, the code base's are quite different now so not easy to do a straight comparison.

—
Reply to this email directly or view it on GitHub.

[wlanmac]

Thus far, this issue hasn't really qualified as a bug (at best a vague feature request). Yes, there is a throughput hit using any chilli - or any tun/tap tunnel. A feature to feature comparison of chillispot is interesting (got numbers?) -- i'd hope coova-chilli is better, it has improvements in packet buffer management, etc. Of course, it depends greatly on what features in coova-chilli are used as well. Larger networks will often take chilli off the AP. Some use the "chilli-redir" feature on multi-core systems, and maybe remove logging (which is something we do need to improve in master -- to support compile time features to strip out logging meant for debugging). In the past, I have spent a lot of time with valgrind trying to squeeze out performance, and that will continue as work goes on. We should also spent time making xt_coova robust enough for default configurations. In terms of hiring someone to address problems, great! Even better if you give it back to the community.

[sevan]

The CPU details of the hardware listed in the first post

TP-Link TL-WR1043ND v2
Qualcomm Atheros QCA9558 720 MHz CPU
MIPS 74Kc

Mikrotik RB951G-2hnd
Atheros AR9344 600MHz CPU
MIPS 74Kc

Ubiquiti Air-router
Atheros AR7241 400 MHz CPU
MIPS 24Kc

[liudf0716]

I don't think it is a simple task to solve the performance problem of chilli

[liudf0716]

it's better to use wifidog in MIPS router , to use chilli, at least in x86 platform

[xewonder]

Hello all..

Has the performance/speed issue ever been solved?

[maxxer]

I'm running 1.6 on mips64 with kernel 4.9.79, and despite being gro on or off, the bandwidth gets trimmed. I don't have the chance to test on x86 right now, anyone has similar experiences, and possibly found a solution? thanks