on high load chilli stop give ip #185

dobrozhanskyi · 2015-12-27T21:41:37Z

Сhiilli running on OpenWRt 15.05.

When I start the torrent nobody else can get ip. PPS ~ 5k. CPU 100%.
I launch on the same device dnsmasq and on same load receive ip from dnsmasq( but get in the debug "Received packet with spoofed source").

With a high number of pps is normal that stops working dhcp?

config chilli 'test'
option tundev 'tun0'
option kname 'chilli'
option uamlisten '101.101.101.1'
option dhcplisten '101.102.103.1'
option dynip '101.102.103.0/24'
option dhcpstart '2'
option dns1 '8.8.8.8'
option dns2 '8.8.4.4'
option radiussecret ''
option network 'lan'
option uamsecret 'uamsecret'
option ipup '/etc/chilli/up.sh'
option ipdown '/etc/chilli/down.sh'
option dnslog '/var/log/111'
option macauth '1'
option macpasswd '****'
option radiusserver1 ''
option radiusserver2 '****'
option uamserver 'https://*************/'
option uamdomain '.facebook.com,.fbcdn.net,.akamaihd.net'

sevan · 2015-12-28T21:19:57Z

What happens if you

run both chilli & your torrent client using nice
eg nice chilli
run you torrent client with a lower priority (using a higher number value)
eg nice 20 rtorrent

Still the same behaviour?

dobrozhanskyi · 2015-12-29T11:22:23Z

Torrent run on client device (not on router).
I run chilli with nice -20 and problem still exist.

sevan · 2015-12-29T11:56:19Z

Ah, sorry, I assumed you're doing both on the same box. What you're describing is that when chilli is maxing out your CPU while forwarding, it's unable to serve other requests.
Next thing to look at is if it's purely a resource issue (if CPU resources were available, would things work ok) or that there are limitations within the code.
Either try running chilli on a faster system or using something like cpulimit https://github.com/opsengine/cpulimit

shkrid · 2016-01-03T22:53:43Z

@sevan
I have the same problem. But if I got the IP address before logged in - redirect stops working at 100% CPU load(100Mbps of traffic passing through Coova-chilli on Carambola2 board)
Is it possible to set in a priority issue dhcp address leases, redirection and authorization? Maybe I missed some compile options that would allow Chilli to work in non-blocking mode?
Since the problem is not in the Chilli under heavy load, but in getting the IP, redirection and authorization.

For example, if 5 users connect to access point, logged in, and started download 20Mbps each, then 6-th user can neither connect to AP nor authorize in Chilli.

Thanks... Hope for your help

sevan · 2016-01-04T02:41:53Z

is the issue there if you run 1.3.0?

gareth41 · 2016-01-04T05:04:31Z

I run the old 1.3.0 from 2012 and have never had any of these problems its rock solid, I run a commercial hotspot operation at a lot of accommodation places and this is what we use. I tried several versions/commits from github back in 2014/15 and only ended up with issues so abandoned it, we’re sticking with the old 1.3.0 for now.

From: Sevan Janiyan [mailto:[email protected]]
Sent: Monday, 4 January 2016 3:42 p.m.
To: coova/coova-chilli
Subject: Re: [coova-chilli] on high load chilli stop give ip (#185)

is the issue there if you run 1.3.0 http://coova.github.io/coova-chilli/coova-chilli-1.3.0.tar.gz ?

—
Reply to this email directly or view it on GitHub #185 (comment) . https://github.com/notifications/beacon/AL0MnZb4O8Ug68xU9X0C7411a43VAHwCks5pWdN0gaJpZM4G7pBq.gif

shkrid · 2016-01-04T09:11:48Z

@gareth41
Do you use kmod(xt_coova) in your HotSpot? Can you show your configuration?
When I disable the kmod(xt_coova) - I have everything working.
Chilli issues the IP addresses, even when 100% CPU utilization, but throughput below 30Mbps on the Carambola2 board #61
That's why I think that the problem is not in the CPU load.

top without kmod(xt_coova) during load(I can get ip):

PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
Mem: 31680K used, 29516K free, 736K shrd, 3252K buff, 11604K cached
CPU:  18% usr  38% sys   0% nic   0% idle   0% io   0% irq  43% sirq
Load average: 0.69 0.74 0.54 2/34 3890
  PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND
 3826     1 root     R     1524   2%  98% chilli
........

top with kmod(xt_coova) during load:

Mem: 31364K used, 29832K free, 740K shrd, 3252K buff, 11608K cached
CPU:   7% usr  11% sys   0% nic   0% idle   0% io   0% irq  81% sirq
Load average: 0.89 0.34 0.34 2/35 4015
  PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND
 3943     1 root     R     1528   2%  90% chilli
.....

I can't get ip already. May be the problem related to kmod(xt_coova) or to sirq?

But I need the maximum throughput so I'm forced to use kmod (xt_coova)

@sevan
I tried 1.3.0, which in the source OpenWrt

PKG_VERSION:=1.3.0+20141128
PKG_SOURCE_URL:=git://github.com/coova/coova-chilli
PKG_SOURCE_VERSION:=b93de20a288c01c2ba28e96e31ad6da01627f45f

Also one of the latest versions of:

PKG_VERSION:=git_a4a67f19
PKG_SOURCE_URL:=git://github.com/coova/coova-chilli
PKG_SOURCE_VERSION:=a4a67f193e6a8087050a8148e26c5e03f5dd12b8

Same result. :-(

Anything advise?
Thank you.

P.S. With regard to redirection and authorization. I'm probably exaggerating the problem as the redirection after a long delay but it is certainly going on and it is related with the link loading probably.
But the main problem of the issue of the IP addresses are relevant.

sevan · 2016-01-04T15:31:05Z

@shkrid as you asked, with what flags/features did you build coova?

shkrid · 2016-01-04T16:28:59Z

@sevan
./configure --target=mips-openwrt-linux --host=mips-openwrt-linux --build=x86_64-linux-gnu --program-prefix="" --program-suffix="" --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --libexecdir=/usr/lib --sysconfdir=/etc --datadir=/usr/share --localstatedir=/var --mandir=/usr/man --infodir=/usr/info --disable-chilliproxy --disable-chilliredir --disable-dnslog --disable-miniportal --disable-useragent --disable-largelimits --disable-uamdomainfile --without-matrixssl --without-cyassl --without-openssl --with-nfcoova

sevan · 2016-01-04T20:57:14Z

This is the board you're using?
Without chilli involved, how much traffic can you pass between two systems through this device?

shkrid · 2016-01-04T21:15:26Z

This is the board you're using?

Yes.

Without chilli involved, how much traffic can you pass between two systems through this device?

It has 2x10/100Base-T(X) ports [eth1 - wan, eth0 - lan ]
Without chilli - wan=>lan - 100Mbps
Chilli without xt_coova - wan=>lan - ~30Mbps
Chilli with xt_coova - wan=>lan - 100Mbps

sevan · 2016-01-05T00:09:34Z

Impressed, do you have systemtap or perf on that system?

dobrozhanskyi · 2016-01-05T13:56:41Z

After I made a dhcp on dnsmasq I see another problem - at a load stops working redirect to the captive portal (unauthorized users go to the requested resource).

In debug I see many records as (00-1F-F3-BF-56-56 - my mac):
Tue Jan 5 13:48:03 2016 local6.debug coova-chilli[5594]: dhcp.c: 3830: Not for our MAC or broadcast: 00-1F-F3-BF-56-56
Tue Jan 5 13:48:03 2016 local6.debug coova-chilli[5594]: dhcp.c: 3830: Not for our MAC or broadcast: 00-1F-F3-BF-56-56
Tue Jan 5 13:48:03 2016 local6.debug coova-chilli[5594]: dhcp.c: 3830: Not for our MAC or broadcast: 00-1F-F3-BF-56-56

Also, I can not understand why the authorized users when enabled xt_coova load process chilli to 100% .

shkrid · 2016-01-06T15:11:29Z

Impressed, do you have systemtap or perf on that system?

@sevan
I compiled image with perf support.
How to use it? With wich parameters to start? What i should to do?

sevan · 2016-01-06T17:18:31Z

Ok. Coova needs to be built with debug symbols (-G) either set your CFLAGS variable with -G during the configure stage or add AM_CFLAGS += -G to configure.ac & re-bootstrap.

then try perf record -g chilli
You'll get a file called perf.data which has the profiling information. You can then analyze the information using perf report --stdio
Capture the data & share it so we can see what's going on :)

shkrid · 2016-01-06T17:30:03Z

@sevan
I use such commands:
perf record -p <pidof chilli> sleep 60
perf.data: perf.data.zip

Screenshots:

root@OpenWrt:/# perf report
# Overhead|Command|Shared Object         |Symbol                                 
9.38%|chilli |[ip_tables]           |[k] ipt_do_table                       
4.19%|chilli |[kernel.kallsyms]     |[k] ag71xx_poll                        
2.68%|chilli |[kernel.kallsyms]     |[k] dev_hard_start_xmit                
2.67%|chilli |[kernel.kallsyms]     |[k] __netif_receive_skb_core           
2.63%|chilli |[kernel.kallsyms]     |[k] nf_iterate                         
2.57%|chilli |[kernel.kallsyms]     |[k] ag71xx_hard_start_xmit             
2.48%|chilli |[kernel.kallsyms]     |[k] ip_forward                         
2.42%|chilli |[kernel.kallsyms]     |[k] __dev_queue_xmit                   
2.41%|chilli |[nf_conntrack]        |[k] nf_conntrack_in                    
2.27%|chilli |[kernel.kallsyms]     |[k] r4k_dma_cache_inv                  
2.25%|chilli |[kernel.kallsyms]     |[k] ip_rcv                             
2.14%|chilli |[kernel.kallsyms]     |[k] eth_type_trans                     
1.87%|chilli |[nf_conntrack]        |[k] __nf_conntrack_find_get            
1.79%|chilli |[kernel.kallsyms]     |[k] __build_skb                        
1.76%|chilli |[kernel.kallsyms]     |[k] ip_finish_output                   
1.74%|chilli |[kernel.kallsyms]     |[k] __local_bh_enable_ip               
1.63%|chilli |[kernel.kallsyms]     |[k] ag71xx_tx_packets                  
1.61%|chilli |[kernel.kallsyms]     |[k] ip_output                          
1.57%|chilli |[kernel.kallsyms]     |[k] __slab_alloc.isra.74.constprop.78  
1.37%|chilli |[nf_conntrack]        |[k] tcp_error                          
1.34%|chilli |[kernel.kallsyms]     |[k] __slab_free.isra.75                
1.19%|chilli |[kernel.kallsyms]     |[k] __copy_user_common                 
1.14%|chilli |[kernel.kallsyms]     |[k] r4k_dma_cache_wback_inv            
1.13%|chilli |[kernel.kallsyms]     |[k] br_handle_frame                    
1.09%|chilli |[kernel.kallsyms]     |[k] __bzero                            
1.07%|chilli |[kernel.kallsyms]     |[k] __do_softirq                       
1.05%|chilli |[xt_tcpudp]           |[k] tcp_mt                             
1.00%|chilli |[kernel.kallsyms]     |[k] memcmp                             
0.98%|chilli |[nf_conntrack_ipv4]   |[k] ipv4_pkt_to_tuple                  
0.95%|chilli |[nf_conntrack]        |[k] tcp_packet                         
0.92%|chilli |[kernel.kallsyms]     |[k] ag71xx_fill_rx_buf                 
0.91%|chilli |[kernel.kallsyms]     |[k] __kmalloc                          
0.90%|chilli |[kernel.kallsyms]     |[k] packet_rcv                         
0.87%|chilli |[kernel.kallsyms]     |[k] kmem_cache_free                    
0.82%|chilli |[kernel.kallsyms]     |[k] nf_hook_slow                       
0.82%|chilli |[nf_conntrack]        |[k] tcp_pkt_to_tuple                   
0.75%|chilli |[xt_coova]            |[k] coova_table_lookup                 
0.73%|chilli |[kernel.kallsyms]     |[k] br_dev_xmit                        
0.72%|chilli |[kernel.kallsyms]     |[k] skb_put
[....]

chilli config:

root@OpenWrt:/# cat /etc/chilli.conf 
ipup=/etc/chilli/up.sh
ipdown=/etc/chilli/down.sh
kname   chilli
dhcplisten 101.102.103.1
uamlisten   101.101.101.1
dhcpif      br-lan
net 101.102.103.0/24
dns1 "8.8.8.8"
localusers /etc/chilli/localusers
radiusserver1   "127.0.0.1"
radiussecret    "xxxxxxxx"
uamserver   "http://xxxxx.net/portal/"
uamsecret "xxxxxxx"

shkrid · 2016-01-06T17:34:06Z

Ok. Coova needs to be built with debug symbols (-G) either set your CFLAGS variable with -G during the configure stage or add AM_CFLAGS += -G to configure.ac & re-bootstrap.

then try perf record -g chilli
You'll get a file called perf.data which has the profiling information. You can then analyze the information using perf report --stdio
Capture the data & share it so we can see what's going on :)

Missed your above message. Should i am recompile coova anyway?

sevan · 2016-01-06T17:43:00Z

perf report --stdio > something.txt & post the results up.

If you run the report on another system without the same shared libraries/modules, you get

Failed to open /lib/modules/3.18.23/nf_nat_ipv4.ko, continuing without symbols Failed to open /lib/modules/3.18.23/nf_conntrack.ko, continuing without symbols Failed to open /lib/modules/3.18.23/xt_coova.ko, continuing without symbols Failed to open /lib/modules/3.18.23/ip_tables.ko, continuing without symbols Failed to open /lib/modules/3.18.23/nf_conntrack_rtcache.ko, continuing without symbols Failed to open /lib/modules/3.18.23/xt_tcpudp.ko, continuing without symbols Failed to open /lib/modules/3.18.23/nf_conntrack_ipv4.ko, continuing without symbols Failed to open /lib/modules/3.18.23/nf_nat.ko, continuing without symbols Failed to open /lib/modules/3.18.23/nf_defrag_ipv4.ko, continuing without symbols Failed to open /lib/modules/3.18.23/iptable_raw.ko, continuing without symbols Failed to open /lib/modules/3.18.23/iptable_mangle.ko, continuing without symbols Failed to open /lib/modules/3.18.23/iptable_filter.ko, continuing without symbols Failed to open /usr/lib/libchilli.so.0.0.0, continuing without symbols Failed to open /lib/librt-2.19.so, continuing without symbols Failed to open /lib/modules/3.18.23/mac80211.ko, continuing without symbols Failed to open /lib/modules/3.18.23/nf_reject_ipv6.ko, continuing without symbols Failed to open /lib/modules/3.18.23/iptable_nat.ko, continuing without symbols Failed to open /lib/libc-2.19.so, continuing without symbols Failed to open /lib/modules/3.18.23/ath9k.ko, continuing without symbols Failed to open /lib/modules/3.18.23/ath9k_hw.ko, continuing without symbols Failed to open /lib/modules/3.18.23/ath9k_common.ko, continuing without symbols Failed to open /lib/modules/3.18.23/tun.ko, continuing without symbols Failed to open /lib/modules/3.18.23/ip6_tables.ko, continuing without symbols Failed to open /lib/modules/3.18.23/cfg80211.ko, continuing without symbols Failed to open /lib/libpthread-2.19.so, continuing without symbols Failed to open /lib/modules/3.18.23/ath.ko, continuing without symbols

sevan · 2016-01-06T17:46:28Z

Also, just caught this, it looks like you ran out of memory? (bottom left hand side)

shkrid · 2016-01-06T17:52:49Z

perf report --stdio > something.txt & post the results up.

perf.txt

Also, just caught this, it looks like you ran out of memory? (bottom left hand side)

Maybe, but what it mean? Also I see 19000K free

sevan · 2016-01-06T17:57:50Z

that shows after the error had happened, right? what was the memory counter leading up to that?

shkrid · 2016-01-06T18:37:13Z

Let's go such way. :-)
I made a screen capture of the process.
screen_capture_perf.zip

shkrid · 2016-01-09T07:50:16Z

@sevan
Any idea?

shkrid mentioned this issue Aug 14, 2016

High userland chilli CPU when using kmod #273

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

on high load chilli stop give ip #185

on high load chilli stop give ip #185

dobrozhanskyi commented Dec 27, 2015

sevan commented Dec 28, 2015

dobrozhanskyi commented Dec 29, 2015

sevan commented Dec 29, 2015

shkrid commented Jan 3, 2016

sevan commented Jan 4, 2016

gareth41 commented Jan 4, 2016

shkrid commented Jan 4, 2016

sevan commented Jan 4, 2016

shkrid commented Jan 4, 2016

sevan commented Jan 4, 2016

shkrid commented Jan 4, 2016

sevan commented Jan 5, 2016

dobrozhanskyi commented Jan 5, 2016

shkrid commented Jan 6, 2016

sevan commented Jan 6, 2016

shkrid commented Jan 6, 2016

shkrid commented Jan 6, 2016

sevan commented Jan 6, 2016

sevan commented Jan 6, 2016

shkrid commented Jan 6, 2016

sevan commented Jan 6, 2016

shkrid commented Jan 6, 2016

shkrid commented Jan 9, 2016

on high load chilli stop give ip #185

on high load chilli stop give ip #185

Comments

dobrozhanskyi commented Dec 27, 2015

sevan commented Dec 28, 2015

dobrozhanskyi commented Dec 29, 2015

sevan commented Dec 29, 2015

shkrid commented Jan 3, 2016

sevan commented Jan 4, 2016

gareth41 commented Jan 4, 2016

shkrid commented Jan 4, 2016

sevan commented Jan 4, 2016

shkrid commented Jan 4, 2016

sevan commented Jan 4, 2016

shkrid commented Jan 4, 2016

sevan commented Jan 5, 2016

dobrozhanskyi commented Jan 5, 2016

shkrid commented Jan 6, 2016

sevan commented Jan 6, 2016

shkrid commented Jan 6, 2016

shkrid commented Jan 6, 2016

sevan commented Jan 6, 2016

sevan commented Jan 6, 2016

shkrid commented Jan 6, 2016

sevan commented Jan 6, 2016

shkrid commented Jan 6, 2016

shkrid commented Jan 9, 2016