Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UPDATE: QNAP-UC5G1T - Drastic performance decrease when writing write to NAS #32

Open
jmrickerby opened this issue Nov 27, 2020 · 18 comments
Labels
performance Performance issue

Comments

@jmrickerby
Copy link

jmrickerby commented Nov 27, 2020

Recent an issue has developed with the DS718+ and a QNAP-UC5G1T. They were initially working well together, (see previous report). However, recently write performance has dropped off to 5MB/s while write performance, but retains 423/MB read second (to and from the NAS). The QNAP-UC5G1t is outside of the stock enclosure with an additional heat sink, so I do not believe thermal throttling is the culprit.

##uname -a:

Linux Alexandria 4.4.59+ #25426 SMP PREEMPT Wed Jul 8 03:21:29 CST 2020 x86_64 GNU/Linux synology_apollolake_718+

##dmesg:

[  109.646544] usbcore: registered new interface driver aqc111
[  109.662871] aqc111 2-3:1.0 eth2: register 'aqc111' at usb-0000:00:15.0-3, QNAP QNA-UC5G1T USB to 5GbE Adapter, 24:5e:be:4e:29:97
[  110.237328] IPv6: ADDRCONF(NETDEV_UP): eth2: link is not ready
[  117.765037] aqc111 2-3:1.0 eth2: Link Speed 5000, USB 3
[  117.778759] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
[  138.463405] usb 3-1: ep 0x81 - rounding interval to 128 microframes, ep desc says 160 microframes

##lsusb:

|__usb1          1d6b:0002:0404 09  2.00  480MBit/s 0mA 1IF  (Linux 4.4.59+ xhci-hcd xHCI Host Controller 0000:00:15.0) hub
  |__1-4         f400:f400:0100 00  2.00  480MBit/s 200mA 1IF  (Synology DiskStation 65007863CDE41651)
|__usb2          1d6b:0003:0404 09  3.00 5000MBit/s 0mA 1IF  (Linux 4.4.59+ xhci-hcd xHCI Host Controller 0000:00:15.0) hub
  |__2-3         1c04:0015:0101 00  3.20 5000MBit/s 896mA 1IF  (QNAP QNAP QNA-UC5G1T USB to 5GbE Adapter 04I00545)
|__usb3          1d6b:0002:0404 09  2.00  480MBit/s 0mA 1IF  (Linux 4.4.59+ etxhci_hcd-170202 Etron xHCI Host Controller 0000:02:00.0) hub
  |__3-1         0463:ffff:0202 00  1.10  1.5MBit/s 20mA 1IF  (EATON Eaton 5P P124J29M61)
|__usb4          1d6b:0003:0404 09  3.00 5000MBit/s 0mA 1IF  (Linux 4.4.59+ etxhci_hcd-170202 Etron xHCI Host Controller 0000:02:00.0) hub

ifcongif -a:

eth0      Link encap:Ethernet  HWaddr 00:11:32:79:63:E9
          inet6 addr: fe80::211:32ff:fe79:63e9/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1
          RX packets:9826 errors:0 dropped:0 overruns:0 frame:0
          TX packets:10645 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2098249 (2.0 MiB)  TX bytes:9781617 (9.3 MiB)

eth1      Link encap:Ethernet  HWaddr 00:11:32:79:63:EA
          inet6 addr: fe80::211:32ff:fe79:63ea/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1
          RX packets:3634 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1631 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:484992 (473.6 KiB)  TX bytes:1340223 (1.2 MiB)

eth2      Link encap:Ethernet  HWaddr 24:5E:BE:4E:29:97
          inet addr:192.168.1.99  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::265e:beff:fe4e:2997/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1
          RX packets:185996 errors:0 dropped:0 overruns:0 frame:0
          TX packets:144432 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1071696601 (1022.0 MiB)  TX bytes:1076902320 (1.0 GiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:6860 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6860 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:1307320 (1.2 MiB)  TX bytes:1307320 (1.2 MiB)
@jmrickerby jmrickerby changed the title QNAP-UC5G1T - Drastic perforamnce change QNAP-UC5G1T - Drastic performance (write to NAS) decrease Nov 27, 2020
@jmrickerby jmrickerby changed the title QNAP-UC5G1T - Drastic performance (write to NAS) decrease UPDATE: QNAP-UC5G1T - Drastic performance decrease when writing write to NAS Nov 28, 2020
@bb-qq
Copy link
Owner

bb-qq commented Nov 29, 2020

It appears data inbound to the DS718+ is peaking at 1.1Gbps.

I would guess that disk write speed is the bottleneck.
Measuring pure communication speed using iperf would help isolate the problem.

Recently seeing the performance of the 5Gpbs link, drop to tens of megabits.

  • Are you using the front USB port?
  • What was the output of dmesg when the problem occurred?

@bb-qq bb-qq mentioned this issue Nov 29, 2020
@jmrickerby
Copy link
Author

It appears data inbound to the DS718+ is peaking at 1.1Gbps.

I would guess that disk write speed is the bottleneck.
Measuring pure communication speed using iperf would help isolate the problem.

Recently seeing the performance of the 5Gpbs link, drop to tens of megabits.

* Are you using the front USB port?

* What was the output of dmesg when the problem occurred?

Yes, using the front USB. Want me to swap to the back? It was previously performing well connected to the front USB port.

dmesg output - not sure if this is relevant:

[28156.774896] usb 3-1: ep 0x81 - rounding interval to 128 microframes, ep desc says 160 microframes
[30986.407999] perf interrupt took too long (2514 > 2500), lowering kernel.perf_event_max_sample_rate to 50000

Thank you for all your time and effort to create and support this driver.

@jmrickerby
Copy link
Author

It appears data inbound to the DS718+ is peaking at 1.1Gbps.

I would guess that disk write speed is the bottleneck.
Measuring pure communication speed using iperf would help isolate the problem.

Recently seeing the performance of the 5Gpbs link, drop to tens of megabits.

* Are you using the front USB port?

* What was the output of dmesg when the problem occurred?

iPerf numbers look good - looks like I have a Synology SMB issue :-( Not sure why, as it was working nicely before.

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.10 GBytes  1.80 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  2.10 GBytes  1.80 Gbits/sec                  receiver
[  7]   0.00-10.00  sec  2.10 GBytes  1.80 Gbits/sec    0             sender
[  7]   0.00-10.00  sec  2.10 GBytes  1.80 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  4.19 GBytes  3.60 Gbits/sec    0             sender
[SUM]   0.00-10.00  sec  4.19 GBytes  3.60 Gbits/sec                  receiver

Disk performance:

echo 3 > /proc/sys/vm/drop_caches; time dd if=/dev/sda1 of=/dev/null bs=1M count=1K
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 4.86485 s, 221 MB/s

Ran SMART tests on the HDs and they both report healthy. The volume also reports as healthy. Shut down optional packages and indexing, restarted the NAS and still seeing this poor write performance. Also seeing the write issue on the Synology network interface.

Heading over to the Synology forums. Sorry to waste your time.

@bb-qq
Copy link
Owner

bb-qq commented Dec 13, 2020

FYI: Changing the version of SMB or some configurations in smb.conf might help improving performance.
https://www.synology.com/en-us/knowledgebase/DSM/help/DSM/AdminCenter/file_winmacnfs_win

I appreciate it if you share the solution to the issue with other people when you resolved the issue.

@sgman1908
Copy link

Recent an issue has developed with the DS718+ and a QNAP-UC5G1T. They were initially working well together, (see previous report). However, recently write performance has dropped off to 5MB/s while write performance, but retains 423/MB read second (to and from the NAS). The QNAP-UC5G1t is outside of the stock enclosure with an additional heat sink, so I do not believe thermal throttling is the culprit.

##uname -a:

Linux Alexandria 4.4.59+ #25426 SMP PREEMPT Wed Jul 8 03:21:29 CST 2020 x86_64 GNU/Linux synology_apollolake_718+

##dmesg:

[  109.646544] usbcore: registered new interface driver aqc111
[  109.662871] aqc111 2-3:1.0 eth2: register 'aqc111' at usb-0000:00:15.0-3, QNAP QNA-UC5G1T USB to 5GbE Adapter, 24:5e:be:4e:29:97
[  110.237328] IPv6: ADDRCONF(NETDEV_UP): eth2: link is not ready
[  117.765037] aqc111 2-3:1.0 eth2: Link Speed 5000, USB 3
[  117.778759] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
[  138.463405] usb 3-1: ep 0x81 - rounding interval to 128 microframes, ep desc says 160 microframes

##lsusb:

|__usb1          1d6b:0002:0404 09  2.00  480MBit/s 0mA 1IF  (Linux 4.4.59+ xhci-hcd xHCI Host Controller 0000:00:15.0) hub
  |__1-4         f400:f400:0100 00  2.00  480MBit/s 200mA 1IF  (Synology DiskStation 65007863CDE41651)
|__usb2          1d6b:0003:0404 09  3.00 5000MBit/s 0mA 1IF  (Linux 4.4.59+ xhci-hcd xHCI Host Controller 0000:00:15.0) hub
  |__2-3         1c04:0015:0101 00  3.20 5000MBit/s 896mA 1IF  (QNAP QNAP QNA-UC5G1T USB to 5GbE Adapter 04I00545)
|__usb3          1d6b:0002:0404 09  2.00  480MBit/s 0mA 1IF  (Linux 4.4.59+ etxhci_hcd-170202 Etron xHCI Host Controller 0000:02:00.0) hub
  |__3-1         0463:ffff:0202 00  1.10  1.5MBit/s 20mA 1IF  (EATON Eaton 5P P124J29M61)
|__usb4          1d6b:0003:0404 09  3.00 5000MBit/s 0mA 1IF  (Linux 4.4.59+ etxhci_hcd-170202 Etron xHCI Host Controller 0000:02:00.0) hub

ifcongif -a:

eth0      Link encap:Ethernet  HWaddr 00:11:32:79:63:E9
          inet6 addr: fe80::211:32ff:fe79:63e9/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1
          RX packets:9826 errors:0 dropped:0 overruns:0 frame:0
          TX packets:10645 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2098249 (2.0 MiB)  TX bytes:9781617 (9.3 MiB)

eth1      Link encap:Ethernet  HWaddr 00:11:32:79:63:EA
          inet6 addr: fe80::211:32ff:fe79:63ea/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1
          RX packets:3634 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1631 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:484992 (473.6 KiB)  TX bytes:1340223 (1.2 MiB)

eth2      Link encap:Ethernet  HWaddr 24:5E:BE:4E:29:97
          inet addr:192.168.1.99  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::265e:beff:fe4e:2997/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1
          RX packets:185996 errors:0 dropped:0 overruns:0 frame:0
          TX packets:144432 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1071696601 (1022.0 MiB)  TX bytes:1076902320 (1.0 GiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:6860 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6860 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:1307320 (1.2 MiB)  TX bytes:1307320 (1.2 MiB)

I have the same issue except on the DS918+. I used to get about 300MBps write but now I get less than 5MBps write. Read speed is still over 300MBps. Issue started sometime in November. I've had to return to using the Gigabit built-in ethernet due to such low speeds. Please let me know if I can supply information. I created an account just to add my thoughts. I've been troubleshooting this for weeks (on and off) and cannot fix it.

@jmrickerby
Copy link
Author

My issues appears to be that Jumbo Frames (Synology interface MTU 9000) have broken, but in a weird way!

  • The two internal 1Gbps interfaces are bonded with LACP and are configured with a Jumbo Frame MTU of 9000. This is still working great!
  • The DS718+ attached QNAP-UC5G1T was configured with an MTU of 9000; Windows 10 an MTU of 9014, and everything was working well. Then, at some point, it broke.
  • I experimented with different ping packet sizes and it appears the Jumbo packets (9014) sent from Windows 10 are fragmenting. (Again, was working fine before.)
  • For the time bring, I have disabled Jumbo frames on the Synology attached QNAP-UC5G1T and the Windows 10 system. Performance is close to what it was when Jumbo frames were working.
  • Cisco 2960X switch has always been configured with "system mtu jumbo 9198"
  • Again, the weird thing is that Jumbo frames are working fine with the Synology 1gb interfaces, just not the QNAP-UC5G1T!?!

@bb-qq
Copy link
Owner

bb-qq commented Jan 4, 2021

I forgot to reply to your previous comment....

Want me to swap to the back?

No, I just wanted to confirm you are using the front port.

dmesg output - not sure if this is relevant:

I think this output is not relevant because the value of my environment is lower than yours.

% cat /proc/sys/kernel/perf_event_max_sample_rate
25000

I would like to know what SMB version you are using. Using "SMB2 with large MTU" or SMB3 may change something. I'm using SMB3.
https://www.synology.com/en-us/knowledgebase/DSM/tutorial/File_Sharing/What_can_I_do_when_the_file_transfer_via_Windows_SMB_CIFS_is_slow

And, if you can capture the packets before/after the issue occurred I will try to take a look at them.

@jmrickerby
Copy link
Author

I forgot to reply to your previous comment....

Want me to swap to the back?

No, I just wanted to confirm you are using the front port.

dmesg output - not sure if this is relevant:

I think this output is not relevant because the value of my environment is lower than yours.

% cat /proc/sys/kernel/perf_event_max_sample_rate
25000

I would like to know what SMB version you are using. Using "SMB2 with large MTU" or SMB3 may change something. I'm using SMB3.
https://www.synology.com/en-us/knowledgebase/DSM/tutorial/File_Sharing/What_can_I_do_when_the_file_transfer_via_Windows_SMB_CIFS_is_slow

And, if you can capture the packets before/after the issue occurred I will try to take a look at them.

To confirm I am using SMB3 as the "maximum protocol". Current Synology configuration as follows:

  • SMB2 as the minimum protocol
  • SMB3 as maximum protocol
  • Transport Encryption Mode set to "disable"
  • Unchecked "Enable Opportunistic File Locking"
  • Synology MTU 9000 on both internal and QNAP interfaces

After setting netsh ipv4 settings back to defaults, and Windows NIC set to MTU of 9014, the results below are looking normal:

C:\>ping 192.168.1.99 -f -l 8972
Pinging 192.168.1.99 with 8972 bytes of data:
Reply from 192.168.1.99: bytes=8972 time<1ms TTL=64

C:\>ping 192.168.1.99 -f -l 9000
Pinging 192.168.1.99 with 9000 bytes of data:
Packet needs to be fragmented but DF set.

After a lot of troubleshooting, writing to the Synology with MTU 9000 settings all around, is still an issue. Test with QNAP interface and Windows 10 set to MTU 9014 (In NIC Advanced tab, netsh indicates MTU 9000):
image

Test with Synology 1gbps interface and Windows 10 MTU 9014:
image

Test with QNAP interface and Windows MTU 4088 (In NIC Advanced tab, netsh indicates MTU 4074):
image

Test with Synology 1gbps interface and Windows 10 MTU 4088:
image

Can you recommend a packet sniffer (and command line options) you want me to use? Windump?

@bb-qq
Copy link
Owner

bb-qq commented Jan 10, 2021

Hmm, can you test under direct connection using a cross cable without the switch to isolate the problem?
And I would like to know the output of Get-SmbConnection that indicates the actual SMB version you are using.

Can you recommend a packet sniffer (and command line options) you want me to use? Windump?

I recommend using the Wireshark on the Windows side.

@bb-qq bb-qq added the performance Performance issue label Aug 11, 2022
@Linlinwho1
Copy link

Linlinwho1 commented Oct 6, 2022

Hi all,

I have recently just re added back my QNA-UC5G1T using aqc111-apollolake-1.3.3.0-8.spk driver on my ds218+ with DSM7. It has very fast read speed(330MB/s max) but the write speed is 30-50X slower(5-17MB/s). I have try to do the same task on with the in build nic and was able to get 1gb speed on both way(faster on the esata ssd). have rebooted the nas each time i change the nic or driver version.

I use to be able to run run all of my 5 and 2.5gb nic at full speed back at Sept 2021. But i was moving around and did not use those nic till now. was wondering if it was some settings that need to be tweak? I will try with older driver and see if it helps.

test file use:
7.7GB mp4 file
2 copy of 7.7GB mp4 file
a folder(36.5GB) with 270 video file

so my setup is
DS218+
DSM 7.0-41890
internal Lan is 1gb mtu 9000
front usb 3.0 port: QNA-UC5G1T
eSata port: Samsung 840 pro 1TB sata ssd

Switch
mikrotik crs305-1g-4s+in
mtu 9000

pc
Windows 10 21H1
i5 11400
40gb ram
Samsung 980 500gb ssd(nvme) Up to 3,500 MB/s sequential read and 3,000 MB/s write
Intel X520 10GB nic

SMB setting: min SMB1, max SMB3, Transport encryption : Client define
using the QNA-UC5G1T nic
file copy from nas(esata ssd) to pc : 316-330 MB/s speed
file copy from pc to nas(esata ssd) : 5-17MB/s speed
file copy from nas(HDD raid 1) to pc : 140-167 MB/s speed
file copy from pc to nas(HDD raid 1) : 5-17MB/s speed

using the onbaord nic
file copy from nas(esata ssd) to pc : 114-119MB/s speed
file copy from pc to nas(esata ssd) : 114-119MB/s speed
file copy from nas(HDD raid 1) to pc : 114-119 MB/s speed
file copy from pc to nas(HDD raid 1) : 114-119MB/s speed

@bb-qq
Copy link
Owner

bb-qq commented Oct 9, 2022

What happens if the QNA-UC5G1T dongle is attached to the PC without changing the environment such as switches and cables?

@Linlinwho1
Copy link

so i have play around with my 10G/5gb nic.

I notice that the fastest read speed using my 10gb intel x520(PC) at 330MB/s with the NAS with the Sabrent ‎NT-SS5G 5GB usb nic
but read is still around 5-17MB/s

when I switch to QNA-UC5G1T on my pc get around 250MB/s with the same nic on the nas. but write is around 190MB/s.

So i did more test swapping NIC/network cable on the nas. having mixed issue. slow write speed to nas/not able to access nas(just hang when i try to access). have to reboot nas/pc/switch/router to restart all the test. i have also updated to the latest dsm version(could not get the version as my nas is no reachable now LOL.

I will try more test after i get my other 10G switch setup. as the CRS305 has only 4 10G and it is affecting my home wifi. so i will do all the test on another network .

@bb-qq
Copy link
Owner

bb-qq commented Oct 10, 2022

Some transceivers are not compatible with NBASE-T, which makes communication unstable, so I suspected that case. However, since you are getting correct performance in some setups, that seems unlikely.

@Linlinwho1
Copy link

It is working now after I move all of my 2.5/5/10GB NIC to my Mikrotik CRS312 10G switch. Could be my network issue or the nas hdd, As I have just clean it up to have more than 20% free space and run a defrag.

But great thanks for providing the updated driver and support for all of them .

@sgman1908
Copy link

sgman1908 commented Oct 11, 2022 via email

@jmrickerby
Copy link
Author

Try turning off jumbo frames on the NAS 10Gb network adapter. I’ve seen this issue before and I think it was jumbo frames. Also change minimum SMB version from 1 to 2, unless you specifically need support for version 1. Not sure if that will help but there are some potential vulnerabilities in with the old version of SMB I think.

I have disabled Jumbo Frames on the NAS and this has helped somewhat. After a lot of investigation, I think a firmware update on my Cisco switch may be the root of the issue. My Canon MF5960 printer/scanner/copier requires SMB1 to upload scanned pages to the Synology CIFS share :-(

@jmrickerby
Copy link
Author

Some transceivers are not compatible with NBASE-T, which makes communication unstable, so I suspected that case. However, since you are getting correct performance in some setups, that seems unlikely.

The QNAP-UC5G1T is connected to a "FS 10GBASE-T SFP+ Copper RJ-45 30m Module, Cisco Compatible" in my Cisco switch. It's going to be a huge pain, but I will try rolling back my Cisco switch firmware.

@sgman1908
Copy link

sgman1908 commented Oct 12, 2022 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
performance Performance issue
Projects
None yet
Development

No branches or pull requests

4 participants