network, USB, FTP server github branch

[nanner55]

Hi. I added a branch in a forked repo which has very buggy support for networking, USB SoundFont/ROM files, and a very basic, single-threaded FTP server. I figured you may find it useful as a reference if you haven't already started working on your own.

https://github.com/nanner55/mt32-pi/tree/ftpserver

There are still many issues and I haven't decided if I will continue working on it. It has only been tested on a 3B.

• The USB file support is simple but works ok. The drive named USB: is prioritized over the SD card (SD:) in both the ROM and SoundFont manager. Hot swapping is not really supported.
• Only open SSID wifi works consistently. Wired ethernet crashes on 64b kernels - perhaps due the AARCH/QEMU thing? It does seem to work on 32b kernels.
• The network code will hang if it doesn't connect.
• The FTP server is single threaded and has only been tested with WinSCP and CLI clients. Anything that is threaded (e.g. FileZilla) will stall. Only a small set of commands are implemented. The error checking and return messages need some work.
• The FTP server runs on core3. I was originally planning on having it run in a non-blocking mode on core0 but couldn't find support for non-blocking socket calls in the Circle net API.
• Uploads are very slow (300KB/s) independent of connection type. The cores start throttling almost immediately, although, I haven't seen this affect audio playback. Downloads are a little faster (3-3.5MB/s). Not sure if this is something in my code (core throttling, lack of cores available?) , the net socket library, or something else.
• I have not been able to get the WPA supplicant WiFi to work. It always hangs with no connection made to my router independent of protocols.
• The kernel is significantly larger: 800KB->1.5MB. The FTP server contributes to most of this and I'm not sure if it's all in my code or caused by library calls.
• I started with a high-level implementation using C++ std APIs and had to move several calls to either standard C or Circle specific APIs due to missing functionality or strange errors (sockets, fatfs, etc).

Despite the bugs, it was much more pleasant to work on after both a standalone x86/VirtualBox and mt32-pi versions started working and supported upload/download. Not having to move around SD cards is a nice dev feature.

[dwhinham]

Thanks for this attempt, the FTP stuff especially will be helpful when I get round to that. I do already have WIP stuff locally for the other features (USB, general network bringup/config options).

Yes; the kernel file size gets bloated quite a lot by libstdc++. I used to have it linked but I removed it when I noticed only one file I/O class from Munt used it in the entire project; there was also a measurable performance boost from using FatFS file functions directly rather than fopen/fread, etc (for some reason).

I want to try and keep things as minimalistic as possible and in the long-term eventually drop circle-stdlib aswell, so I'd be reluctant to accept the FTP code in its current form, but if you're done working on it (totally understandable), I can certainly use it as a reference for writing a plain Circle-native FTP server.

Hope this makes sense.

If you do decide to try working on other features in future, it might be a good idea to ask in case I've already started working on them; I wouldn't want anyone to waste time duplicating work!

Thanks again.

[nanner55]

I agree, it's not in a form worth merging. It was my introduction to pi programming and I had an unused 3B.

It's straightforward for me to change everything to C api and Circle calls. The stdc++ calls are a convenience for testing it outside of pi without having to add more ifdefs or implement Circle library APIs. I did end up implementing circle/net/socket.h using the berkeley networking C api calls.

The reasons I stopped working on it is the RETR/STOR performance over 3B wifi is poor, difficulty in getting the networking to work consistently across both ethernet and wifi interfaces for 32b and aarch64 builds, and abstractions/limitations of the Circle sockets api vs Berkeley sockets. Changing the fstream calls to fatfs unfortunately didn't change RETR/STOR perf. Perhaps converting all stdlibc++ code to native would solve all of these problems.