Added socket support

[colemancda]

Added wrapper for POSIX socket APIs that takes advantage of Swift's generics and protocol extensions.

[colemancda]

Should merge #64 first.

[Lukasa]

I think this name is a little misleading: we should probably call it PollEvents or something similar.

[Lukasa]

Let's try to be consistent about where we put extension access modifiers.

[milseman]

@colemancda did you see #30 ? That went through a few rounds of full API design. It ultimately ended up not landing for non-API-design-related reasons. I realize you cover a not-totally-overlapping set of functionality, but its good to refer to for API design.

The main reason it didn't land is we need to figure out the linkage situation for Windows (perhaps no initial socket support) as well as figure out the official Darwin story - sockets are supposedly "deprecated" in favor of Network.framework, but their Swift API is not marked deprecated and Network.framework has adoption issues (e.g. depends on ObjC, IIRC).

I refactored that PR to be into a separate module "SystemSockets" hoping to help the situation, but it might not be enough. It's unfortunate that Swift's module story is still so monolithic. Ideally, we'd have something more like the following:

1. A library-only types layer

E.g. get access to FilePath type and its nice syntactic API without pulling in your platform's syscalls. This could be a potential place for IPAddress types, if not in another "networking types" library layer.

Things in here could even be argued to be considered part of the stdlib.

2. An internal-only "util" module that should be slurped into the using module

We don't want separate compilation, much less dynamic linking here. We shouldn't need ABI annotations if there is no ABI. We shouldn't be trying to infer ABI annotations during optimization time if there is no ABI.

3. A normal C stdlib and syscall layer

This gives you the natively available sys calls without use of extra libraries.

3. An extra syscalls layer with linkage

This can include things that need extra linkage, like pthreads on Linux and sockets on Windows. Darwin has libSystem, which is approximately the kind of scope Swift System targets, and includes pthreads sockets etc. So on Darwin we'd just have this module.

4. Potentially a deprecated syscalls layer

If Darwin is formally deprecating sockets, then we could consider taking those API on that platform out of the "System" namespace and into legacy module on Darwin. Personally, I don't think the distinction between this option and #3 above is that significant, but it's an option too if #3 falls through for some reason.

[colemancda]

@milseman So... would that be another library? Another module in the same package or another package all together? I can't imagine BSD sockets never being a part of Darwin, its more Unix than Linux is. I understand that for Networking it makes sense to possibly not use this, but BSD Sockets are essential or hardware usage on all platforms. I use ioctl , file descriptors and sockets for Netlink (controlling WiFi hardware in Linux), Bluetooth, and on macOS for Terminal (termios). Not to mention normal Unix sockets for IPC. We need this base generic framework as its an essential part of the OS, I am not really involved in Swift networking or Server code, I understand that can be separate, but as long as sockets and file descriptors are an essential part of Linux and macOS, we need this lowest common denominator code, thats why I opted for generics, so it can be extended and used outside of the System module. At the very least consider removing all the IP related code, keeping the generic parts. Windows networking code being separate makes sense, they dont even have real file descriptors like on Unix.

[colemancda]

I strongly believe that this package is the correct place for all File Descriptor related APIs, which would include poll(), select(), socket() and ioctl().

I would suggest the following names for a separate package or module for networking code:

• SystemNetworking, CoreNetworking
• DarwinNetworking, LinuxNetworking, WindowsNetworking
  Again, the purpose of this PR is not for networking code (at least not my use case), just a Swift API for sockets on macOS and Linux. The serial aspect for example is needed for writing user space code for IoT projects, including LoRa USB transceivers (LoStik) that only expose a character based serial driver on macOS and Linux. Another example where this is needed is for serial (via RS232 or USB) communication with cheap Taiwanese solar systems (MPPSolar). I really do not want to have any of my libraries for IoT and I/O ever import Darwin or Glibc directly, and I currently have a lot of duplicated code and wrappers for precisely FileDescriptor and socket programming.

[colemancda]

@milseman I removed everything related to IP addresses, so this code now really has nothing to do with networking anymore. Local (Unix) sockets (for IPC), terminal tty control, ioctl for Linux, GPIO on Armv7, Linux Netlink protocol (WiFi), Bluetooth RFCOMM and L2CAP (could be considered networking) and Bluetooth HCI (direct hardware control) are all use cases for this code that has nothing do with server side swift, HTTP clients, or traditional networking programming. I have another PR for ioctl but a lot of code on Linux requires a combination of ioctl and socket() usage, ioctl() alone is not enough, especially for Bluetooth and Netlink.

[milseman]

So... would that be another library? Another module in the same package or another package all together?

Sorry, I didn't clarify. Another library doesn't exist, so for now it makes the most sense for it to live in the System package. I'm mostly talking about separate modules inside the package, if feasible (within Swift's monolithic modules and monolithic packages story).

The basic point behind a types module is that, long-term, people should be able to access these types without also pulling in syscalls. Syscalls will always have platform differences, but we want to enable a library to provide a cross-platform layer using FilePaths. IPAddress (and we can discuss this here or the other thread) seems like another kind of library type that could be useful outside of contexts which pull in syscalls.

We have, or will have, the ability to sink types to lower-level layers, so I'm not overly concerned. But this might help provide framing for the library types discussion.

I can't imagine BSD sockets never being a part of Darwin, its more Unix than Linux is.

Unfortuantely I have zero say and nearly zero power over this. I agree with you, FWIW.

I use ioctl , file descriptors and sockets for Netlink (controlling WiFi hardware in Linux), Bluetooth, and on macOS for Terminal (termios)

Yup, also needed additions to System.

As long as sockets and file descriptors are an essential part of Linux and macOS, we need this lowest common denominator code

Yup, we should find some way to land it that doesn't violate the platform's sensibilities. Have you thought about the linkage story here at all?

thats why I opted for generics, so it can be extended and used outside of the System module.

This is where I think we might start to diverge. In the bigger-picture API design, generics and protocols define more abstract semantics to be refined and extended elsewhere. Small-picture they're also used for workarounds, reducing API bloat, etc.

I think that System should be hesitant to stake out new ground in the former. System intefaces becomes messy and non-orthogonal very quickly, even within a single platform. System can't in general make broad or sweeping semantic guarantees beyond what's afforded by fairly direct surfacing of system interfaces.

What are you hoping for libraries to extend for SocketAddress, SocketProtocol, SocketOption, etc?

I strongly believe that this package is the correct place for all File Descriptor related APIs, which would include poll(), select(), socket() and ioctl().

Yes, and platform-native interfaces like kqueue, epoll, and io_uring (if we can figure out versioning).

I would suggest the following names for a separate package or module for networking code:

I don't know if "networking" is the right line of demarcation here, but if it is that works for me.

The serial aspect for example is needed for writing user space code for IoT projects, including LoRa USB transceivers (LoStik) that only expose a character based serial driver on macOS and Linux. Another example where this is needed is for serial (via RS232 or USB) communication with cheap Taiwanese solar systems (MPPSolar). I really do not want to have any of my libraries for IoT and I/O ever import Darwin or Glibc directly, and I currently have a lot of duplicated code and wrappers for precisely FileDescriptor and socket programming.

Great, I think this is good justification for it being in System (the package). Just figuring out how to make it happen.

I removed everything related to IP addresses, so this code now really has nothing to do with networking anymore.

Do hold on to that code though, we'll want to figure out a place for it.

[colemancda]

Sorry, I didn't clarify. Another library doesn't exist, so for now it makes the most sense for it to live in the System package. I'm mostly talking about separate modules inside the package, if feasible (within Swift's monolithic modules and monolithic packages story).

So, SystemSockets?

The basic point behind a types module is that, long-term, people should be able to access these types without also pulling in syscalls. Syscalls will always have platform differences, but we want to enable a library to provide a cross-platform layer using FilePaths.

Thats great, but thats not how the library is currently structured for FileDescriptor, I tried to design an API that fits with the current design, therefore a part of System and pulling in syscalls. If there was a design document I could work with that.

What are you hoping for libraries to extend for SocketAddress, SocketProtocol, SocketOption, etc?

IDK if extend is the right word, really just provide concrete implementations.

• BluetoothLinux provides HCISocketAddress, L2CAPSocketAddress, RFCOMMSocketAddress, BluetoothSocketProtocol, BluetoothSocketOption, L2CAPSocketOption, RFCOMMSocketOption and extends SocketOptionLevel.
• Netlink provides NetlinkSocketAddress, NetlinkSocketProtocol and NetlinkSocketOption.

Please view Netlink/FileDescriptor.swift for an example of how the SocketAddress protocol provides a clean APIs while the implementation provides the underlying C buffer.

Yup, we should find some way to land it that doesn't violate the platform's sensibilities. Have you thought about the linkage story here at all?

The way the library is currently structured, we import the C platform library for each platform (Darwin, Glibc, ucrt) and CSystem headers. It was my understanding this would always be a sort of Swift overlay for Darwin and Glibc.

Right now I have a cross platform library called GATT, its main functionality is to provide a CentralManager protocol that each native platform can implement, but by using generics we can target a common API, and also use platform specific APIs (Like CBPeripheralManager.state), of course this library has a cross platform model layer (Bluetooth) which provides a Bluetooth module with basic types like BluetoothAddress, BluetoothUUID, CompanyIdentifier, etc. Mostly structs and enums with RawRepresentable, no C dependences, object graphs, state machines, or even PDUs. Then another module BluetoothHCI provides the PDUs for the HCI protocol, BluetoothGATT for the GATT PDUs and connection state machine, BluetoothGAP for the GAP (Advertisement) PDUs. On iOS, GATT also has a DarwinGATT module which just wraps CBPeripheralManager to be used with the CentralProtocol API + Darwin specific additions, and only depends on Bluetooth and GATT for the model types, resulting in a <2mb binary size.

On Linux however, using GATT requires importing Bluetooth, BluetoothHCI, BluetoothGATT, BluetoothGAP, and BluetoothLinux, which itself depends now on this System module, and more importantly requires socket(), connect(), listen(), accept() and ioctl(), so I am using conditional target dependencies (which was ironically specifically pitched for usage with BluetoothLinux). The binary on Linux is obviously much larger than when just using GATT on iOS, but the common API is more or less the same. Without Bluetooth and GATT I cant implement DarwinGATT which is platform specific, but those libraries themselves are very abstract.

@colemancda did you see #30 ? That went through a few rounds of full API design. It ultimately ended up not landing for non-API-design-related reasons. I realize you cover a not-totally-overlapping set of functionality, but its good to refer to for API design.

I did after I started working on this (like 60% done). The problem with that code is that SocketAddress is a concrete type that is cast to Unix, IPv4 and IPv6 socket addresses. I need SocketAddress protocol so Netlink, BluetoothLinux, possibly Bluetooth on Windows, and future protocols can implement it and not have to provide a UnsafeMutableRawPointer to these C APIs.

[colemancda]

Please provide me some guidance on how you would like these APIs implemented so they can be merged, there is a very real use case for all this and I don't want a monolithic library either is a burden on code size for projects, but neither do I want a bunch of code duplication as is currently the case. I would be very interested in Syscalls being a separate module, but not sure how that is possible at the moment. Also I think requiring a POSIX environment is reasonable, I dont think it makes since to consider WASM as a deployment target for this project.

[lorentey]

(Popping in very quickly; I'll read through history and offer some more considered opinions soon.)

My baseline expectation is that the APIs in #30 will be the starting point for any socket additions that land in this package. We've spent significant effort on designing those APIs to fit the goals of this module. I don't think we would want to start from scratch without an extremely good reason.

As Michael said, exposing socket APIs on Darwin is unlikely to be in the cards, as they are no longer recommended for use there. (The platform position is that sockets are obsolete technology, and newly written code must not use them.) The use cases you listed on Linux are convincing, however; it would likely make sense that this package provided some socket wrappers on that platform. (It's not currently a given that System would ship IPv4/v6 support even on Linux -- our official networking library is Swift NIO, and the party position is that people shouldn't use raw networking APIs.)

(Background note: (just to make sure we are on the same page -- not necessarily related to any part of this PR)

System is intended to provide high-performance, low-level system call interfaces that eliminate the safety issues that arise from directly calling the underlying C APIs, while remaining extremely faithful to them. There is no expectation that System will provide the same APIs across any two platforms, although we do try to keep common things consistent when it makes sense to do so. This also applies to POSIX interfaces -- e.g. we may very well end up adding sockets on Linux and BSDs, but not on Darwin.

Of particular note is that it's not System's job to invent abstractions to improve the usability of these syscalls, or to make them more portable. System's job is (1) to eliminate memory safety problems, (2) to apply the Swift naming conventions, and (3) to faithfully expose the preexisting constructs, warts and all. It is essential that System introduces no non-zero cost abstractions beyond what's absolutely necessary to achieve goal 1: there must be no good technical reason for anyone to prefer using the raw imports rather than System's APIs.

Swift System also ships as an ABI stable SDK component on Darwin. As platform owners, Apple has full control over which syscalls in Darwin gain official wrappers in Swift System, and what these wrappers look like. As such, System's Darwin APIs are subject to Apple's internal API review processes, and we in Apple's Swift stdlib team are mediating between open source work and the experts at the company.)

The problem with that code is that SocketAddress is a concrete type that is cast to Unix, IPv4 and IPv6 socket addresses. I need SocketAddress protocol so Netlink, BluetoothLinux, possibly Bluetooth on Windows, and future protocols can implement it and not have to provide a UnsafeMutableRawPointer to these C APIs.

SocketAddress in #30 is designed to be extensible; it is very much able to support additional address families as needed. The fact that SocketAddress is a concrete type was very much an intentional choice; replacing it with a protocol does not sound like an obviously good idea.