Remix.run Logo
adrian_b 2 days ago

Ioperm, which was introduced by 80386 in 1985, is useful only for a few legacy peripherals.

It allows the operating system to allocate peripherals mapped in the special I/O address space to dedicated device driver processes. For example it could allocate the real-time clock to such a process and a serial UART interface to another process.

Most modern peripherals are memory-mapped. However, you are right that even on the older systems without an IOMMU, the operating system could allocate peripherals to separate device driver processes, using the standard CPU MMU and mapping the peripherals using appropriate page attributes, e.g. as uncacheable memory regions.

The IOMMU is not needed to protect the OS and the processes between themselves, but it is needed to protect both the OS and the processes from rogue peripherals. This is needed regardless if the kernel is monolithic or a microkernel.

The microkernel overhead in the old microkernels, like Mach and all those inspired by it, was indeed too high due to context switches, but that has nothing to do with microkernels but those microkernels just implemented an extremely inefficient method of IPC, which unfortunately has tarnished the reputation of microkernels.

The only right form of IPC does not use any context changes, i.e. it uses message queues located in shared memory pages, with zero copying and no context changes, which is much faster than the traditional syscalls used with monolithic kernels.

In TFA there is another thing with which I do not agree, it is said that message queues need atomic CAS (compare-and-swap, a.k.a. compare-and-exchange in Intel parlance).

In my opinion, using CAS for implementing message queues is a big mistake, because this instruction is non-deterministic and the execution time for any program that uses CAS is unbounded for the worst case.

One-to-one message queues do not need any atomic instructions, they just need store-release instructions and either wait-for-not-equal loops when the waiting times are expected to be short or futex_wait (or similar) syscalls when the waiting times are expected to be long.

Waiting for (multiple) events is one of the principal functions that must be implemented by a microkernel for enabling interprocess communication, together with a memory mapping API. IPC per se must not be implemented by the microkernel, but by user-space libraries.

The message queues that are many-to-one, one-to-many or many-to-many, can use the atomic fetch-and-add instruction (LOCK XADD on x86-64), to dynamically partition the queue, allowing the concurrent insertion and extraction of data into/from the queue by all communicating processes. In this case, only the updating of the queue pointers is serialized, resulting in very low overheads for the communication through the message queue, even in cases of high contention.

While I agree with you that TFA is incorrect in some details, I agree with the general idea that today one could implement microkernels that would be simultaneously faster and safer and easier to maintain than a monolithic kernel like Linux.

Nonetheless, the chances for such a microkernel to appear and to be successful are very low.

The reason is that the only way to be successful would be to implement a very easy transition to it from one of the popular OSes, e.g. from Linux.

This means that it would have to also implement a syscall interface completely compatible with that of Linux, to allow running unmodified legacy programs, and also a device driver API compatible to that of Linux, to allow the reuse of the existing device drivers.

Probably the easiest way to do this would be to start from Xen, still using the root Xen domain with a Linux kernel that manages the I/O, and running legacy Linux applications in virtual machines.

From this, another special domain with the new microkernel should be added, and then the peripheral handling should be migrated gradually from the Xen Dom0 Linux to the microkernel with its associated device driver processes, starting with those more important, i.e. time, keyboard & graphic pointer, GPU, NVMe, Ethernet, USB.

In parallel with migrating peripherals to the microkernel, user applications should be migrated gradually, to use the new I/O IPC instead of the Linux syscalls.

Only such a gradual evolution could succeed. Otherwise, a nice microkernel with which you cannot run any useful application would fail, like the many attempts of creating better operating systems that have failed in the past, even when they were backed by huge companies, like IBM OS/2.

I believe that the I/O API used by applications with such a microkernel should be extremely similar with that of liburing (Linux io_uring), but which would use internally IPC with the appropriate processes, instead of kernel syscalls.

theamk 2 days ago | parent | next [-]

The only way to make it work with zero contest pages is to dedicate a whole core per process. There would be 1 core for filesystem, 1 core for network, 1 core for GPU, 1 core per device... You are going to run out of cores pretty fast! And even if you allow context switching for stuff like keyboard and mouse, you'd still need 3 cores for major functions (network, disk, graphics). Since a lot of modern systems might have as few as 8 cores, the overhead will be tremendous.

adrian_b a day ago | parent [-]

There is no need whatsoever to dedicate a core for each process which owns peripherals and which services the requests for them.

All the I/O requests would be submitted in queues and whenever a queue would be empty the server process would sleep by invoking a wait for event syscall, like futex_wait of Linux. Obviously, when a device driver sleeps and must wake up, that involves context changes, but this will happen only when the peripheral is seldom used, so the performance loss is not important. Once the device driver process is active, no context changes are necessary for data transfers between processes. When there is heavy I/O activity, a few cores may be occupied with it, but they will always correspond to the active I/O subsystems, not to any that exist but are not currently in use. The bulk data transfers should be done by DMA directly from the user buffers, so the device driver processes should only orchestrate all the transfers, which should not consume much CPU time.

Nonetheless, I believe that the performance could be improved by reserving a single core for the microkernel, for hardware interrupt handling and also for the keyboard and graphic pointer drivers. Only in very big systems with hundreds of cores it might be necessary to reserve more than 1 core.

A user API like that of liburing of Linux can be reimplemented almost with no changes in a system with a micro-kernel. That means that a user application would have an AIO completion queue, which instead of receiving data from the Linux kernel would multiplex the completion messages from all the device driver processes.

For the opposite direction, from a user application to the device driver processes, the implementation of the I/O request submission queue would be a little more complex, because of security requirements. Processes that trust each other, e.g. the children of a single process, could inherit and use a common submission queue. Otherwise, the submission queues must be distinct, i.e. in mapped pages that are shared with the device driver processes, but not between the user processes. Because polling multiple queues might waste time in a device driver process, it is likely that submitting a batch of I/O requests should be done by writing the shared buffers with data, then invoking a syscall similar to io_uring_enter, which would alert the corresponding device driver processes (by posting the request in single queues monitored by each device driver process for new events, but the posting of this small message would be done by the micro-kernel, so that rogue processes cannot mess with it).

Simpler but less performant I/O APIs, like the stdio of libc, can be implemented on top of a multiplex AIO API, like that of liburing.

ahartmetz 2 days ago | parent | prev [-]

There exist several microkernels that don't suffer much from overhead, it's not hypothetical but decades old technology, from oldest to newest: QNX, the L4 family, Fuchsia, RedoxOS.

adrian_b 2 days ago | parent | next [-]

Yes, but they remain confined to niche applications because none of them has been designed to be incorporated in a framework that would allow the gradual migration to them of the legacy applications and device drivers.

Only Fuchsia might have a chance, because of the disproportionate control that Google has over the Android applications and because of the constraints imposed on those.

vsgherzi a day ago | parent [-]

Unfortunately Fuchsia seems all but dead these days

lproven a day ago | parent | prev [-]

What about Minix 3?

Probably the most widely deployed microkernel ever: it's inside every Intel CPU for the last ~20 years.

Already has a mostly-working port of the NetBSD userland.

It was shaping up quite well until Andy Tanenbaum retired. Since then, nothing.

Of course, Intel has never released any of its code, which is terrible -- but legal and license-compliant.

https://www.osnews.com/story/136174/minix-is-dead/

Rochus 11 hours ago | parent | next [-]

I spent a lot of time with Minix 3 (see https://github.com/rochus-keller/Minix3/tree/Minix3_Book_TCC). The intention was to make the book version compile with TCC. It turned out that this version only covers a fraction of the relevant parts necessary to build a working system. Minix 3 in addition includes hundereds of assembly files. The impression created by the book of a lean C implemented kernel, or the claim "for compiling a small Minix system" are wrong. Not even the kernel is complete; the central IPC is actually implemented in assembler and linked with the kernel via some library not described in the book.

Concerning performance: studies suggest that Minix 3 is significantly slower (10 to > 100 times) than L4 and even Linux (see e.g. https://www.cscjournals.org/download/issuearchive/IJE/Volume... or https://www.minix3.org/theses/priescu_thesis_2012.pdf).

Concerning "inside every Intel CPU": we actually don't know which version that was supposed to be and how much it was modified.

ahartmetz a day ago | parent | prev [-]

The reason why I omitted Minix is that I don't know about its performance characteristics. Same for TRON / eTRON which is or used to be huge in the Japanese industry.