Does real-time matter any more in Linux?

Real-time frustration

From the earliest days of its adoption in embedded devices, Linux has had a number of alternative add-ons that give it real-time capabilities. Many of these have come and gone. None were ever deemed suitable for inclusion in the mainline.

Another project, PREEMPT_RT, aims to produce a set of patches that add real-time support to the Linux kernel in a way which the kernel maintainers will accept for merging. Much of that work has already been merged upstream. But recently, Thomas Gleixner, the project leader, suggested that the project was losing momentum and would likely finish in 2014, whether or not its final work is merged upstream. The problem isn't so much with the code, it's with the effort needed to maintain it in the longer term since the code touches much of the core kernel. There is obviously reluctance to integrate any code if there aren't enough developers to look after it. It seems that the enthusiasm for improving real-time capabilities for the mainline Linux kernel still further is waning. So the question is: does real-time matter any more?

Before answering that question, let me clarify what I mean by real-time.

What is real-time, anyway?

The term real-time is often misused when discussing OS behaviour. There are two cases to consider:

  1. Soft real-time is where specific time constraints exist, e.g.: event X must be handled every 10ms, but failure to do so isn’t catastrophic and the system is able to recover if the deadline is missed. Such failures shouldn’t happen too often and the software should be designed to minimise that possibility. An audio player, for example, has soft real-time constraints on pulling data buffers for playback. If a buffer isn’t ready when it is needed, an audio glitch occurs but the system continues to operate.

  2. Hard real-time is where there are scheduling deadlines that the software absolutely must meet in order to maintain system operation. Failure to meet a hard deadline results in catastrophic system failure. A manufacturing control system is an example of a system which might have hard real-time constraints.

Real-time is concerned with scheduling predictability, not speed.

Linux already has kernel config options in the mainline to tune it for most soft real-time needs. It is meeting hard real-time constraints which present the bigger challenge.

What are the design considerations for real-time?

Real-time requirements can be minimised by both hardware and software design.

  • Hardware design, e.g.: interrupt queues, DMA request rings, FIFOs etc. Essentially these are in-hardware buffering mechanisms which mean that the hard deadlines are still there, but the timescales are extended such that the OS can easily be sure of meeting them. Modern network hardware, for example, uses these techniques to minimise real-time constraints on the OS software.

  • Software design, e.g.: work queues, scheduling algorithms, event pre-emption, high resolution timers etc. Event response latency is critical for adhering to real-time constraints.

In order to react to an event within a fixed time constraint, the OS must deschedule the current activity and take the appropriate action to handle the real-time event. The event is typically an asynchronous interrupt or timer. But during normal operation, the OS disables all interrupts around critical sections. Clearly the maximum time that such events are disabled determines how quickly the OS can action an interrupt and schedule the high priority real-time work.

The Linux kernel historically did a lot of work in interrupt context, e.g.: network packet processing and disk events. And the maximum time it could spend handling interrupts wasn't deterministic. Over time, the kernel has been improved to move more of this work out of interrupt context, which gives the kernel scheduler more control over what runs when.

The proliferation of multimedia has also resulted in improved real-time behaviour of Linux. Users won't accept audio or video glitches, even if they are using their computer for other CPU intensive work at the same time. So real-time characteristics became important to most Linux users, not just to the embedded developer, and a larger pool of developers have contributed to improving the situation. In addition, faster, multi-core CPUs make it a little easier for the OS to handle real-time events quickly, even without real-time features such as PREEMPT_RT.

There was a time when product developers needed to choose whether to use an RTOS or Linux on their CPU, depending on the projects' real-time requirements. Today, however, embedded SOCs come with multi-core CPUs, some of which can be dedicated to handle hard real-time events, or individual cores within the CPU can be tuned to handle specific interrupts within a standard Linux OS environment. The number of situations where further real-time improvements are needed therefore just gets smaller.

So what now?

The PREEMPT_RT project has resulted in substantial improvements in real-time behaviour of the Linux kernel. The fact that there seems to be little interest in improving it further is testimony to the success of the project.

It isn't clear yet whether enough people in the Linux developer community care enough about hard real-time behaviour to justify the work required to have it integrated into the upstream kernel. I hope that it is integrated. It would expose more real-time considerations to the wider Linux developer community and therefore help to avoid features being integrated in the future which are not real-time friendly.