Excellent article! Interesting to read alot of the same lessons I've learned in the past 3 years with Embedded Linux.
We started out with a commercial distribution that turned out to be a huge disappointment. On our next project we rolled our own which was a big learning curve but worked out better in the end.
Sounds like you're well on the way to making successful use of Embedded Linux. Were you using an RTOS before Linux? One observation I've made is that ex-RTOS users expect Embedded Linux distros to provide the tools and environment with which you can continue to develop embedded products the same RTOS way. It doesn't work. It's best to understand where Embedded Linux is different and design for it. The distros won't say this because they fear they might lose potential customers.
We're about to start another Embedded Linux project. Based on my experience, the things I'm hoping to improve on are:
- How to better manage an embedded linux build environment. We rolled our own build including the kernel, busybox, and other applications. We had some cumbersome low-tech scripts that would partition a compactflash card, install the bootloader (grub), create a file system and copy over the directory structure from the host machine. I'd really like to come up with a nice way to manage this better in the future.
is ideal for this. It can build all the components, create a rootfs (initrd, squashfs, jffs2, tarball etc) which is trivial to install on the target.
- Come up with a better "Unix-way" of implementing the application programs, instead of the one-big-blob method. Our system monitors and controls several different devices and peripherals. Many of the control functions are independent, but they need a way to send messages to each other to coordinate their behaviour. I really like the idea of having multiple daemon processes to implement the control of each device, but am not sure of the best/simplest way to do the interprocess communication required.
Avoid the one-big-blob.
I assume you don't have user-mode drivers because they tend to enforce the big-blob model. Find a book about Unix IPC. Simple Unix-domain sockets are a simple way to exchange messages between processes. Also consider shared memory. For more complex IPC, tools such as RPC may be suitable. The following links might also be useful:-
Making use of the process model turns Linux to your advantage. But I guess I'm only repeating stuff I wrote in the paper.