Yocto Project

As the calendar year winds down, I find myself tapping away at the keyboard at my sister's house in Denver, Colorado in a snowstorm. I just spent the morning digging out our rental car and shoveling my sister's driveway and her next-door neighbors. ... [More] It is a time like this to reflect and, yes, to remember that one of the reasons I moved to Portland, Oregon was to escape the snow! This has been a busy month in the Yocto Project, with all kinds of activity jumping along: We launched a maintenance release 1.0, dubbing it Yocto Project v1.0.2. Our 1.0 release was last April, but we try to provide update support for at least a year after release. This release includes security patches as promised and some fixes to make sure the build works with more recent Linux distributions. We're also working on a maintenance release for 1.1, which came out last October. Nobody should ever fear that compatability has been broken in a maintenance release. My mental rule of thumb after a few decades in the software game is that you should have no more than 10 - 12 bug fixes in a maintenance release, which also constrains the amount of QA you need to do. 1.1.1 though will have a few more changes in place because we want to address some issues raised by Matthew at FreeScale and to make sure that the release meets their needs. More on this later. Our development, QA and release engineers also produced the first milestone from our 1.2 release, due this coming April. We produce these milestone releases every 6 weeks or so. Our idea is to constrain the length of time that the development window is open to make sure we can freeze, stabilize and run a full QA sweep. Then we provide these little milestone releases to you so that you can have some demonstratable features to use in a somewhat more stable form. At least, it should be more stable than developing on the Master branch! As I wrote this, we decided to go ahead and release the M1 milestone. It may seem like we're putting an inordinate effort into producing these maintenance releases. In fact, we had hoped to have 1.1.1 out in December, but putting three releases out in one month was just too much for our release engine to handle. Why make the effort to do maintenance releases? Here's the way I think about it: The Yocto Project is an open source project, not a software product. So we're not supporting our releases for years and years. We're looking for our friends at Wind River, Timesys, Mentor Graphics, Montavista and others to produce software products which use Yocto as the upstream. On the other hand, by providing these additional stable releases, it shows that we care about the developers who base their products on the Yocto Project. We know not every product development cycle perfectly aligns with the every six month Yocto Project cycle. Having these maintenance releases helps to support them. As I mentioned, we try to keep these releases as stable as possible, just fixing critical bugs or security patches from upstream. So we think these maintenance releases matter, to demonstrate that we care about you the developer who uses the Yocto Project to make the magic happen on your devices. By the way, I'd like to offer a shout out to Joshua Lock, who has been spearheading the development side of these maintenance releases, Beth Flanagan for release engineering work, Jiajun and the project-wide QA team. And also to Paul Eggleton for reminding us to get the 1.0.2 maintenance release on the schedule. [Less]

Thanks very much to all who participated in the Yocto Project community survey! We have a winner for the t-shirt: Marc F., you'll be receiving your shirt shortly. Thanks again to everyone, and have a happy holiday!

These videos were shown at the Embedded Linux Conference / LinuxCon in Prague, CZ in October. Enjoy!

The Linux Foundation did a nice little two-minute video clip of Jefro, our Yocto Project Community Manager, talking about last October's 1.1 release, and the upcoming 1.2 release planning. Check it out! Jefro claims that he looks kind of sleepy, but clearly the camera loves him!

I've noticed a disturbing trend amongst a few of the high quality wineries in my state. They have abandoned the cork to close their high-end wine bottles and turned to screw caps. This is good news to people who struggle with how to get a cork out of ... [More] a wine bottle. And wine snobs will point to the countless studies which show that metal tops eliminates the possibility that the cork has gone bad and spoiled the wine. I think this totally misses the point. These people are making a product that costs $40 US per bottle and up. Why in the world would anyone spend this much on a bottle of booze? They must be getting more out of it than just high-priced liquid. They are in essence buying into an experience which includes the ritual of pulling a cork. (And, I guess, the ritual of liver disease, dilerium and all the rest, but I degress). This is in part why people fall in love with the iPhone. Every cell phone makes calls, many will allow you to take photos or load applications. But people fall in love with the magical experience that the iPhone offers. (I am quite immune to this love, by the way). What does this particular rant have to do with embedded Linux? The Yocto Project is still very focused on providing the best build system, metadata and application development toolkit that we for developing your own custom embedded Linux. In addition, we're trying to radically improve the experience that developers have, particularly first-time developers trying their hand at Yocto. Why should we care? I have talked to a few people who don't have a terrific out-of-the-box experience with Yocto. These are not dummies - they are brilliant. I can only conclude that we can do better in this area. I hope that this will make a big difference as the developer base continues to expand on the Yocto Project and helps make it easier for Linux to grow in embedded. I hope we can have a good experience as well as good wine. [Less]

In October, following the release of the 1.1 (Edison) version, Yocto Project engineers embarked on a challenging project: to create a new embedded product using the Yocto Project tools in three weeks. The result is a Network-Attached Storage (NAS) ... [More] device called Baryon, based on the Intel Black Sand (Intel Atom N450) single-board computer. This layer turns a Black Sand into a fully-functional NAS. The wiki page about the project shows all of the details about the project, including profiling results. Says Shane Wang, lead engineer on the project: "In order to test Yocto's usability to build images for embedded devices, we choose a personal NAS, based on Intel's Black Sand hardware, and expected to complete in a pretty constrained time frame (3 weeks). We did some investigation on different packages, prioritized some key features, and worked out a NAS solution based on the Yocto Project from Oct. 12 to Oct 28. Now the code for Baryon is ready, the demo is also ready, the data for preliminary profiling and tracing with ADT is ready." Major efforts for this project included: Samba server (Edwin) webmin, a web interface for administration and configuration (Paul) NFS server (Ke) Media Tomb (Dongxiao) Proftpd server (Dexuan) SSH and RSYNC (Dexuan) Soft RAID (Dexuan) Profiling and Tracing by ADT tools (Dongxiao, Lianhao and Jiajun) Testing (Jiajun) For more information, see the Baryon wiki page. [Less]

Much has been written about how the Internet has revolutionized collaboration and made it possible for your brilliant ideas to make a difference no matter where you live on the planet. Bill Gates is famously quoted in Nick Kristoff's "The World is ... [More] Flat" that "... so many people can plug and play from anywhere, natural talent has started to trump geography." This is of course true, but even with the Internet, there is no replacement for face-to-face interaction. The tribe, it seems, still needs to gather around the fire to have a talk now and then. The conference setup was in a very new and modern hotel called the Hotel Clarion Congress in Prague, which was a terrific venue. Since the Yocto Project was a co-sponsor of the event, we got a nice booth location, and Tracey Erway did a fantastic job setting up the booth and populating it with demos, videos, giveaways and t-shirts. You can also see Darren Hart manning the booth behind the Gource video that he rendered showing the many contributors on the project from many places. I was impressed by how much the booth became a gathering spot for people wanting to talk about Yocto and what we were doing in the project.  There was a lot of opportunity for people to interact with Richard Purdie, who is the Yocto Project architect, and a very approachable guy. Koen Kooi is a TI guy and a long-time Open Embedded and Angstrom maintainer, who really helped us out a lot in the booth, and showed off his Beagle Board as a demo. I also appreciated the work of Jeff Osier-Mixon, better known as jefro, who is the Yocto Project Community Manager, and always helps us make sure we are taking care of the community and helping it to be nurtured and grow. It was fun catching up with Marcin, who works for Linaro, but is a long time heavy contributor to Poky and OpenEmbedded. Here is a photo of him with Richard and Dirk Hohndel from Intel. And like any good Linux Foundation event, there were some excellent parties to give us a space to hang out with each other and appreciate the unique culture and food of Prague. But in spite of being in such a beautiful city, these folks are sometimes hard to break away from hacking. On a Saturday with nothing planned but some tourist activities, I actually had to "encourage" Saul to take a break from his computer. (I actually closed the lid of his laptop to make sure he actually stopped working. I hope he forgives me.) More kudos to Sean Hudson (Mentor Graphics), Paul Eggleton (Intel), Bill Mills (TI), Nithya Ruff (Wind River), Philip Ballister (OpenSDR), Jessica Zhang (Intel) and too many others to count who helped us so much in the talks, booth and discussions about Yocto. Oh yeah, I guess there were some more or less official things going on as well: There were talks and classes on a number of new developments in the new release of the Yocto Project. I was very surprised that my overview talk drew so many people. The Yocto Project Advisory Board had a combination face-to-face / conference call meeting, where we talked about the the new Shoeleather lab, the new neutral board lab contributed by Mentor Graphics and about the project's budget (woo hoo). The OpenEmbedded e.V had its annual General Assembly meeting. I learned more about German law in that three hour meeting than I ever knew existed, because the OpenEmbedded Project's non-profit entity is chartered in Germany. We did have some useful talking points about which conferences to cover. The problem with mentioning anyone in a blog post like this is that I'm sure I have missed somebody who will be hurt because I didn't mention them. I am so sorry about that, and I hope you can forgivde me! [Less]

Back in my college days, I sang in the University Chorus, one of those big choirs who sang a variety of pieces, mostly classical and rarely a more contemporary song. One time we had a young music director who was rehearsing us on a newer piece with ... [More] just a piano player. When we had most of the vocal parts worked out, he announced that at the next rehearsal, the piano would be joined by guitars and percussion. "Then," he said to us with a twinkle in his eye, "the piece will really begin to cook." And it was amazing! The addition of those impact players really added a lot to the experience of the music, more than just adding individual singers. The whole song cooked and sizzled and sprang to life.  This is the kind of experience I have seen with the Yocto Project over the past six months. In addition to the growing chorus of individual developers, we are seeing key impact players like TI, FreeScale, Intel, Mentor Graphics, Wind River and MontaVista join the ensemble and make critical contributions. And it seems to show by the number of downloads we are seeing - our build and release engineer Beth Flanagan tells me that in the first two days after the bits came online, we already had like one fifth of the downloads we had through the entire lifetime of the 1.0 release. The complete release notes for "edison" are a good read, since you see everything from the complete list of 21 features, the 76 unique contributors and the number of yocto-seconds it took to do the official build. Here are a few key highlights that I picked out, see the release notes for the complete list. Hob - Using the Yocto Project to build Linux images, you usually need to learn which configuration file is located where so you can change it with a text editor. We wanted an easier and quicker way for someone to build an image by bringing all of these options together in a single place. The Hob is our answer to this. We have plans to enhance it further, making it a place to solve other usability issues as we discover them. System builder support in Eclipse - similar to our work on the Hob, we have added automation for the system developer in Eclipse in addition to the application developer support. What this means is that with 1.1 you can use Eclipse as the center of your embedded development world. Now you can load up the recipes for a Linux system into an Eclipse project, edit the recipes right in Eclipse, then kick off the Hob to do the build. You can still use Eclipse to create an application, deploy it to the embedded device, poke at it with analysis tools (like the newly added systemtap) and debug it remotely. We have a little video which shows these features working. OE Core branding - With the Open Embedded Core as the common upstream project between the Yocto Project and the Open Embedded Project, we have renamed some things derived from that core. For example, to build small footprint images in Yocto 1.0, you would build "poky-image-minimal" which raised questions about why "poky" was used in this context. This is now the less confusing "core-image-minimal". Layer Tooling - A very powerful part of the Yocto Project architecture is "layers." This feature allows customizations to be added to the system at every step in the value chain from sand to finished device. Some of the clear feedback we received from folks was that we needed an enhanced set of tools to work with layers. These perform a variety of functions, like complaining when a .bbappend file refers to a .bb file which doesn't exist to combining layers together into a single one. Multi-lib and x32 - Common processors in embedded systems are coming with some not-so-common features these days. 64 bit support and multi-core used to be features you would find only in big iron servers. But these features come with a price. For example, to take advantage of 64 bit data types, you would normally be forced to compile the entire system to run in 64 bits. But if there are only a few parts of the software which need this large data support, then you have wasted a considerable amount of the system's resources by compiling everything to use 64 bit support. And frankly, some common applications have not been ported to work with 64 bits and might never be. Multi-lib support is an excellent solution, allowing the developer to select 32 or 64 bits as appropriate. X32 is another option, which allows an x86-64 system to run with 64 bit registers but 32 bit data types. x32 is still being developed in the Linux ecosystem, but we have the first steps of this support in the Yocto 1.1 release. Developer Guide and Videos - We're constantly trying to make embedded Linux development more accessible to more users, whether they are experience Linux geeks or not. To enable a broader community of developers, we have a new Developer's Manual and instructional videos for using Hob and the Eclipse tools. And as usual, we have updated the kernel (v3.0.4) and toolchain (gcc 4.6.1) to the most recent stable community releases, as well as upgrading numerous other Linux user-land versions. We hope you enjoy this release, that it enables you to create the next insanely great device. Thanks to everyone who contributed and to the community at large who make this song really cook. If you are at the Embedded Linux Conference - Europe, I hope I get a chance to see you. We will have a number of Yocto Project talks and demos and contributors on hand in Prague. [Less]

For the last 18 months or so I've been working full-time on the Yocto Project, where I've touched various parts of the system. I'm really proud of what our team has achieved - we have an amazing tool that people want to use! Despite that, there are ... [More] still some common tasks that users can't achieve as easily as we'd like... Our system, Poky, has traditionally assumed a certain familiarity (to varying degrees) with Open Source software, Linux and building software. However the more competent our build system becomes the more we attract the interest of potential users who aren't a typical UNIX greybeard. The Yocto Project has been making various efforts to improve the usability of the project at all levels. We have an excellent technical writer on our team who works (laregely underappreciated) on improving the documentation for our project, we have a small but dedicated team of people making application development for Poky built images easier - much of this work focused around enabling as much of the development as possible to happen in the premier Open Source IDE; Eclipse and a project-wide focus on making things simpler and easier to understand. In line with this effort I've spent most of my development time over the last cycle working on a GUI for the Poky build system called Hob (the name ties in with the baking theme - a hob is what we call the top cooking surface on an oven in England). The aim of Hob is to enable a user to perform common tasks graphically, we focused primarily on enabling you to generate a custom image for the initial release but have several plans for enhancements over the coming months. By way of introduction I spent some time producing a video to demonstrate use of the Hob (I'll avoid ranting about the state of Linux video editing tools), it's available to watch on YouTube and Vimeo. The challenge of the Hob was turning the traditionally batch-run BitBake program into something more interactive, unfortunately I underestimated how much effort this would involve such that what I was able to deliver in the Yocto 1.1 time-frame isn't quite as functional as originally designed. For Hob the problem is that there is only so much information we can know about the how a package will be built by looking at the metadata. For example: we can't know which of the build dependencies will actually get linked into a resulting binary, so we can't do accurate image contents prediction by operating purely on the build system metadata. To that end this initial version of Hob comes with a caveat, the reported image contents are only an estimate. As well established by Brooks, this version of Hob ended up being sort of a Pilot System ("write one to throw-away") and we've learned a lot from developing this tool. Fortunately I followed best practices as already implemented in the BitBake code base to modularise as much of the Hob GUI as possible. Further, as Hob is actually a BitBake GUI, rather than some kind of wrapper program, many of the improvements I made are in the core BitBake libraries and much of the code will live on as we develop Hob in future releases. For the next release we'll certainly be improving Hob, and whilst nothing is set in stone at this stage, we are contemplating switching to a two-phase build - where the custom image generation is done once packages have been built. If we go this route we can be more accurate about dependencies and the size of the resulting image. Our initial feedback on Hob leads us to believe that as many people care about customising the image after the build as care about customising how it's built. With a two-phase mechanism we'll be able to build GUI's that work well for both scenarios and can be used seamlessly together in tandem. [Less]

  With the next version of Yocto (1.1, codenamed edison) coming up in a few weeks, and with a new cycle (1.2) beginning immediately following that, I thought it might be a good time to reflect a little bit on the development and release process from ... [More] my perspective as the maintainer of most of the BSPs in the meta-intel repo, having now been through two complete cycles.   Mostly it's as you'd expect - the Yocto codebase is extremely fast-moving, and just maintaining the current set of BSPs against the changes to the core metadata can at times be a full-time job in itself.  During the active development cycle, and sometimes even during stabilization phases of the project, builds break out of the blue, things randomly break at run-time, and most of the time the causes aren't immediately apparent and some amount of the investigation or bisection is necessary to get things back to a stable state.  This is completely expected state of affairs in an active and growing project, and in most cases is a healthy sign - for it to be otherwise would actually be somewhat worrying.   Still, it can be frustrating and requires constant vigilance - the longer problems go undetected, the more difficult it is to find the culprit (e.g. at the moment I'm bisecting a problem where the desktop icons disappeared again due to some commit of the past couple days - had I not built and boot-tested this machine yesterday, it would have taken much longer and been more difficult to locate the problem, as it has in the past).   Being vigilant about the six meta-intel BSPs I currently maintain (crownbay, crownbay-noemgd, emenlow, fishriver, sugarbay, and jasperforest) has been manageable so far, but just barely so.  In addition to making sure everything still works most of the time, during each release cycle I also need to go through the exercise of upgrading all the BSPs to work with new package versions, and upgrading some of those packages themselves, the main one for BSPs being the kernel.  There actually is quite a bit of work involved in moving the BSPs to a new kernel, mostly things like moving out-of-tree drivers (mainly graphics) to the new kernel version, keeping on top of changes to kernel options or additional options that we should be taking advantage of, and in general some amount of movement between what we keep in our kernel branches vs what's gone upstream since the last cycle, etc.  Most of the time it goes smoothly, but every cycle there's always one or two BSPs that give me real problems - for 1.1 it was emenlow graphics, for 1.0 crownbay the gigabit ethernet driver.   My guess is that I could add two or three more BSPs to the list (which is actually what I expect to be doing during the next cycle) and still be able to maintain both the BSPs and my sanity, but it's apparent to me that beyond that, things aren't going to scale without some kind of help.  Obviously, one form that kind of help could take would be to simply throw more people at the problem, but even if there was a budget for that, which there typically isn't, you'd never do that without first maxing out what you could do by instead working more efficiently.  Thankfully, at least half of the problem should be very amenable to being taken off the table by a bit of automation. Obviously, you can't automate the tasks mentioned above, which require digging into problems or looking at options or forward-porting code, but you really should be able to automate most of the rest of it, the rote things that actually do point you to the tasks that need intervention.  Doing those things manually can take a non-trivial amount of time and attention and for the most part really shouldn't be done by a human i.e. building and booting images for all of the supported BSPs, at least to the point where you can point a human tester (me, to start with) at the box and say, 'this BSP is ready for testing, switch the monitor over and see if all the icons are still there'.   It just so happens, not coincidentally, that I also have a lot of idle hardware lying around, which pretty much exactly matches the platforms I need to test on. ;-).  It also happens that I'm a big proponent of eating my own bird food, killing multiple birds with one stone, and whatever other ornithological metaphors make sense, so what I'm planning to do is take all of these machines, some of which are actually first-class build nodes e.g. Sandy Bridge and Jasper Forest, put them all on a remote power strip and kvm switch and have them coordinate pulling down and building the latest yocto source, iterating through each image and machine type.  Once each image is ready, it will automatically get 'installed' on the appropriate test machine (using gpxe network boot support or some such) and booted and at that point a message will be sent to the 'operator' (me, to start with) that image X on machine X is ready for testing.  Once I know that, I can then switch over and do as little or as much testing on the machine as necessary (much of the time just looking at the screen and seeing icons would suffice).  While it doesn't sound like much, automating all of this would save me the trouble of doing it manually, which is tedious and forces me to context-switch into and generally spend more time than I'd like focusing on non-core tasks that actually do require a human to look at.  It would also serve to more consistently detect problems as early as possible, making the potential bisection distance between breakage as small and manageable as possible.   As mentioned, many of these idle machines are heavy-hitters and obviously cut out for the hard-core task of building Yocto images. Many of the other ones, however e.g. the Atom machines aren't quite up to such heavy lifting, but being the taskmaster that I am, in this economy I expect even the babies to work, and am hoping in a later iteration to be able to split up the work of building and testing an image into smaller pieces that can be farmed off to whatever machine is available, each according to its abilities.  How to determine those abilities is an interesting question - what I'm actually hoping to do is to tie into performance data that we can make available on each machine using the various performance tools we have available in the Yocto images all of these machines will be running.  These performance metrics would be used in some sort of simple 'cloud protocol' to partition and farm out work to the set of available machines in the grid.  It might even be useful to anticipate this and start out with a trivially simple protocol that could be used to bootstrap the basic build coordination described above, but that's a stretch goal as I see it at this point - for now it would suffice to simply have the work taken off my hands.   At this point in the development of the Yocto Project, I'm probably the only person faced with these problems related to the scalability of BSP maintenance, but there are also a couple other things that I'll be working on in the near future that I'm expecting will create similar problems for other people and which I'll describe in a future blog post.  Without going into too much detail now, it's basically a plan to make creating and maintaining new BSPs very simple and easy to do. If successful, there should soon be other people facing these problems too (which of course are the kind of problems you don't mind having), and hopefully by the time that happens, we'll have something like what I described above in place to make BSP maintenance more scalable to go along with it.             [Less]