If you follow Linux news, you have probably noticed confusion about what
makes a good
Linux BSP. I’m reminded of an old friend in IT whose definition of
“released” meant that the program worked at least once on his
While the tech industry drones on relentlessly about quality, the fact is that
even after Linux code reaches millions of lines, many software engineers still
expect that the code is only reference code and hasn’t been tested
thoroughly. More than that, most people believe that when you adopt Linux,
you’re on your own – without support. Companies often task
their engineering teams to validate, harden and test the code before
it’s released into production – this critical step can drain
your precious time and resources. This is why we took the approach
to develop a Software Development Kit (SDK)
that helps you shave months and months off your development schedule.
What is our SDK really? Is it just a BSP?
tendency for most customers is to call the
a Board Support Package (BSP). It’s more than that.
While the thought has much truth in it, the fact is that our SDK
is really much more than a BSP. I will use BSP where it makes sense in
context, but when I say BSP in context, I assume the BSP is created from the
SDK. SDK means multiple BSPs in many ways.
What’s in a name?
It really starts with the
name. When our software teams wanted to support
QorIQ Multicore Processors, and
communications processors built on Power Architecture® technology, with
Linux BSPs, they put lots of thought into how they would do this.
Since we have a long history of common IP blocks that are rendered in our
silicon, we created multiple BSPs out of a single body of work. The body of
work is similar to a library of Linux IP that would support the silicon IP.
All an engineer would have to do would be to “assemble” the
Linux IP pieces in the same configuration as the processor to create an
overall processor-level BSP. To support this overall, all-in-one library
of Linux IP, we called it a software development kit. Think of it like this
– you get the software development kit – you add some of your
own IP, then press a button (an exaggeration) and shazzam – you get the
ideal BSP for your board. So now you know why we call the SDK
We also assembled software tools and other Linux software along with the Linux
IP for the processors, so you get an all-in-one starting point for Linux
development for our communications processing platforms and
But that isn’t the only place you can get our SDK. In fact, if you
think about where you can get the BSP (I shifted gears on you – sorry),
you might learn something about us. One thing that separates
us from the pack is that we are a company that strives to develop much of
our software completely in the open.
How would you know we develop code in open source?
would know this because you can find some of the most recent Linux code for
our communications processors directly on
kernel.org, and now Yocto. How recent is our code in the public? Well, on our
GIT, it is very, very recent. On places like kernel.org, we are one of
the top contributors. We submit our code as a release candidate to
kernel.org (and others), and we have a strong track record for getting code
accepted into the mainline. I’m answering this question this way
because it is important to understand that you don’t just jam code into
kernel.org. It goes through a vetting process. It isn’t
automatic. But you can test us on this… our code is pretty
darn recent. Many, many of our customers simply point to kernel.org as their
starting place for development.
Our networking software development kit, in-depth
Let’s dive into some details and see how we might be a little bit
different than others.
Accelerators and specialty silicon IP
One of the really
neat things about our silicon IP is that our communications processors
have specialty IP in them that you simply can’t get from your corner
semiconductor foundry store. This IP literally has thousands of
development years. It has the cumulative feedback of thousands of customers
and tens of thousands of software developers. The numbers are staggering.
We’ve had feedback from all different industries. When you get a
communications processor, you are getting the cumulative wisdom of potentially
hundreds of thousands of engineers. That creates a special challenge for
the Linux developer though. It isn’t reasonable to expect someone who
knows Linux and a little bit about communications to properly support a
specialty peripheral that has been undergoing constant improvement for 10
– 15 years. The point is: When you start with our SDK;
you can bet you are getting the LATEST and most modern support for some of
the world’s most advanced semiconductor peripherals. Let’s go
over just a couple of peripherals.
The answer is probably
“yes.” The question is “Do we accelerate
[insert your favorite esoteric cryptographic algorithm here]?”
How do we do this?
We maintain a cryptographic center of excellence.
We don’t call it Area 51, but as I write this, it
strikes me as funny because it isn’t too far from Area 51 –
hint, hint. What they’ve created is a common security IP block
which we have used over and over again on many, many semiconductors. Not
only do we use this center of excellence for developing our crypto
acceleration engines, we use them to provide security consultation to other
parts of our company. It becomes a virtuous circle. We use them more, they
learn more, they learn more, share more and then we use them more.
The output is an amazing piece of programmable crypto acceleration that
you’ll find across many solutions, including
QorIQ communications processors and
i.MX applications processors. We’ve even created a new standalone variant called the
C29x family crypto coprocessors
because customers sometimes want access to our engine, but for some reason or
another don’t need a fully loaded processor.
I had one marketing person get frustrated because he created this huge
spreadsheet full of all of the combinations of security algorithms he wanted
supported, and the software team wasn’t filling them in fast enough.
But what he failed to appreciate is that the crypto acceleration engine is
programmable. Wanting all combinations of every algorithm under the sun
instantly is like saying you want every single program you could think of on
your PC … right now!
So, here is the million dollar question: If you want accelerated crypto
operations on your Linux-based project and you’re using a QorIQ
communications processor, how would you expect anyone but us to support
the crypto in the best way? Think about it. Our software engineers have
immediate access to the engineers behind our crypto engine – and when
you partner with us, you can TOO.
There is no way I can do justice to the
communications IP from us. Not in a blog article. Not even in a book. But
maybe some short notes.
We have really changed the history of networking silicon when we
introduced the 68HC302 processor. While for many that processor is simply
something that happened in the past, the truth is that the 68HC302 lives on.
That enlightened processor became the lightening rod for learning how to
accelerate Internet and Ethernet packets. We have examined every bit
of every line item in every algorithm and learned how to best reduce it to
silicon, while retaining all of the software flexibility.
We didn’t do it all at once. We started with our communications
processor (CP), then created the DPAA and now, far from done, we’re
once again innovating by creating an entirely new way of thinking about packet-based communications with our
Layerscape architecture. Through all of this, we had the first Linux running — on our
communications processors. We will always have the first experience with
putting software on the IP.
Even though we may have the most experience
with our silicon IP, many customers wonder “How good is your Linux SDK,
really?” In my mind, I equate quality with testing, but it really
is much more, for example, it is the process you use. Let’s go into
what we do behind the scenes with our SDK.
Every SDK is tested rigorously before
we release it. By releasing, I mean that we make a tested snapshot available.
We do this monumental task approximately twice a year. As monumental as our
releases are, we do a considerable amount of testing, in far less time than
you might expect. Why?
Over the years, we decided that to be efficient in Linux development we
needed to create the ability to manage multiple Linux kernel versions,
multiple boards, multiple chips, multiple tests and keep track of all of
these multiples. The problem is a geometric progression. As a result,
behind Linux development is a very large and interconnected Linux development
and test environment. Within reason, at any one point in time, a software
engineer can connect to a board contained in one of our worldwide board farms
(more on board farms later), create a BSP from any number of different Linux
kernels, program the kernel into the board’s flash memory and then
stimulate the board any number of ways. Finally a fully capable CodeWarrior
multicore remote debug session is really no problem. If we wanted, we can even
hook into our IC design systems to help match the design to the upcoming
software even better. Since a single software engineer can do what I
described, it turns out that we can also do this with automated programs.
Because we can do this with automated programs – we do. In fact,
we do this all the time. We do 24/7 testing. We are constantly monitoring
key performance parameters, and when engineers check-in code in any one part
of Linux, it is predictably exercised thoroughly with a fairly short wait. By
the time our code gets to kernel.org, it has been ran — and ran and
ran. We also make decisions on the kernel versions that the code should run
on. This means we are testing on the backward and forward ports that we have
as well. Quick note on debugging: While there are many development
tools on the market, the one thing that
CodeWarrior Development Suites for Networked Applications
gives you is the ability to call us and get processor-Linux-debug support
with no third party involved.
Custom Linux tests and more tests … and even more
It is no secret that in Linux there are lots and lots of tests floating
around. It is no secret that we will use as many of these as possible. But if
we just tested using the test suites you would find on the Internet, then we
would be no better than the next company. So we have written a few tests of
our own. Several thousand to be exact . Even better – we create our own
reference designs. One, as an example, is the network attached storage (NAS)
and we run the NAS reference design code through a few paces as part of our
testing. So, when we say we write a test, sometimes it is an entire
application. Some of our tests have a manual component to them. For example,
testing hot-plugging USB is a manual test. However, we keep track of our
percentage of manual tests and are adding equipment all the time so that as
many manual tests as possible can be completely converted to 100% automated
tests. When we run these tests, we monitor key performance indicators. We
track these indicators regularly to make sure that they stay within
tolerance. Each test has key performance indicators, if applicable, as
well as expected results.
Production test equipment
I remember going to a
customer site several years back and seeing the customer hook up a beat up
laptop via Ethernet to their board under test. Then they started some program,
I think maybe even it was still named “a.out”, to reflect
incoming packets from the board under test back to the board under test. This
was the extent of much of their testing. Nothing could be farther from the
case. We both purchase and lease production network test equipment. These
are not little rinky dink lab PCs. These are industrial strength pieces of
networking test equipment, capable of injecting all different types of packets
– voice, data, streaming, video and specialty, including injecting
packet errors and many other things that make sure our processors handle your
packets the right way. We don’t have just one or two of these things
lying around. When we think of network traffic, we’re thinking about
train car loads of traffic. We measure our capacity in Terabits. The point is,
our SDKs are tested with production level traffic.
Earlier I discussed the notion of a
board farm. A board farm is a room full of interconnected development boards
with network managed power, reset, console, critical IO and network accessed
JTAG. We keep on hand boards that are very old as well as boards that are very
new. These boards are accessible for automated testing, as well as for
individual developers. You might wonder how this helps the quality of the SDK?
What it does, is it makes it so that nuances that might normally be missed
when a piece of code only runs on a single board get exposed when running on
many boards. It is just one more way we try to make the SDK higher
quality than anyone would ever expect. Do we support all boards that
we’ve ever created for networking? The honest answer is
“no”. Since this is a geometric multiplication problem (Linux
kernels supported * boards * software architectures * etc.) we eventually will
allow older boards to stay on an older SDK. If a customer wants to get an
older board updated to the latest SDK, we do engage in custom forward ports.
But as a general rule, we’ll manage 20 of our latest boards in our
Multiple CPU Architectures
When you think about the
fact that our processors support multiple CPU architectures (32-bit Power
Architecture® e500 core, 64-bit e5500 and e6500 Power Architecture
cores, 32-bit ARM® Cortex®-A7 cores and 64-bit Cortex-A8 cores),
you would expect that it would be a problem. No. What it has allowed
us to do is to exercise how Linux compiles between the different
architectures and to learn how to manage and accommodate for the differences.
For example, our tests “just know how” to map themselves from
32-bit to 64-bit tests. They also need to be created to handle the differences
in timing without causing false failures. By squashing out as many differences
as possible by conquering them, we can utilize this to make a better,
more useful SDK that makes better BSPs. In some ways, you are more likely to
be chip independent by using our BSP than a BSP that only supports a single
Multiple versions of Linux
The way we handle Linux
is that we maintain large repositories of different versions of Linux –
sort of like a release history file. We’ll watch Linux closely, and
about once a year we’ll sync to the latest kernel that is going on Long
Term Support (LTS – a Linux term). Then we’ll refresh that
particular kernel version in about six months. This keeps us very fresh with
the latest kernels, but also does a great trade-off with our customers in
terms of making sure our SDK is solid as a rock. But if you think about it,
what it means is that we have forward and backward porting software built
into our DNA. Our automated systems wouldn’t work well if we
didn’t have this part figured out. And knowing how the code will have
to travel between kernel versions means we can write the code knowing this
– ergo code that simply just “works better”.
Multiple Software Architectures
The last quality item
I’ll discuss is the fact that we are as far along with
virtualization as any semiconductor company in the world. As previously noted,
our SDK has to be able to accommodate multiple software architectures.
It’s important to note that we actually TEST all of the architectures
and check the key performance indicators for each. In other words, whether you
are running SMP, AMP, mixed SMP/AMP, Hypervisor (two different types), or
complex guest OS situations, we’ve already tested most of these for
you. Obviously, for guest OS testing, we use Linux due to licensing and
availability for our engineering teams. Since we test so many software
architectures, we also have to write the tests that can work in multiple
software architectures as well as the CPU architectures.
I’m hoping that by now you realize that we are thinking about the
Linux quality at a level that is hard to beat. It would simply cost
hundreds of millions of dollars to match the work that have done.
So far I’ve discussed our technology
in detail, but you get so much more than you’d expect when you choose
us. One of the big “wins” you get with the SDK is experience.
All of the monumental work I’ve described was performed by dedicated
engineers. Most people know we have many, many chip designers working on
our state of the art communications processors. But what you might not know is
that the ratio of software developers to chip developers is currently 1 to 1
and growing. The point is: To leverage our high performance, feature rich,
security-enabled processors, you need software and we
are committed to the complete solution. Just to underscore
our engineering experience, you can go to
and find the yearly report called “Who is writing Linux”.
In the 2013 report, we ranked as the #13 contributor in the world,
alongside networking and IT leaders like IBM and Cisco. You can’t be a
fly-by-night and be on that list. In fact, you have to be pretty good since
the keepers of Linux scrutinize contributions. Related to our experience
is the emphasis that we put on ease of use when it comes to the
customer’s experience with our products. We really want people, even
hobbyists who are working on projects for maker fairs, to be able to utilize
our processors. As a result, we are committed to the Yocto project. We
experienced first-hand the difficulties of supporting all the multiples (CPUs,
CPU architecture, kernels, accelerators, boards, etc.) and we brought that
experience to the Yocto project. We are literally trying to make it so that
you simply get a recipe for the board, type a command to execute the recipe
and you get instant BSP – working perfectly for your board. While
I’m oversimplifying, I’m not understating the goal and wisdom
we bring to the table.
One area where our technology goes deep
security. The very word sounds so specific, but it isn’t because it is such a
multifaceted topic. You have a need for security. They need to
manufacture, deploy, boot, execute and communicate in a way that keeps bad
actors from infiltrating their system. Especially when it comes to security,
our SDK has the collective wisdom of all of our customers, our developers and
our semiconductors. We’ve thought through the entire value chain. Even
Arm Technology-Based Solutions, we’ve wrapped them in IP, which means that the TrustZone technology
from Arm gets all the benefit of our proven manufacturing and boot logic.
If you are struggling in making the entire value chain secure and you have
concerns, you might consider calling us.
Now that you know, can you afford NOT to consider our SDK?
for communications processors is simply NOT your ordinary Linux BSP. It really
is so much more. Ask your Linux developer what they are doing on Linux. Ask
them specific questions about what you’ve learned here. Then ask
yourself… what if you got in a bind? When the going got tough who could
you rely on to answer your questions? What if you needed help in any of the
following areas? Who would be better suited to answer your questions?
- How do you forward port an older BSP to the latest version?
- How can I increase the performance of my system?
- What do I need to develop products for long-term production?
- How can I increase the quality of my system?
- How can I lower the power consumption of my system?
Can you help me find a terribly deep and difficult bug in my system? (we
have tools and techniques that I simply didn’t have space to mention
It will be us, the company that knows Linux inside and out, who knows
their products better than anyone on the planet. Finally, because
you really can lean on us when needed, because we have so much to offer,
and we now invite you to ask us if we can help you with your Linux-based projects.