Tracing and visualization of embedded Linux systems

by Dr. Johan Kraft, Percepio 

Debugging embedded software can be a challenging, time-consuming and unpredictable factor in development of embedded systems. Detecting errant program execution begs the question “How did the software reach this state?”  What combination of inputs and timing resulted in the error, and why? Tracing can often provide the answer.

Tracing entails recording software behaviour during runtime, allowing for later analysis of collected trace data. Tracing is most often a development bench activity, but tracing can also be enabled for production use, continuously active to record behaviours and catch errors post-deployment. Production tracing can be an effective technique for detecting rarely-manifested errors that are therefore are difficult to reproduce in a debugger. These can include situations where the system responds more slowly than expected, gives incorrect or suboptimal output, freezes up or crashes.  Tracing can be performed either in hardware (in the processor) or in software. Hardware-based tracing generates a detailed instruction-level execution history, while software-based tracing focuses on selected events, typically in the operating system and important application-level interfaces. Hardware-generated trace provides details regarding control-flow and does not impact the execution of the traced system, but does require special equipment and a trace-enabled hardware platform.

Software-generated trace does not require any special hardware and can even be deployed in shipped products like a “black box” flight recorder used in aviation. Moreover, software trace allows for storing data, such as system call parameters, while hardware trace is typically limited to control-flow only. Software tracing does induce some CPU overhead, but typically less than 1 %. Software tracing relies on target system RAM for storing the trace data, at least initially, but the RAM buffers are usually configurable to allow for balancing RAM usage vs. execution history length.  Tracing is especially important for systems integrating an operating system. A central feature of OSes is multi-threading - the ability to run multiple programs (threads) on a single processor core by rapidly switching amongst execution contexts. Multi-threading, however, makes software behaviour more complex, and affords the developer less control over run-time behaviour as execution is pre-empted by the OS.

Tracing in Linux using LTTng

LTTng is an open source technology for software-based tracing in Linux. LTTng is very efficient, proven in use and supported by most Linux distributions. The LTTng kernel tracer can record scheduling events, system calls, IRQs, memory management, and other kernel-level activities. LTTng user-space tracer (LTTng-UST) tracks user-defined events in application code. Trace data is initially stored in RAM, but can be continuously flushed to disk or offloaded to another system over a network connection. Another option is to keep the trace data in a RAM ring buffer, overwriting earlier events when the buffer becomes full. Although LTTng is based on software instrumentation, it does not require a rebuild of the Linux kernel. Rather, LTTng leverages tracepoints which already exist at strategic locations throughout the kernel. These tracepoints are placeholder function calls, dormant by default but activated by LTTng.

Using LTTng-UST, developers can trace custom events in user-space application and library code. It is even possible to trace selected function calls without modifying source code, by creating a shared object file with wrapper functions (Figure 1) containing tracepoints, specified in LD_PRELOAD when launching the application.

percepio140313fig1

Figure 1 Transparent tracing of function calls using wrapper functions and LD_PRELOAD.

LTTng-UST function wrapping is completely transparent to application code, with no need for recompilation. On the first call, the wrapper function looks up the address of the original function and stores it in a function pointer (as with malloc in Figure 1 above)

Visualization of LTTng traces in Tracealyzer

LTTng outputs binary data files, so a tool is required for analysis. Babeltrace can convert the trace data to text files, but it is hard to “see the big picture” from vast amounts of trace data in text format. A visualization tool greatly facilitates analysis since the human brain is much better at spotting patterns in images than in text data. Tracealyzer is a family of trace visualization tools developed by Percepio AB in Sweden. Tracealyzer for Linux is designed to visualize LTTng trace data and provide multiple graphical perspectives to facilitate analysis. Tracealyzer understands and highlights dependencies among related events in trace data, for instance sending and receiving of a semaphore signal. Highlighting such dependencies makes it easier to understand operating system behaviour, e.g., why some threads are blocked and others triggered.

percepio140313fig2

Figure 2 Tracealyzer main view - scheduling and various events.

The main trace view in Tracealyzer (Figure 2) presents events (e.g., system calls) along a vertical time-line using colour-coded labels. Labels can be filtered in several ways and automatically adjust to avoid overlapping. Label background colour indicates status and type of operation, e.g., red labels show system calls that block the calling thread. Custom application events from LTTng-UST can be configured to appear either as service calls (e.g., malloc) or as user events (generic debug messages). The Tracealyzer main view is complemented by 20+ additional views, showing CPU load, response times, kernel blocking, scheduling intensity, kernel calls and other traced events. User events can be shown separately in a text log view, and user event arguments can be visualized as signal plots. All views are interconnected, meaning that clicking on a task, event or data-point will open another relevant view with the clicked entity highlighted. This interconnection makes it easier to use multiple perspectives when studying a specific interval in the trace.

Summary

Tracing provides a powerful tool for analysing multi-threaded software systems. On Linux, tracing is enabled by LTTng, a mature and proven open source solution. Percepio Tracealyzer for Linux lets developers visualize LTTng trace data through multiple, interconnected graphical views. Tracealyzer makes dense and voluminous trace data more accessible to software developers, giving them greater benefit from tracing. Tracealyzer helps developers make sense of complex trace data, find bugs and tune performance, and thereby produce better software.


Related


Hardware-based AES Encrypted Storage Solution

Secure data encryption is essential for a wide variety of mission-critical applications pertaining to both civilian matters and national security. These sectors both require comprehensive safeguards t...

Give Your Product a Voice with Alexa

Join us for a deep dive into the system architecture for voice-enabled products with Alexa Built-In. Device makers can use the Alexa Voice Service (AVS) to add conversational AI to a variety of produc...

The two big traps of code coverage

Code coverage is important, and improving coverage is a worthy goal. But simply chasing the percentage is not nearly so valuable as writing stable, maintainable, meaningful tests. By Arthur Hick...

Securing the smart and connected home

With the Internet of Things and Smart Home technologies, more and more devices are becoming connected and therefore can potentially become entry points for attackers to break into the system to steal,...

 

nVent Schroff at Embedded World 2019

The theme of the nVent Schroff booth at Embedded World 2019 was “Experience Expertise – Modularity, Performance, Protection and Design”. Join us as our experts give an overview of th...


Garz & Fricke Interview at Embedded World 2019 with Dr. Arne Dethlefs: We are strengthening our presence in North America

Through its US subsidiary, located in Minnesota, Garz & Fricke is providing support for its growing HMI and Panel-PC business in the USA and Canada while also strengthening its presence in North A...


SECO's innovations at embedded world 2019

In a much larger stand than in previous years, at embedded world 2019 SECO showcases its wide range of solutions and services for the industrial domain and IoT. Among the main innovations, in this vid...


Design and Manufacturing Services at Portwell

Since about two years Portwell is part of the Posiflex Group. Together with KIOSK, the US market leader in KIOSK systems, the Posiflex Group is a strong player in the Retail, KIOSK and Embedded market...


Arrow capabilities in design support

Florian Freund, Engineering Director DACH at Arrow Electronics talks us through Arrow’s transformation from distributor to Technology Platform Provider and how Arrow is positioned in both, Custo...


Arm launches PSA Certified to improve trust in IoT security

Arm’s Platform Security Architecture (PSA) has taken a step forward with the launch of PSA Certified, a scheme where independent labs will verify that IoT devices have the right level of securit...


DIN-Rail Embedded Computers from MEN Mikro

The DIN-Rail system from MEN is a selection of individual pre-fabricated modules that can variably combine features as required for a range of embedded Rail Onboard and Rail Wayside applications. The ...


Embedded Graphics Accelerates AI at the Edge

The adoption of graphics in embedded and AI applications are growing exponentially. While graphics are widely available in the market, product lifecycle, custom change and harsh operating environments...


ADLINK Optimizes Edge AI with Heterogeneous Computing Platforms

With increasing complexity of applications, no single type of computing core can fulfill all application requirements. To optimize AI performance at the edge, an optimized solution will often employ a...


Synchronized Debugging of Multi-Target Systems

The UDE Multi-Target Debug Solution from PLS provides synchronous debugging of AURIX multi-chip systems. A special adapter handles the communication between two MCUs and the UAD3+ access device and pr...


Smart Panel Fulfills Application Needs with Flexibility

To meet all requirement of vertical applications, ADLINK’s Smart Panel is engineered for flexible configuration and expansion to reduce R&D time and effort and accelerate time to market. The...


Artificial Intelligence

Morten Kreiberg-Block, Director of Supplier & Technology Marketing EMEA at Arrow Electronics talks about the power of AI and enabling platforms. Morten shares some examples of traditional designin...


Arrow’s IoT Technology Platform – Sensor to Sunset

Andrew Bickley, Director IoT EMEA at Arrow Electronics talks about challenges in the IoT world and how Arrow is facing those through the Sensor to Sunset approach. Over the lifecycle of the connected ...


AAEON – Spreading Intelligence in the connected World

AAEON is moving from creating the simple hardware to creating the great solutions within Artificial Intelligence and IoT. AAEON is offering the new solutions for emerging markets, like robotics, drone...


Arrow as a Technology Provider drive Solutions selling approach

Amir Sherman, Director of Engineering Solutions & Embedded Technology at Arrow Electronics talks about the transition started couple of years ago from a components’ distributor to Technology...


Riding the Technology wave

David Spragg, VP, Engineering – EMEA at Arrow Electronics talks about improvements in software and hardware enabling to utilize the AI capabilities. David shares how Arrow with its solutions is ...


ASIC Design Services explains their Core Deep Learning framework for FPGA design

In this video Robert Green from ASIC Design Services describes their Core Deep Learning (CDL) framework for FPGA design at electronica 2018 in Munich, Germany. CDL technology accelerates Convolutional...


Microchip explains some of their latest smart home and facility solutions

In this video Caesar from Microchip talks about the company's latest smart home solutions at electronica 2018 in Munich, Germany. One demonstrator shown highlights the convenience and functionalit...


Infineon explains their latest CoolGaN devices at electronica 2018

In this video Infineon talks about their new CoolGaN 600 V e-mode HEMTs and GaN EiceDRIVER ICs, offering a higher power density enabling smaller and lighter designs, lower overall system cost. The nor...


Analog Devices demonstrates a novel high-efficiency charge pump with hybrid tech

In this video Frederik Dostal from Analog Devices explains a very high-efficiency charge-pump demonstration at their boot at electronica 2018 in Munich, Germany. Able to achieve an operating efficienc...