Posts tagged cmake
Using address sanitizer (ASan) to debug C/C++ memory issues in Android applications

At NRB Tech, we sometimes use C or C++ libraries across multiple platforms, i.e. in the firmware, apps, and even on the server. This code sharing enables a solid foundation to be built which is well unit tested and works the same everywhere. It is very useful for data processing – encoding and decoding binary data formats for data transmission over Bluetooth, and cryptography – encrypting and decrypting data transmitted between devices.

However, this approach can cause issues. C is notorious for its unsafe memory management, which allows great flexibility, but can lead to memory corruption, data loss, data leakage, and crashes. Using best practices and unit testing code can mitigate some risk, but bugs can still occur, and debugging these issues can be tricky.

When developing for Android, Google have a number of tools available to help debug memory issues in NDK (Native Development Kit) C/C++ code. The approach recommended by Google is to use ASan, however the ASan documentation is currently (March ‘22, Android Studio 2021.1.1, Gradle 7, APIs 27+) missing crucial information to enable developers to debug these issues. In this article, we will provide a step-by-step guide to debug a memory issue. We have also create a working example project on GitHub.

From a stock Android project with CMake:

  1. To start your App with the address sanitizer library, your App needs to be run with a wrapper script. This is a script that is run on your test device and starts the application, modifying the environment variables and/or command arguments. As such, this script needs to be copied into your App at build time. Additionally, we need to copy the ASan libraries out of the NDK and into our project so they are copied into your App when building. Both of these steps need to be done for 4 CPU architectures, so can be laborious. Instead of doing this by hand, we can use a script to set this up. Run this from the root of your project, modifying the NDK version if required.

    #!/bin/bash
    
    cd "$(dirname "$0")"
    
    # This script sets up the required directories in app/src/sanitize. It can be run again if you change the ndk version.
    
    ndkversion="24.0.8215888"
    
    # store wrap.sh to variable
    read -r -d '' script <<'EOT'
    #!/system/bin/sh
    HERE=$(cd "$(dirname "$0")" && pwd)
    
    cmd=$1
    shift
    
    export ASAN_OPTIONS=log_to_syslog=false,allow_user_segv_handler=1
    ASAN_LIB=$(ls "$HERE"/libclang_rt.asan-*-android.so)
    if [ -f "$HERE/libc++_shared.so" ]; then
      # Workaround for https://github.com/android-ndk/ndk/issues/988.
      export LD_PRELOAD="$ASAN_LIB $HERE/libc++_shared.so"
    else
      export LD_PRELOAD="$ASAN_LIB"
    fi
    
    os_version=$(getprop ro.build.version.sdk)
    
    if [ "$os_version" -eq "27" ]; then
    cmd="$cmd -Xrunjdwp:transport=dt_android_adb,suspend=n,server=y -Xcompiler-option --debuggable $@"
    elif [ "$os_version" -eq "28" ]; then
    cmd="$cmd -XjdwpProvider:adbconnection -XjdwpOptions:suspend=n,server=y -Xcompiler-option --debuggable $@"
    else
    cmd="$cmd -XjdwpProvider:adbconnection -XjdwpOptions:suspend=n,server=y $@"
    fi
    
    exec $cmd
    
    EOT
    
    # loop through all architectures and write/copy the required files
    archs=( "arm64-v8a" "armeabi-v7a" "x86" "x86_64" )
    asanarchs=( "aarch64" "arm" "i686" "x86_64" )
    
    for i in "${!archs[@]}"; do
      mkdir -p app/src/sanitize/{jniLibs,resources/lib}/${archs[$i]}
      cp $ANDROID_HOME/ndk/$ndkversion/toolchains/llvm/prebuilt/*/lib64/clang/*/lib/linux/libclang_rt.asan-${asanarchs[$i]}-android.so app/src/sanitize/jniLibs/${archs[$i]}/
      echo "$script" > app/src/sanitize/resources/lib/${archs[$i]}/wrap.sh
    done
  2. In your CMakeLists.txt, add the following (replacing ${TARGET} as required with the name of your target):
    if(SANITIZE)
         target_compile_options(${TARGET} PUBLIC -fsanitize=address -fno-omit-frame-pointer)
         set_target_properties(${TARGET} PROPERTIES LINK_FLAGS -fsanitize=address)
     endif()
  3. In your app's build.gradle, to your android.buildTypes closure, add:
    sanitize {
        initWith debug
        externalNativeBuild {
            cmake {
                arguments "-DANDROID_STL=c++_shared", "-DSANITIZE=TRUE"
            }
        }
    }
  4. Sync Gradle and switch your active variant to "sanitize" in the "Build Variants" tab (bottom left)
  5. Debug your App and reproduce the issue. You should see output from address sanitizer (filter by "wrap.sh"). In the stack traces you will see lines like:

    #1 0x5e2cdaf76c  (/data/app/~~lTqqcXu2okmchd_nMeTm7w==/io.nrbtech.asanexample-spc73OiysYUjJ2984-N4kw==/lib/arm64/libasanexample.so+0x76c)

    Where libasanexample.so is the name of the compiled C library and 0x76c represents the offset into the library the stack trace is pointing to.

  6. We need to convert that address offset to a file and line number to understand where the problem occurred. The following script uses llvm-symbolizer in the NDK to do this.

    #!/bin/bash
    
    ndkversion="24.0.8215888"
    libname="asanexample" # change to your library name
    
    # default arch
    arch="arm64-v8a"
    
    POSITIONAL_ARGS=()
    
    if [ $# -eq 0 ]; then
       echo "Pass an address as the last argument, use -a to provide architecture"
       exit 0
    fi
    
    while [[ $# -gt 0 ]]; do
     case $1 in
       -a|--arch)
         arch="$2"
         shift # past argument
         shift # past value
         ;;
       -*|--*)
         echo "Unknown option $1"
         exit 1
         ;;
       *)
         POSITIONAL_ARGS+=("$1") # save positional arg
         shift # past argument
         ;;
     esac
    done
    
    if [ ${#POSITIONAL_ARGS[@]} -eq 0 ]; then
       echo "Pass an address as the last argument"
       exit 1
    fi
    
    cd "$(dirname "$0")"
    
    $ANDROID_HOME/ndk/$ndkversion/toolchains/llvm/prebuilt/*/bin/llvm-symbolizer --obj=app/build/intermediates/cmake/sanitize/obj/$arch/lib$libname.so ${POSITIONAL_ARGS[0]}

    Copy this to the root of your project and modify for your C library name and NDK version. You can then run this script like so:

    % ./symbolizer -a arm64-v8a 0x76c
    Java_io_nrbtech_asanexample_Test_test
    /Users/nick/Documents/Repos/ASanExample/app/src/main/cpp/test.c:7:5

    The output contains the path to the line of code pointed to by the stack trace, enabling you to find where the issue occurred and figure out the problem. In this case, it points to a line where we free'd a pointer twice.

  7. ASan can slow down your App, so be sure to switch back to the debug variant when you have resolved your issue.

Good luck resolving your own memory problems in Android Apps! If you need further help, feel free to reach out to NRB Tech!

Nicktutorial, Android, cmake, C, ASanComment
Using CMake for Nordic nRF52 projects

CMake is a popular C/C++ build system (or an “open-source, cross-platform family of tools designed to build, test and package software” according to the website) that we use at NRB Tech with firmware projects and cross-platform libraries. CMake can also be used with CLion, an excellent C/C++ IDE. Nordic officially recommends and supports Segger Embedded Studio as its IDE of choice, however CMake makes managing dependencies and unit testing much easier.

For example, a common development architecture we use at NRB Tech is to share a C library or libraries between firmware, utilities, and Apps, to reduce code duplication and bugs and ensure mission-critical code is well unit tested. With CMake it is trivial to manage this library, generating Makefiles or Xcode projects as required and integrating as a dependency into other CMake projects.

Nordic do not directly provide support for CMake in their SDK, but their mesh SDK does use CMake, and this can be adapted to use CMake for the Nordic SDK in non-mesh projects.

nRF5-cmake-scripts

Nordic’s mesh SDK was not designed for building external, custom, non-mesh projects, but with a little external support it can be used for this purpose, which is what our nRF5-cmake-scripts project does – it uses the mesh SDK scripts where possible, and defines its own functions for everything else – including adding features that Nordic’s mesh SDK doesn’t have, such as adding bootloader targets and creating DFU packages. It also defines functions for easily including many of the libraries in the Nordic SDK. Let’s have a go at creating a new project from scratch.

Installing the dependencies

  • Download and install JLink

  • Download and install Nordic command line tools

  • Download and install ARM GNU Toolchain

    • On Mac, this can be installed via homebrew:

      brew tap ArmMbed/homebrew-formulae
      brew install arm-none-eabi-gcc
  • Install nrfutil via pip (Python package manager):

    pip install nrfutil
  • Install CMake and Git

Creating a CMake based Nordic firmware project

The finished project can be found here.

First, in a terminal/command window clone the base project to set up the project structure:

git clone --recurse-submodules https://github.com/NRB-Tech/nRF5-cmake-scripts-example-base.git .

Run a script to clean up the project ready for your own use (on Windows, run in git bash by right clicking in directory > "Git Bash here"):

./cleanup.sh

Copy the example CMakeLists.txt as recommended in the nRF5-cmake-scripts readme:

cmake -E copy nRF5-cmake-scripts/example/CMakeLists.txt .

_Note: You may also need to edit some of the variables in this file for your platform, such as setting NRFJPROG, MERGEHEX, NRFUTIL and PATCH_EXECUTABLE manually if they are not in your PATH._

Then we can use the script to download the dependencies:

cmake -Bcmake-build-download -G "Unix Makefiles" .
# on Windows, run `cmake -Bcmake-build-download -G "MinGW Makefiles" .`
cmake --build cmake-build-download/ --target download

Copy across some files from an SDK example project (for the nRF52840 DK, replace pca10040/s132 with pca10056/s140):

cmake -E copy toolchains/nRF5/nRF5_SDK_16.0.0_98a08e2/examples/ble_peripheral/ble_app_template/pca10040/s132/armgcc/ble_app_template_gcc_nrf52.ld src/gcc_nrf52.ld
cmake -E copy toolchains/nRF5/nRF5_SDK_16.0.0_98a08e2/examples/ble_peripheral/ble_app_template/pca10040/s132/config/sdk_config.h src/

At this point, you can open the project in CLion or your editor of choice to edit the files.

Add a file src/main.c and add some source code. Here we add some simple code to log a message.

#include "nrf_log_default_backends.h"
#include "nrf_log.h"
#include "nrf_log_ctrl.h"

int main(void) {
    ret_code_t err_code = NRF_LOG_INIT(NULL);
    APP_ERROR_CHECK(err_code);

    NRF_LOG_DEFAULT_BACKENDS_INIT();

    NRF_LOG_INFO("Hello world");
    while(true) {
        // do nothing
    }
}

Create an src/app_config.h file to override some of the default configuration in sdk_config.h:

#define NRF_LOG_BACKEND_RTT_ENABLED 1 // enable rtt
#define NRF_LOG_BACKEND_UART_ENABLED 0 // disable uart
#define NRF_LOG_DEFERRED 0 // flush logs immediately
#define NRF_LOG_ALLOW_OVERFLOW 0 // no overflow
#define SEGGER_RTT_CONFIG_DEFAULT_MODE 2 // block until processed

We are going to include the DFU bootloader too, so we need to generate keys. In terminal/command prompt:

nrfutil keys generate keys/dfu_private.key
nrfutil keys display --key pk --format code keys/dfu_private.key --out_file keys/dfu_public_key.c

Now we need to create a file src/CMakeLists.txt to build our targets:

set(NRF5_LINKER_SCRIPT ${CMAKE_CURRENT_SOURCE_DIR}/gcc_${NRF_FAMILY})

# DFU requirements
# List the softdevice versions previously used, or use FALSE if no previous softdevices
set(PREVIOUS_SOFTDEVICES FALSE)
# Set the location to the DFU private key
set(PRIVATE_KEY ${CMAKE_CURRENT_SOURCE_DIR}/../keys/dfu_private.key)
set(PUBLIC_KEY ${CMAKE_CURRENT_SOURCE_DIR}/../keys/dfu_public_key.c)
# Set the App validation type. [NO_VALIDATION|VALIDATE_GENERATED_CRC|VALIDATE_GENERATED_SHA256|VALIDATE_ECDSA_P256_SHA256]
set(APP_VALIDATION_TYPE NO_VALIDATION)
# Set the Soft Device validation type. [NO_VALIDATION|VALIDATE_GENERATED_CRC|VALIDATE_GENERATED_SHA256|VALIDATE_ECDSA_P256_SHA256]
set(SD_VALIDATION_TYPE NO_VALIDATION)
# The bootloader version (user defined)
set(BOOTLOADER_VERSION 1)
# The DFU version string (firmware version string)
set(DFU_VERSION_STRING "${VERSION_STRING}")

# Set the target name
set(target example)

# add the required libraries for this example
nRF5_addLog()
nRF5_addSeggerRTT()
nRF5_addAppError()

# include files
list(APPEND SOURCE_FILES
        main.c
        )
list(APPEND INCLUDE_DIRS
        "${CMAKE_CURRENT_SOURCE_DIR}"
        )

nRF5_addExecutable(${target} "${SOURCE_FILES}" "${INCLUDE_DIRS}" "${NRF5_LINKER_SCRIPT}")

# make sdk_config.h import app_config.h
target_compile_definitions(${target} PRIVATE USE_APP_CONFIG)

# Here you can set a list of user variables to be defined in the bootloader makefile (which you have modified yourself)
set(bootloader_vars "")

# add the secure bootloader build target
nRF5_addSecureBootloader(${target} "${PUBLIC_KEY}" "${bootloader_vars}")
# add the bootloader merge target
nRF5_addBootloaderMergeTarget(${target} ${DFU_VERSION_STRING} ${PRIVATE_KEY} ${PREVIOUS_SOFTDEVICES} ${APP_VALIDATION_TYPE} ${SD_VALIDATION_TYPE} ${BOOTLOADER_VERSION})
# add the bootloader merged flash target
nRF5_addFlashTarget(bl_merge_${target} "${CMAKE_CURRENT_BINARY_DIR}/${target}_bl_merged.hex")
# Add the Bootloader + SoftDevice + App package target
nRF5_addDFU_BL_SD_APP_PkgTarget(${target} ${DFU_VERSION_STRING} ${PRIVATE_KEY} ${PREVIOUS_SOFTDEVICES} ${APP_VALIDATION_TYPE} ${SD_VALIDATION_TYPE} ${BOOTLOADER_VERSION})
# Add the App package target
nRF5_addDFU_APP_PkgTarget(${target} ${DFU_VERSION_STRING} ${PRIVATE_KEY} ${PREVIOUS_SOFTDEVICES} ${APP_VALIDATION_TYPE})

# print the size of consumed RAM and flash - does not yet work on Windows
if(NOT ${CMAKE_HOST_SYSTEM_NAME} STREQUAL "Windows")
    nRF5_print_size(${target} ${NRF5_LINKER_SCRIPT} TRUE)
endif()

Then we are ready to build and run our example. First, run JLink tools to get the RTT output:

cmake -Bcmake-build-debug -G "Unix Makefiles" -DCMAKE_BUILD_TYPE=Debug .
# On Windows, run `cmake -Bcmake-build-debug -G "MinGW Makefiles" -DCMAKE_BUILD_TYPE=Debug .` 
# If you get an error that the compiler cannot be found, ensure it is present in your PATH (try running `arm-none-eabi-gcc`). Windows users, see the dependencies section.
cmake --build cmake-build-debug/ --target START_JLINK_RTT

Then, build the merge and flash target:

cmake --build cmake-build-debug/ --target flash_bl_merge_example

You should see the "Hello world" log output in the RTT console! From here you can add source code and include SDK libraries with the macros provided in nRF5-cmake-scripts/includes/libraries.cmake.

NRB Tech uses CMake to create well tested products utilising cross-platform code. If you are looking at creating a new product, or want to work on a diverse range of cross platform product development, get in touch.