How to disable cache support from arm64 Linux

[kuldeep_snps]

================================================================================
ARM64 Linux Cache-Disabled Environment: Busybox Crash Investigation
================================================================================

PROBLEM SUMMARY
================================================================================
Running ARM64 Linux (kernel 6.6.63) with both I-cache and D-cache disabled via
Kconfig causes busybox (and other complex userspace programs) to crash with
SIGSEGV, even when compiled with alignment-safe flags. Simple C programs work
fine. Seeking community input on root cause and potential solutions.

ENVIRONMENT
================================================================================
Platform: ARM Fixed Virtual Platform (FVP Base)
Architecture: ARMv8-A (aarch64)
Kernel: Linux 6.6.63
Toolchain: GCC 11.2.1 (aarch64-none-linux-gnu)
Busybox: 1.36.1 (statically linked with glibc)
Boot: initramfs (cpio.gz)

KERNEL CONFIGURATION
================================================================================
The following cache-disable patches were applied to kernel 6.6.63:

1. Kconfig changes (arch/arm64/Kconfig):
- Added CONFIG_CPU_DCACHE_DISABLE option
- Added CONFIG_CPU_ICACHE_DISABLE option

2. Cache disable implementation (arch/arm64/mm/proc.S):
In __cpu_setup function, modified SCTLR_EL1 initialization:

#ifdef CONFIG_CPU_DCACHE_DISABLE
bic x0, x0, #SCTLR_ELx_C // Clear C bit (disable D-cache)
#endif
#ifdef CONFIG_CPU_ICACHE_DISABLE
bic x0, x0, #SCTLR_ELx_I // Clear I bit (disable I-cache)
#endif

3. Verification code added to print SCTLR_EL1 status at boot

VERIFICATION OF CACHE DISABLE
================================================================================
Boot log confirms successful cache disable:

======================================
CPU0 SCTLR_EL1 Cache Status Report
======================================
SCTLR_EL1 = 0x0200002034f4c999
--------------------------------------
Bit 0 (M-MMU): ENABLED
Bit 1 (A-Align): DISABLED <-- Note: Alignment checking OFF
Bit 2 (C-DCache): DISABLED <-- D-Cache successfully disabled
Bit 3 (SA): ENABLED
Bit 12 (I-ICache): DISABLED <-- I-Cache successfully disabled
--------------------------------------

OBSERVED BEHAVIOR
================================================================================

WHAT WORKS:
-----------
Simple C programs compiled with safe flags work perfectly:

Example: init_simple.c (300+ lines)

• Performs sequential memory access tests (256KB array)
• Performs random memory access tests (2MB array)
• Performs pointer chasing tests (1MB linked list)
• Uses glibc functions: open(), write(), gettimeofday(), sleep()
• Runs without any issues

Compilation flags:
aarch64-none-linux-gnu-gcc -static -O0 -fno-strict-aliasing -march=armv8-a \
-o init init_simple.c

WHAT FAILS:
-----------
Busybox and complex programs crash with:

Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000007
Call trace:
el0_da+0x48/0x54 <-- Data Abort at EL0 (userspace)
el0t_64_sync_handler+0xdc/0x150
el0t_64_sync+0x14c/0x150

Error code 0x00000007 = SIGSEGV (segmentation fault)
el0_da = Data Abort at Exception Level 0 (userspace)

EXPERIMENTS CONDUCTED
================================================================================

Experiment 1: Busybox with -O2 (default)
-----------------------------------------
Result: CRASH at shell initialization
Command: /bin/sh (symlink to busybox)
Status: Crashes before shell prompt appears

Experiment 2: Busybox rebuilt with -O0
---------------------------------------
Build command:
make ARCH=arm64 \
CROSS_COMPILE=aarch64-none-linux-gnu- \
EXTRA_CFLAGS="-O0" \
-j8

Result: CRASH - Same error (exitcode=0x00000007, el0_da)

Experiment 3: Busybox with -O0 -mstrict-align
----------------------------------------------
Build command:
make ARCH=arm64 \
CROSS_COMPILE=aarch64-none-linux-gnu- \
EXTRA_CFLAGS="-O0 -mstrict-align -march=armv8-a" \
-j8

Result: CRASH - Same error (exitcode=0x00000007, el0_da)

Experiment 4: Busybox with -O0 -fno-strict-aliasing -mstrict-align
-------------------------------------------------------------------
Build command:
make ARCH=arm64 \
CROSS_COMPILE=aarch64-none-linux-gnu- \
EXTRA_CFLAGS="-O0 -fno-strict-aliasing -mstrict-align -march=armv8-a" \
-j8

Result: CRASH - Same error (exitcode=0x00000007, el0_da)
Note: These are the EXACT same flags that work for init_simple.c

Experiment 5: Minimal shell (custom C program)
-----------------------------------------------
Created minimal shell (minimal_shell.c) with:

• Built-in commands: cd, pwd, echo, help, exit
• Ability to execute external programs via execve()
• Compiled with same safe flags as init_simple.c

Result: PARTIAL SUCCESS

• Shell starts and runs
• Basic commands work (cd, pwd, echo, help)
• Can execute external programs successfully
• BUT: 'ls' command crashes with same el0_da error

(ls uses opendir/readdir from glibc)

DETAILED ANALYSIS
================================================================================

Working vs Failing Comparison:
-------------------------------
WORKING: init_simple.c (3.0M binary)
- Uses: open(), close(), read(), write(), dup2(), fork(), wait(),
gettimeofday(), sleep()

• Static arrays, simple structures
• No dynamic memory allocation
• No directory operations

FAILING: busybox (2.6M binary)

• Complex command multiplexer (277 applets)
• Statically linked with full glibc
• Uses directory operations (opendir/readdir)
• More complex memory access patterns

FAILING: minimal_shell ls command

• Specifically crashes in opendir() or readdir()
• Other shell functions work fine

Key Observations:
-----------------
1. SCTLR_EL1.A bit is DISABLED (alignment checking off at hardware level)
2. Crash is el0_da (Data Abort at userspace level)
3. Simple programs work, complex programs fail
4. Same compilation flags produce different results for different programs
5. Specific glibc functions (opendir/readdir) cause crashes
6. Busybox is statically linked, so recompiling with safe flags should
have worked (but didn't)

ROOTFS ANALYSIS
================================================================================
Examined initramfs contents:

• Busybox binary: /bin/busybox (2.6M, statically linked)
• 277 symlinks in /bin/, /sbin/, /usr/bin/ pointing to busybox
• One additional binary: /init (debug init, also static)
• No shared libraries (everything is static)

Verified with:
file busybox
Output: ELF 64-bit LSB executable, ARM aarch64, version 1 (GNU/Linux),
statically linked, stripped

QUESTIONS FOR COMMUNITY
================================================================================

1. Why do simple C programs work but busybox crashes, even when both are
compiled with identical alignment-safe flags?

2. Is there something specific about opendir/readdir in glibc that requires
caches to be enabled, or performs unaligned memory accesses that can't
be fixed with compiler flags?

3. Could this be related to:
- ELF binary layout/alignment?
- Statically-linked glibc having precompiled assembly with unaligned access?
- Specific ARM64 instructions that require caches?
- TLB behavior without caches?

4. Is there a way to:
- Identify the exact instruction causing the data abort?
- Rebuild glibc with strict alignment guarantees?
- Use an alternative libc (musl) that might work better?

5. Should SCTLR_EL1.A (alignment check enable) be set to trap unaligned
accesses explicitly, rather than relying on hardware to reject them?

6. Has anyone successfully run complex userspace (busybox/bash/coreutils)
on ARM64 with caches disabled?

HYPOTHESIS
================================================================================
Current hypothesis: Without caches, ARM64 memory controllers or CPU cannot
perform unaligned memory accesses that would normally be handled by cache
subsystem. Even with -mstrict-align, the statically-linked glibc contains
precompiled code (possibly assembly) with unaligned accesses that cannot be
fixed by recompiling busybox alone.

Alternative hypothesis: Specific glibc functions (directory operations) use
data structures or algorithms that inherently create unaligned accesses,
regardless of compiler flags.

POTENTIAL WORKAROUNDS TRIED
================================================================================
1. ✗ Recompile busybox with various alignment flags - FAILED
2. ✗ Use -mstrict-align compiler flag - FAILED
3. ✓ Create minimal shell avoiding problematic glibc functions - PARTIAL SUCCESS
4. Not tried: Rebuild entire glibc with alignment-safe flags
5. Not tried: Use musl libc instead of glibc
6. Not tried: Enable SCTLR_EL1.A bit to trap unaligned accesses explicitly

SAMPLE CODE THAT WORKS
================================================================================
File: init_simple.c
Compilation: aarch64-none-linux-gnu-gcc -static -O0 -fno-strict-aliasing \
-march=armv8-a -o init init_simple.c
Size: 3.0M
Key functions used: open, write, gettimeofday, sleep, fork, wait
Result: Works perfectly in cache-disabled environment

SAMPLE CODE THAT FAILS
================================================================================
File: busybox-1.36.1/busybox
Compilation: make ARCH=arm64 CROSS_COMPILE=aarch64-none-linux-gnu- \
EXTRA_CFLAGS="-O0 -fno-strict-aliasing -mstrict-align -march=armv8-a"
Size: 2.6M (statically linked)
Result: Crashes with SIGSEGV (exitcode=0x00000007) when executing /bin/sh

Also fails:
File: minimal_shell.c (ls command using opendir/readdir)
Same compilation flags as init_simple.c
Result: Shell works, but 'ls' crashes with el0_da

ADDITIONAL INFORMATION
================================================================================

• No kernel crashes observed, only userspace program crashes
• Kernel itself runs fine with caches disabled
• Memory-intensive operations in working programs execute correctly
• Issue appears specifically tied to certain glibc functions
• Problem is reproducible 100% of the time

REQUEST FOR HELP
================================================================================
Looking for:
1. Insights into why this specific behavior occurs
2. Proper way to build userspace for cache-disabled ARM64 environments
3. Tools/techniques to debug the exact unaligned access causing the crash
4. Experience from others running cache-disabled ARM64 systems
5. Whether this is a known limitation of ARM64 architecture
6. Best practices for cache-disabled embedded ARM64 development

RELEVANT FILES
================================================================================

• Kernel patches: Applied to arch/arm64/Kconfig and arch/arm64/mm/proc.S
• Working code: init_simple.c (available on request)
• Failing code: busybox 1.36.1 source with custom EXTRA_CFLAGS
• Minimal shell: minimal_shell.c (partially working)

Thank you for any insights or suggestions!

 

[f33dm3bits]

Hello and welcome! I removed the "Howto" prefix does are for "Howto"guides written by people not for help requests.