Segmentation Fault on Save for Files Larger Than 64KB Containing UTF-8 Multibyte Characters (v2.3.1 Regression)

Summary

The application crashes with a segmentation fault when saving a document larger than 64 KB that contains UTF-8 multibyte characters such as emoji or CJK code points. The fault does not occur for files under 64 KB or for files over 64 KB composed entirely of ASCII. The regression was introduced in the v2.3.0 → v2.3.1 upgrade, strongly suggesting a buffer-sizing defect added or modified in that release.

Root-Cause Analysis

Confirmed Evidence

• Segmentation fault occurs only when both conditions are true simultaneously: file size > 64 KB and content contains UTF-8 multibyte characters.
• Files > 64 KB with ASCII-only content save successfully, ruling out a generic large-file I/O path failure.
• Files < 64 KB with multibyte characters save successfully, ruling out a generic UTF-8 decoding failure.
• The fault is a regression tied precisely to the v2.3.1 release.

Root-Cause Inference (High Confidence)

The intersection of the 64 KB threshold and multibyte character encoding is the classic fingerprint of a buffer allocated by character count but written by byte count (or vice versa), causing a heap or stack buffer overflow that corrupts memory and produces a segmentation fault.

The most likely scenario:

1. A write buffer is allocated using strlen(), wcslen(), or a similar call that returns a character count rather than the byte length of the UTF-8 encoded string. For ASCII, character count == byte count, so no overflow occurs. For multibyte UTF-8 (emoji = 4 bytes per code point; CJK = 3 bytes per code point), the allocated buffer is shorter than the data written into it.
2. The 64 KB boundary suggests a fixed or threshold-gated code path introduced or altered in v2.3.1 — for example, a chunked write loop, a compression pre-buffer, or an intermediate encoding conversion buffer sized as 64 * 1024 characters rather than bytes.
3. The overflow corrupts adjacent heap memory, causing the segmentation fault at the point of the save operation.

Alternative Causes (Lower Confidence)

• An off-by-one error in a boundary check (>= 65536 changed to > 65536, or vice versa) combined with a separate pre-existing multibyte handling issue that was previously never reached.
• A realloc path that fails silently and returns a stale or null pointer when the buffer must grow beyond 64 KB for a multibyte payload, with no null-pointer guard before the subsequent write.
• A locale or codec initialization step that is skipped for the "small file" path but required for multibyte correctness on the "large file" path.

What Would Confirm the Diagnosis

• A stack trace from the core dump (gdb ./application core) identifying the exact function and line of the fault.
• A git diff v2.3.0..v2.3.1 on the file-save, buffer-allocation, or encoding-conversion code paths.
• Reproduction under AddressSanitizer (ASAN), which would report a heap-buffer-overflow with precise size details.

Resolution Steps

1. Capture the core dump stack trace to identify the faulting code path before attempting any code fix.
2. Review the v2.3.0 → v2.3.1 diff for any changes to file-write routines, buffer allocation, or UTF-8/Unicode encoding logic.
3. Locate all buffer allocation sites in the save code path that use character count (strlen, wcslen, code-unit count) as the size argument and verify each accounts for the maximum UTF-8 byte width (up to 4 bytes per code point). Replace with strlen on the UTF-8 encoded byte string, or compute character_count * 4 as the upper-bound allocation size, then trim after encoding.
4. Replace fixed-size 64 KB intermediate buffers with dynamically computed sizes based on the encoded byte length of the content, not the logical character count.
5. Add null-pointer and bounds checks after every malloc/realloc call in the save path before any write operation.
6. Build and test with AddressSanitizer enabled to verify the overflow is eliminated.
7. Release a patch version (v2.3.2) with the fix and a clear changelog entry referencing this regression.

CLI Commands

Inspect the core dump to get the stack trace:

gdb /usr/bin/<APPLICATION_BINARY> core

Inside GDB:

(gdb) bt full
(gdb) info locals
(gdb) frame <FRAME_NUMBER>

Build with AddressSanitizer to reproduce and confirm the overflow:

# Adjust build system flags as appropriate for the project
CFLAGS="-fsanitize=address -fno-omit-frame-pointer -g" \
LDFLAGS="-fsanitize=address" \
make clean && make

Run the ASAN-instrumented binary and reproduce the save with the affected file:

ASAN_OPTIONS=abort_on_error=1:log_path=/tmp/asan.log \
  ./<APPLICATION_BINARY> <TEST_FILE_70KB_WITH_EMOJI>
cat /tmp/asan.log.*

Review the diff between v2.3.0 and v2.3.1 for buffer-related changes:

git diff v2.3.0 v2.3.1 -- <SOURCE_FILE_OR_DIRECTORY> | grep -E '(alloc|malloc|realloc|strlen|wcslen|buffer|buf_size|64)'

Generate a reproducible 70 KB test file containing emoji:

python3 -c "print('😀' * 18000)" > /tmp/test_emoji_70k.txt
wc -c /tmp/test_emoji_70k.txt   # should be ~72000 bytes

Verification

After applying the fix:

1. Confirm the test file saves without a crash:

# Manual smoke test
<APPLICATION_BINARY> --save /tmp/test_emoji_70k.txt
echo "Exit code: $?"   # Expected: 0

1. Verify file integrity after save:

python3 -c "
with open('/tmp/test_emoji_70k.txt', 'r', encoding='utf-8') as f:
    content = f.read()
print(f'Characters: {len(content)}, valid UTF-8: OK')
"

1. Run the ASAN build against the same file and confirm no heap-buffer-overflow or stack-buffer-overflow reports appear in ASAN output.

1. Run the project's existing test suite and confirm no regressions:

make test

1. Rollback indicator: If the crash persists after the fix attempt, revert to v2.3.0 as a temporary mitigation while root cause analysis continues:

sudo apt install <APPLICATION_PACKAGE>=2.3.0

Prevention

• Unit tests for encoding boundary conditions: Add automated tests that save files at exactly 64 KB, 64 KB + 1 byte, and 128 KB, each containing multibyte UTF-8 sequences. These tests must pass before any release.
• Fuzz testing on the save path: Integrate a fuzzer (e.g., libFuzzer or AFL++) targeting the file-write and encoding functions with randomized multibyte content and varying sizes.
• Mandatory ASAN/UBSAN CI gate: Require AddressSanitizer and UndefinedBehaviorSanitizer builds to pass in CI for every pull request that touches I/O, buffer management, or encoding code.
• Static analysis: Enforce a static analysis step (e.g., clang-tidy, cppcheck, or Coverity) in CI to flag unsafe buffer-sizing patterns such as using character count as a byte-length argument.
• Code review checklist: Add an explicit checklist item for PRs modifying file I/O: "Buffer sizes are computed from encoded byte length, not character count."
• Regression test pinning: Lock the v2.3.1 failure scenario as a permanent regression test case so that this exact class of bug cannot ship again without detection.