CVE-2026-54502: Oj Ruby gem stack buffer overflow via large :indent value

The Oj Ruby gem (ohler55/oj) is vulnerable to a stack-based buffer overflow in versions prior to 3.17.2. When Oj.dump is called with a large :indent option (e.g., INT_MAX), the native fill_indent helper in ext/oj/dump.h multiplies the indentation count by out->indent and calls memset(out->cur, ' ', cnt) without validating that the destination buffer can hold the requested bytes. The stack-allocated output buffer is only a few kilobytes, so a 2 GB memset corrupts the stack and crashes the Ruby interpreter with a SIGSEGV. Commit ec368db ("Fix stack limits (#1014)", released as 3.17.2) mitigates the issue by rejecting :indent values greater than 16 at the option-parsing layer.

• Package/component affected: ohler55/oj (Optimized JSON gem for Ruby), specifically the C extension ext/oj/dump.c and the inline fill_indent helper in ext/oj/dump.h.
• Affected versions: Prior to 3.17.2 (vulnerable parent commit 4587e87; fix commit ec368db).
• Risk level and consequences: Medium severity. A developer-controlled :indent value of 2147483647 causes a deterministic native crash (SIGSEGV) due to stack corruption. In processes that expose JSON serialization to untrusted input, this could be used for denial of service or, with further research, potentially memory corruption exploitation.

Impact Parity

• Disclosed/claimed maximum impact: memory corruption (stack buffer overflow) / crash.
• Reproduced impact from this run: Native SIGSEGV crash in Oj.dump on the vulnerable version; the same call is cleanly rejected with an ArgumentError on the fixed version.
• Parity: full — the reproduced crash directly matches the claimed memory-corruption impact.
• Not demonstrated: Full arbitrary code execution was not attempted; only the crash/memory-corruption symptom was proven.

Root Cause

ext/oj/dump.h defines an inline function:

inline static void fill_indent(Out out, int cnt) {
    if (0 < out->indent) {
        cnt *= out->indent;
        *out->cur++ = '\n';
        memset(out->cur, ' ', cnt);
        out->cur += cnt;
    }
}

out->indent is populated from the Ruby :indent option in ext/oj/oj.c (parse_options_cb). In the vulnerable code there is no upper bound on the value, so passing indent: 2147483647 makes cnt equal to INT_MAX and memset attempts to write ~2 GB of spaces into the stack-allocated output buffer, causing a stack overflow and SIGSEGV.

Fix commit ec368db ("Fix stack limits (#1014)") introduces MAX_INDENT 16 and raises rb_raise(rb_eArgError, "indent is limited to %d characters.", MAX_INDENT) when the provided indent exceeds that limit. This validation is performed before the value reaches fill_indent, preventing the overflow.

• Fix commit: ec368dbe936ef0104b782e4b0f67b17d6c7276f7
• Vulnerable commit: 4587e87e23adc9a4163834dc8c9ba9d7206c6501 (parent of fix, matches v3.17.1)

Reproduction Steps

1. Run bundle/repro/reproduction_steps.sh.
2. The script reads bundle/project_cache_context.json and clones the Oj repository from the project cache into bundle/artifacts/oj-vuln and bundle/artifacts/oj-fixed.
3. It checks out the vulnerable commit (4587e87) in one copy, builds the C extension, and runs:

   Oj.dump({a: 1}, indent: 2147483647)
   

   This produces a SIGSEGV (exit code 139) and the Ruby interpreter prints a segmentation-fault backtrace.
4. It checks out the fixed commit (ec368db) in the second copy, builds the C extension, and runs the same Ruby call. The fixed version raises an ArgumentError:

   indent is limited to 16 characters.
   

5. The script compares the two outcomes and writes bundle/repro/runtime_manifest.json and bundle/repro/validation_verdict.json.

Expected evidence of reproduction

• bundle/logs/vulnerable.log: contains [BUG] Segmentation fault at ... and the Ruby/C backtrace.
• bundle/logs/fixed.log: contains ArgumentError: indent is limited to 16 characters.
• bundle/logs/reproduction_steps.log: contains the full build/test output and the final CONFIRMED line.

Evidence

Environment

• Ruby 3.3.8 (x86_64-linux-gnu)
• Oj vulnerable commit 4587e87 (VERSION 3.17.1)
• Oj fixed commit ec368db (VERSION 3.17.2)
• C extension built directly with extconf.rb + make in each checkout

Key excerpts

Vulnerable run (bundle/logs/vulnerable.log):

-e:1: [BUG] Segmentation fault at 0x00007ffc5049e000
ruby 3.3.8 (2025-04-09 revision b200bad6cd) [x86_64-linux-gnu]

-- Control frame information -----------------------------------------------
c:0003 p:---- s:0012 e:000011 CFUNC  :dump
...
-- Machine register context ------------------------------------------------
 ...
 RDX: 0x000000007fffffff
 ...

The RDX register holds 0x7fffffff (INT_MAX), matching the requested indent size.

Fixed run (bundle/logs/fixed.log):

-e:1:in `dump': indent is limited to 16 characters. (ArgumentError)

require 'oj'; puts Oj::VERSION; Oj.dump({a: 1}, indent: 2147483647); puts 'no crash'
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
	from -e:1:in `<main>'
3.17.2

Driver log (bundle/logs/reproduction_steps.log):

VULN_RESULT=0
FIXED_RESULT=1
CONFIRMED: vulnerable version crashes with SIGSEGV, fixed version does not.

Recommendations / Next Steps

• Suggested fix: Apply the upstream patch from ec368db and enforce a maximum :indent value (currently 16) at the option-parsing layer, before any native buffer operation. Any location that accepts user-provided indentation settings should validate the value.
• Upgrade guidance: Upgrade to Oj 3.17.2 or later. The vulnerable behavior is fixed by the upstream validation.
• Testing recommendations: Add regression tests that call Oj.dump with indent: 2147483647 and expect an ArgumentError. Also test with a variety of nested objects/arrays and negative/edge-case indent values to ensure no other path reaches fill_indent with an unbounded size.

Additional Notes

• Idempotency: The script was executed twice successfully from a clean state and from a state where the artifact clones already existed. Both runs produced the same SIGSEGV on the vulnerable build and ArgumentError on the fixed build, then exited with code 0 and wrote the required runtime manifest and verdict.
• Edge cases / limitations: The reproduction uses the exact Ruby API call named in the ticket (Oj.dump(..., indent: INT_MAX)). The crash is a native SIGSEGV, not a sanitizer report; no ASAN/UBSAN build was used, so the primary oracle is the process exit status and the Ruby interpreter's segmentation-fault backtrace.