libretro-db: fix OOM bugs in rmsgpack_dom reader state and query parser

Five related fixes across libretro-db's runtime code (the subset
linked into RetroArch itself - c_converter.c and similar offline
tools are out of scope for this patch).

=== rmsgpack_dom.c: rmsgpack_dom_reader_state_new unchecked callocs ===

    struct rmsgpack_dom_reader_state *s = calloc(1, ...);
    s->i        = 0;          /* NULL-deref on OOM */
    s->capacity = 1024;
    s->growable = true;
    s->stack    = calloc(1024, ...);
    return s;

Both callocs unchecked.  The outer calloc's failure NULL-derefs
on the immediate field writes.  The inner (stack) calloc's
failure is caught later when callers iterate - rmsgpack_dom_read
at line ~422 does 's->stack[0] = out' unconditionally.

Fix: NULL-check the outer calloc and return NULL.  NULL-check
the inner calloc and free the outer struct + return NULL on
failure.  Callers must now tolerate NULL return.

=== rmsgpack_dom.c: rmsgpack_dom_reader_state_free NULL-deref ===

    free(state->stack);     /* NULL-deref if state is NULL */
    free(state);

state could not be NULL pre-patch (construction couldn't fail)
but with reader_state_new now able to return NULL, callers that
defer cleanup (like input/bsv/bsvmovie.c's decompress path)
would hit this free with NULL.

Fix: early return on NULL state.

=== rmsgpack_dom.c: rmsgpack_dom_reader_state_push realloc-assign-self ===

    s->capacity *= 2;
    s->stack     = realloc(s->stack, s->capacity * ...);
    if (!s->stack)
       return -1;

Two stacked bugs:
  1. Classic realloc-assign-self: on OOM s->stack goes NULL,
     leaking the old buffer (the only reference).
  2. 'capacity *= 2' happens BEFORE the realloc, so on failure
     the capacity claimed the new size while the allocation is
     still the old size (or NULL).  The -1 return is checked by
     the direct caller, but the out-of-sync state is a heap-
     buffer-overflow trap if anything later ignored the error
     return.

Fix: realloc-to-tmp.  Compute new_capacity into a local, realloc
into new_stack, commit both only on success.  Preserves the -1
OOM signal semantics.

=== query.c: query_parse_method_call zero-arg false-OOM ===

    invocation->argv = (argi > 0) ? malloc(...) : NULL;

    if (!invocation->argv)
    {
       /* emit 'OOM' error */
       goto clean;
    }

argv is legitimately NULL when argi == 0 (a zero-argument
function call like 'foo()') but the next block treated that as
OOM and raised the 'OOM' error string.

This was a pre-existing logic bug tangentially exposed by OOM
auditing.  The observable symptom pre-patch was that any
genuinely-zero-argument function call would fail query parsing
with a bogus 'OOM' error.

Fix: gate the OOM branch on 'argi > 0 && !invocation->argv'.
Skip the memcpy when argv is NULL (nothing to copy when argi==0).

=== input/bsv/bsvmovie.c: bsv_movie_read_deduped_state consumer ===

    struct rmsgpack_dom_reader_state *reader_state =
       rmsgpack_dom_reader_state_new();
    ...
    rmsgpack_dom_read_with(read_mem, &item, reader_state);

With rmsgpack_dom_reader_state_new now able to return NULL, this
caller needs to NULL-check before passing reader_state into
rmsgpack_dom_read_with (which dereferences it freely).

Fix: early bail to the existing 'exit' cleanup label on NULL.
The cleanup already calls rmsgpack_dom_reader_state_free (now
NULL-tolerant), intfstream_close (already NULL-tolerant), and
sets ret=false, so the user-visible outcome on OOM is the same
as the existing 'malformed frame' error paths.

=== Scope notes ===

Other libretro-db files checked during this pass:
  * bintree.c - the single alloc site at line 35 is already
    NULL-checked.
  * libretrodb.c - two sites at lines 771 (buff in create_index)
    and 831 (cursor_new) both use the NULL-check idiom.
  * rmsgpack.c - zero raw alloc sites.
  * Inner callocs in rmsgpack_dom.c:145 (dom_read_map_start) and
    :173 (dom_read_array_start) use the 'if (!(items = ...))'
    idiom and propagate -1 up the rmsgpack callback chain.

Not in scope (standalone offline utilities, not linked into
RetroArch runtime - see libretro-db/Makefile TARGETS):
  * c_converter.c - 7 alloc sites
  * libretrodb_tool.c
  * rmsgpack_test.c

=== Thread-safety ===

All touched paths run synchronously on whichever thread invoked
the db query.  In practice RetroArch accesses libretro-db from
the main thread (menu content scanning, explore-by-metadata)
and from task threads (background scan).  No shared-state
changes; no lock discipline changes.

=== Reachability ===

  * rmsgpack_dom_reader_state_new/free: the allocating variant
    is only called from input/bsv/bsvmovie.c's STATESTREAM
    decode path, used during savestate runahead on cores that
    support bsv movie recording.
  * rmsgpack_dom_reader_state_push: called recursively while
    parsing deep msgpack maps/arrays inside a libretrodb .rdb
    query result.  Capacity doublings trigger on deeply nested
    data (the default capacity is 1024 for the allocating
    variant, MAX_DEPTH stack-alloca for rmsgpack_dom_read - the
    stack variant sets growable=false so never reaches this
    realloc path).
  * query_parse_method_call: every libretrodb query string
    parse - menu-driven 'Explore' category navigation and any
    core's database-backed metadata lookup.

Author: LibretroAdmin

input/bsv/bsvmovie.c

@@ -1658,6 +1658,17 @@ bool bsv_movie_read_deduped_state(bsv_movie_t *movie, uint8_t *encoded, size_t e
    size_t superblock_byte_size = movie->superblocks->object_size*block_byte_size;
    intfstream_t *read_mem      = intfstream_open_memory(encoded,
          RETRO_VFS_FILE_ACCESS_READ, RETRO_VFS_FILE_ACCESS_HINT_NONE, encoded_size);
+   /* NULL-check reader_state: rmsgpack_dom_reader_state_new can
+    * now return NULL on OOM.  Bailing to 'exit' hits the
+    * rmsgpack_dom_reader_state_free call at the cleanup label
+    * (which now tolerates NULL) and returns false; the user-
+    * visible outcome is 'STATESTREAM decode failed' which
+    * matches the existing error paths for a malformed frame. */
+   if (!reader_state)
+   {
+      RARCH_ERR("[STATESTREAM] failed to allocate reader state\n");
+      goto exit;
+   }
    if (state_size > movie->last_save_size && movie->superblock_seq)
    {
       free(movie->superblock_seq);

libretro-db/query.c

@@ -943,7 +943,11 @@ static struct buffer query_parse_method_call(
    invocation->argv = (argi > 0) ? (struct argument*)
       malloc(sizeof(struct argument) * argi) : NULL;
 
-   if (!invocation->argv)
+   /* Gate the OOM branch on 'argi > 0 && !argv' - before this
+    * change a valid zero-arg function call ('foo()') was being
+    * erroneously treated as OOM because argv is legitimately
+    * NULL when argi==0. */
+   if (argi > 0 && !invocation->argv)
    {
       s[0] = 'O';
       s[1] = 'O';
@@ -952,8 +956,9 @@ static struct buffer query_parse_method_call(
       *err = s;
       goto clean;
    }
-   memcpy(invocation->argv, args,
-         sizeof(struct argument) * argi);
+   if (invocation->argv)
+      memcpy(invocation->argv, args,
+            sizeof(struct argument) * argi);
 
    return buff;
 

libretro-db/rmsgpack_dom.c

@@ -54,14 +54,28 @@ static int rmsgpack_dom_reader_state_push(
 {
    if ((s->i + 1) == s->capacity)
    {
+      size_t new_capacity;
+      struct rmsgpack_dom_value **new_stack;
+
       if (!s->growable)
          return -1;
 
-      s->capacity *= 2;
-      s->stack     = (struct rmsgpack_dom_value **)
-         realloc(s->stack, s->capacity * sizeof(struct rmsgpack_dom_value *));
-      if (!s->stack)
+      /* realloc-to-tmp: pre-patch was 's->stack = realloc(s->stack,
+       * ...)' which on OOM would leak the old buffer (self-assign
+       * drops the only reference).  Also pre-patch did
+       * 'capacity *= 2' before the realloc, so on failure the
+       * capacity claimed the new size while the allocation was
+       * still the old size - out-of-sync state that would manifest
+       * as a heap-buffer-overflow later if anything else ignored
+       * the -1 return.  Compute new_capacity into a local, realloc
+       * into a tmp, commit both only on success. */
+      new_capacity = s->capacity * 2;
+      new_stack    = (struct rmsgpack_dom_value **)
+         realloc(s->stack, new_capacity * sizeof(struct rmsgpack_dom_value *));
+      if (!new_stack)
          return -1;
+      s->stack    = new_stack;
+      s->capacity = new_capacity;
    }
    s->i++;
    s->stack[s->i] = v;
@@ -429,15 +443,35 @@ struct rmsgpack_dom_reader_state *rmsgpack_dom_reader_state_new(void)
 {
    struct rmsgpack_dom_reader_state *s = (struct rmsgpack_dom_reader_state *)calloc(1,
          sizeof(struct rmsgpack_dom_reader_state));
+   /* NULL-check: the field writes below NULL-deref on OOM. */
+   if (!s)
+      return NULL;
    s->i        = 0;
    s->capacity = 1024;
    s->growable = true;
    s->stack    = (struct rmsgpack_dom_value **)calloc(1024, sizeof(struct rmsgpack_dom_value *));
+   /* NULL-check the stack calloc: consumers (rmsgpack_dom_read
+    * above at line ~422 writes s->stack[0] unconditionally) would
+    * NULL-deref on OOM.  Free the containing state struct and
+    * return NULL so the caller (rmsgpack_dom_read does this via
+    * its own caller chain) can fall through gracefully. */
+   if (!s->stack)
+   {
+      free(s);
+      return NULL;
+   }
    return s;
 }
 
 void rmsgpack_dom_reader_state_free(struct rmsgpack_dom_reader_state *state)
 {
+   /* NULL-check: rmsgpack_dom_reader_state_new can now return
+    * NULL on OOM (this commit).  Callers that defer cleanup to
+    * the end of a function (e.g. input/bsv/bsvmovie.c's
+    * decompress path at line ~1833) may hit this with a NULL
+    * state pointer if the reader_state_new call failed. */
+   if (!state)
+      return;
    free(state->stack);
    free(state);
 }