perf: lazy range arrays for O(1) std.range creation (#771)

## Motivation

`std.range(from, to)` eagerly allocates an `Array[Eval]` of `Val.Num`
objects. For large ranges like `std.range(1, 1000000)`, this creates 1M
objects upfront even when only a subset of elements are ever accessed.
This is particularly wasteful in patterns like array comparison where
two large ranges share a common prefix and only the tail elements
differ.

## Key Design Decision

Encode lazy range state directly in `Val.Arr` using a boolean `_isRange`
flag and `_rangeFrom` integer field. When `isRange` is true, `arr` is
null and elements are computed on demand via `Val.cachedNum(pos,
_rangeFrom + i)`. This avoids introducing a new subclass and keeps the
hot-path `value(i)` dispatch to a single boolean check.

A separate `_isRange` boolean flag is used instead of a sentinel value
(e.g., `Int.MinValue`) to avoid collisions with valid range start
values.

Reversed ranges store the high end as `_rangeFrom` and count down
(`_rangeFrom - i`), with correct double-reverse handling that restores
the original forward range.


## Modification

- **`Val.scala`**: Add `_isRange` boolean + `_rangeFrom` field to
`Val.Arr`. Update `value(i)`, `eval(i)`, `asLazyArray`, `reversed()` to
handle range state. Add `materializeRange()` for bulk access. Add
`Arr.range()` factory.
- **`ArrayModule.scala`**: Replace eager `Array[Eval]` allocation in
`std.range` with `Val.Arr.range(pos, from, size)` for O(1) creation.
- **New test**: `lazy_range_correctness.jsonnet` — covers iteration,
concat+comparison, slicing, sort, reverse, double-reverse, member, map,
foldl, empty ranges, large ranges.

## Benchmark Results

### JMH (JVM, Scala 3.3.7)

| Benchmark | Before (ms/op) | After (ms/op) | Change |
|-----------|----------------|---------------|--------|
| comparison | 16.204 | 0.028 | **579x faster** 🔥 |
| reverse | 7.033 | 6.823 | 3% faster ✅ |
| bench.06 | 0.226 | 0.218 | 4% faster ✅ |
| setUnion | 0.638 | 0.623 | No regression ✅ |
| gen_big_object | 1.122 | 0.973 | No regression ✅ |
| All others | — | — | No regression ✅ |

### Hyperfine (Scala Native vs jrsonnet)

| Benchmark | sjsonnet (ms) | jrsonnet (ms) | Ratio |
|-----------|--------------|---------------|-------|
| comparison | 6.5 ± 1.4 | 13.4 ± 1.9 | **sjsonnet 2.07x faster** ✅ |
| reverse | — | — | ~1.04x faster (no regression) |
| bench.06 | ~6.8 | ~5.3 | jrsonnet 1.29x (startup-dominated, unchanged)
|

**Previous**: comparison was 1.36x slower than jrsonnet → **Now 2.07x
faster** 🎉

## Analysis

The `comparison` benchmark (`std.range(1, 1000000) + [1] < std.range(1,
1000000) + [2]`) previously allocated 2M `Val.Num` objects upfront, then
compared element by element. With lazy ranges:
1. **O(1) range creation** — only stores start + length, no allocation
2. **O(1) concat** — `ConcatView` wraps range + singleton without
copying
3. **O(1) comparison** — `sharedConcatPrefixLength` detects shared range
prefix (same object reference), skips to the differing tail element

This turns an O(n) operation into O(1) end-to-end.

No regressions detected across all 35+ JMH benchmarks and full test
suite (420 tests × 3 Scala versions × 4 platforms).

## References
The same optimization lives in jrsonnet and scala to too.

## Result

All tests pass (`./mill __.test`). Comparison benchmark flipped from
1.36x slower to 2.07x faster than jrsonnet.

Author: He-Pin

sjsonnet/src/sjsonnet/Val.scala

@@ -342,7 +342,18 @@ object Val {
     private var _concatRight: Arr = _
     private var _length: Int = -1
 
+    // Lazy range state. When _isRange is true, this array represents
+    // a contiguous integer sequence [from, from+1, ..., from+length-1].
+    // Elements are computed on demand via Val.cachedNum, avoiding upfront allocation
+    // of the full backing array. Inspired by jrsonnet's RangeArray (arr/spec.rs)
+    // which uses O(1) creation for std.range results.
+    // Uses a separate boolean flag instead of a sentinel value to avoid collisions
+    // with valid range start values (e.g. Int.MinValue).
+    private var _isRange: Boolean = false
+    private var _rangeFrom: Int = 0
+
     @inline private def isConcatView: Boolean = _concatLeft ne null
+    @inline private def isRange: Boolean = _isRange
 
     override def asArr: Arr = this
 
@@ -352,7 +363,7 @@ object Val {
       else {
         val computed =
           if (isConcatView) _concatLeft.length + _concatRight.length
-          else arr.length
+          else arr.length // isRange always has _length pre-set, never reaches here
         _length = computed
         computed
       }
@@ -362,6 +373,10 @@ object Val {
       if (isConcatView) {
         val leftLen = _concatLeft.length
         if (i < leftLen) _concatLeft.value(i) else _concatRight.value(i - leftLen)
+      } else if (isRange) {
+        // For reversed ranges, _rangeFrom is the last element and we count down
+        if (_reversed) Val.cachedNum(pos, _rangeFrom - i)
+        else Val.cachedNum(pos, _rangeFrom + i)
       } else if (_reversed) {
         arr(arr.length - 1 - i).value
       } else {
@@ -378,6 +393,9 @@ object Val {
       if (isConcatView) {
         val leftLen = _concatLeft.length
         if (i < leftLen) _concatLeft.eval(i) else _concatRight.eval(i - leftLen)
+      } else if (isRange) {
+        if (_reversed) Val.cachedNum(pos, _rangeFrom - i)
+        else Val.cachedNum(pos, _rangeFrom + i)
       } else if (_reversed) {
         arr(arr.length - 1 - i)
       } else {
@@ -403,6 +421,7 @@ object Val {
      */
     def asLazyArray: Array[Eval] = {
       if (isConcatView) materialize()
+      if (isRange) materializeRange()
       if (_reversed) {
         val len = arr.length
         val result = new Array[Eval](len)
@@ -447,6 +466,26 @@ object Val {
       _concatRight = null
     }
 
+    /**
+     * Materialize a lazy range view into a flat array. After this call, `arr` holds the full
+     * Val.Num elements and the range flag is cleared. Handles both forward and reversed ranges.
+     */
+    private def materializeRange(): Unit = {
+      val len = _length
+      val from = _rangeFrom
+      val rev = _reversed
+      val p = pos
+      val result = new Array[Eval](len)
+      var i = 0
+      while (i < len) {
+        result(i) = Val.cachedNum(p, if (rev) from - i else from + i)
+        i += 1
+      }
+      arr = result
+      _isRange = false // clear range flag
+      _reversed = false // range is now materialized in correct order
+    }
+
     /**
      * Concatenate two arrays. For large left-side arrays where neither operand is already a concat
      * view, creates a lazy ConcatView that defers the copy until bulk access is needed. This is
@@ -513,16 +552,56 @@ object Val {
      * Double-reversal cancels out.
      */
     def reversed(newPos: Position): Arr = {
-      if (isConcatView) materialize() // flatten before reverse
-      val result = Arr(newPos, arr)
-      result._reversed = !this._reversed
-      result
+      if (isRange) {
+        // Double-reverse of a range cancels out: return a forward range with original start
+        if (_reversed) {
+          // Currently reversed: _rangeFrom is the high end, counting down.
+          // Reversing again restores the original forward range.
+          val originalFrom = _rangeFrom - _length + 1
+          val result = new Arr(newPos, null)
+          result._isRange = true
+          result._rangeFrom = originalFrom
+          result._length = _length
+          // _reversed defaults to false — forward range
+          result
+        } else {
+          // Forward range: reverse to [from+len-1, from+len-2, ..., from]
+          val len = _length
+          val newFrom = _rangeFrom + len - 1
+          val result = new Arr(newPos, null)
+          result._isRange = true
+          result._rangeFrom = newFrom
+          result._length = len
+          result._reversed = true // signal to compute from-i instead of from+i
+          result
+        }
+      } else {
+        if (isConcatView) materialize() // flatten before reverse
+        val result = Arr(newPos, arr)
+        result._reversed = !this._reversed
+        result
+      }
     }
   }
 
   object Arr {
     def apply(pos: Position, arr: Array[? <: Eval]): Arr = new Arr(pos, arr)
 
+    /**
+     * Create a lazy range array representing the integer sequence [from, from+1, ..., from+size-1].
+     * Elements are computed on demand via Val.cachedNum, avoiding upfront allocation of the full
+     * backing array. This turns `std.range(1, 1000000)` from O(n) to O(1) creation.
+     *
+     * Inspired by jrsonnet's RangeArray (arr/spec.rs) which uses the same deferred approach.
+     */
+    def range(pos: Position, from: Int, size: Int): Arr = {
+      val a = new Arr(pos, null)
+      a._isRange = true
+      a._rangeFrom = from
+      a._length = size
+      a
+    }
+
     /**
      * Threshold for lazy concat. Arrays with left.length >= this value use a virtual ConcatView
      * instead of eager arraycopy. Below this size, arraycopy is cheap enough that the indirection

sjsonnet/src/sjsonnet/stdlib/ArrayModule.scala

@@ -292,14 +292,11 @@ object ArrayModule extends AbstractFunctionModule {
         else if (sizeLong > Int.MaxValue)
           Error.fail("std.range result too large: " + sizeLong + " elements")
         else sizeLong.toInt
-      // Direct array allocation avoids Scala Range.map.toArray intermediates
-      val arr = new Array[Eval](size)
-      var i = 0
-      while (i < size) {
-        arr(i) = Val.cachedNum(pos, fromInt + i)
-        i += 1
-      }
-      Val.Arr(pos, arr)
+      // Lazy range: O(1) creation, elements computed on demand.
+      // Particularly beneficial for patterns like `std.range(1, 1000000) + [x]`
+      // where element-wise comparison never needs the full backing array.
+      if (size == 0) Val.Arr(pos, Val.Arr.EMPTY_EVAL_ARRAY)
+      else Val.Arr.range(pos, fromInt, size)
     }
   }
 

sjsonnet/test/resources/new_test_suite/lazy_range_correctness.jsonnet

@@ -0,0 +1,47 @@
+// Test that lazy range arrays produce correct results for all operations.
+local range10 = std.range(1, 10);
+local range5 = std.range(0, 4);
+local emptyRange = std.range(5, 3);  // empty when from > to
+
+std.assertEqual(std.length(range10), 10) &&
+std.assertEqual(range10[0], 1) &&
+std.assertEqual(range10[9], 10) &&
+std.assertEqual(range10[4], 5) &&
+// Iteration via array comprehension
+std.assertEqual([x * 2 for x in range5], [0, 2, 4, 6, 8]) &&
+// Concat with comparison (the key pattern for sharedConcatPrefixLength)
+std.assertEqual(range10 + [11] < range10 + [12], true) &&
+std.assertEqual(range10 + [11] > range10 + [12], false) &&
+std.assertEqual(range10 + [11] == range10 + [11], true) &&
+// Slicing
+std.assertEqual(range10[2:5], [3, 4, 5]) &&
+// std.sort on range (already sorted)
+std.assertEqual(std.sort(range5), [0, 1, 2, 3, 4]) &&
+// std.reverse on range
+std.assertEqual(std.reverse(range5), [4, 3, 2, 1, 0]) &&
+// std.member on range
+std.assertEqual(std.member(range10, 5), true) &&
+std.assertEqual(std.member(range10, 11), false) &&
+// std.map on range
+std.assertEqual(std.map(function(x) x + 1, range5), [1, 2, 3, 4, 5]) &&
+// Empty range
+std.assertEqual(std.length(emptyRange), 0) &&
+std.assertEqual(emptyRange, []) &&
+// Range equality
+std.assertEqual(std.range(1, 5), [1, 2, 3, 4, 5]) &&
+// Double-reverse preserves original range
+std.assertEqual(std.reverse(std.reverse(range5)), [0, 1, 2, 3, 4]) &&
+std.assertEqual(std.reverse(std.reverse(range10)), std.range(1, 10)) &&
+// Nested range operations
+std.assertEqual(std.foldl(function(acc, x) acc + x, range5, 0), 10) &&
+// Large range (deferred creation, only access subset)
+local bigRange = std.range(1, 100000);
+std.assertEqual(bigRange[0], 1) &&
+std.assertEqual(bigRange[99999], 100000) &&
+std.assertEqual(std.length(bigRange), 100000) &&
+// Range concat with shared prefix comparison
+local big = std.range(1, 10000);
+std.assertEqual(big + [1] < big + [2], true) &&
+std.assertEqual(big + [2] < big + [1], false) &&
+std.assertEqual(big + [1] == big + [1], true) &&
+true

sjsonnet/test/resources/new_test_suite/lazy_range_correctness.jsonnet.golden

@@ -0,0 +1 @@
+true