Performance: Pipelined SET+EXPIRE is ~2.3x slower than MSET

[REASY]

There is a significant performance difference when comparing Pipelined SET + EXPIRE operations against Native MSET commands.

Benchmark details

• The total number of elements in the benchmark is 100 000.
• Keys are randomly (Alphanumeric) generated strings of length 80.
• Values are randomly generated bytes of length 20.

According to the End-to-End Benchmarks: Write Throughput results (single-threaded AsyncRedisClientV1 with batch size 10000):

1. MSET achieves an average throughput of ~1.41 M elem/s or ~134.96 MiB/s with average latency 70.664 ms.
2. Pipelined SET + EXPIRE (Manual) achieves an average throughput of ~610.88 K elem/s or ~58.25 MiB/s with significantly higher latency, 163.70 ms.

To Reproduce

Steps to reproduce the behavior:

1. Clone https://github.com/REASY/dragonfly-playground-rs
2. Start the Dragonfly instance using the provided Docker Compose configuration (v1.34.2).
3. Build the benchmark image: dragonfly-playground-rs-dev:0.1.0.
4. Run the write_throughput benchmark:

   docker run -it --network=host \
     dragonfly-playground-rs-dev:0.1.0 \
     -c "REDIS_BENCH_SERVER=127.0.0.1:6379 REDIS_BENCH_MULTI_SET=1 cargo bench --bench write_throughput"

Environment

• OS: Ubuntu 22.04.5 LTS
• Kernel: 5.15.0-153-generic, x86_64
• Containerized?: Docker
• Dragonfly Version: v1.34.2 (image ghcr.io/dragonflydb/dragonfly:v1.34.2)
• Hardware: Bare-metal, 2 x AMD EPYC 7643 48-Core Processor (192 vCPU), 1536 GiB RAM

docker-compose.yml

name: dragonfly-playground-rs
services:
  dragonfly:
    image: ghcr.io/dragonflydb/dragonfly:v1.34.2
    environment:
      DFLY_max_multi_bulk_len: 2000100
      DFLY_pipeline_buffer_limit: 872415232
      DFLY_pipeline_queue_limit: 2000100
      DFLY_maxmemory: "40g"
    ulimits:
      memlock: -1
    ports:
      - "6379:6379"
    deploy:
      resources:
        limits:
          cpus: "32"
          memory: "48g"
    cpuset: "0-32"

Reproducible Code Snippet

The logic producing the lower performance (Pipelined Manual SET + EXAT) is defined in redis_client.rs, the benchmark write_throughput.rs

[romange]

why it is a bug?

[REASY]

why it is a bug?

Ah, misclicked, this isn't a bug, more a question, sorry.

[romange]

MSET can be run in parallel for all the keys in the command.
pipelined commands are run sequentially for a single connection. Dragonfly must ensure consistency as there could be interdependencies between commands. This is why pipelined commands are slower from a single connection.
If your commands operate on differrent keys, it is advised to use multiple connections when working with Dragonfly to be able to parallelize IO streams and utilize the server more efficiently.

[REASY]

@romange this is a benchmark of running pipelined commands in parallel, pipelined_set_with_expiry_manual (x1, x2, ..., means the write parallelism). It seems like there is a sweet spot for the write parallelism because increasing it beyond 4 provides only diminishing returns.

I know it might be too general question, but any advice to get the best write throughout of pipelined SET + expiry? And do you think there can be a single bulk version of it like MSET?

[REASY]

Another interesting observation is that the size of key and value are important for MSET (benchmark multi_set), the smaller the key (and value), the faster it is. However, it isn't the case for pipelined SET + expiry.