Description

While load testing a configuration that uses NATS subjects for communicating between WorkloadDeployments, one of the WorkloadDeployments stopped processing due to no cursor being returned when there was >1000 keys in the NATS subject. This occurred when spamming the requests in to create a high volume.

This happens when there is an error with unwrapping the key from the stream, which the error would then impact the "skip cursor count" and shifting the page by 1.

Fix

This fix moves the "skip cursor count" tracking to after unwrapping the key value. This ensures that if there was an error, the stream processing while loop returns immediately, and thus preserving the rest of the stream values (vs just eating them and now offset by the error count), so that the next call will have the cursor at the next location.

Testing

This was tested with 4 wasm workloads using NATS to communicate processing of incoming request. I load script was used to push a total 5000 requests in at 300 concurrently. Prior to the change the load script would hang until timeout and would have values "missing in pipeline" reported.

Test architecture used:

HTTP ?message=red,green,blue
        │
        ▼
    gateway (WorkloadDeployment w/k8s Service)
        │  publish
        ▼
    NATS subject1
        │
        ▼
    subject1-handler (WorkloadDeployment)
        │  stores values in KV bucket "nats-example" under generated keys
        │  publishes the keys to subject2
        ▼
    NATS subject2
        │
        ▼
    subject2-handler (WorkloadDeployment)
        │  retrieves the values by key from the KV bucket
        │  publishes the resolved key=value pairs
        ▼
    NATS processed
        │
        ▼
    monitor (WorkloadDeployment w/k8s Service)
           stores every message in KV bucket "processed-log"
           serves http://monitor.localhost.cosmonic.sh/ — a live page
           listing the messages (polls /messages JSON every 2s)

Signed-off-by: Jeremy Fleitz jeremy@cosmonic.com