Week 37 note: Client rate-limiting techniques
Edit: I received some feedback that what I am describing here may be rate-limiting techniques rather than backpressure techniques. I've edited the title to reflect this while keeping the URL the same as before.
As inspired first week note blog post by Miikka Koskinen, I'm experimenting with week notes. This is my attempt to write more about my rough experiences, thoughts, and ideas from the past week.
This week at work, I did load testing preparations. We are doing capacity planning for our services due to the need to adjust the horizontal scale. And for that, I needed to create some test data. Not too much but enough to simulate some scale and variance, we're talking roughly 5000 items.
For generating this data & querying afterwards, I found myself thinking about resilience techniques, eg. limiting the amount of requests I am making concurrently, putting Thread/sleep before/after the call is being made, and placing some jitter. The jitter was for inititating the requests at different times to avoid jamming the servers.
For pooled concurrency, there's fixed executor which is offered by promesa, a promise and concurrency library for Clojure(script). I ended up using promesa.exec/fixed-executor, Thread/sleep, and some jittering together to control the rate of concurrency.
A small demonstration of the techniques using Clojure:
clj -Sdeps '{:deps {funcool/promesa {:mvn/version "11.0.678"}}}'
(ns rate-limiting-example
(:require [clojure.math :as math]
[promesa.exec :as px]))
(defn jitter
[n interval]
(let [jitter-factor (- 1 (rand 2))
val (+ n (* interval jitter-factor))]
(if (pos? val)
(math/round val)
(int (math/floor val)))))
(defn nitz
[i x]
(prn (str (java.time.Instant/now)) (format "(%d) %d ms" i x)))
(defn frobnitzer
[n]
(letfn [(frob [i]
(let [jitter (jitter 5000 500)]
(Thread/sleep jitter)
(nitz i jitter)))]
(px/with-executor ^:shutdown (px/fixed-executor :parallelism 5)
(doall (px/pmap frob (range n))))))
rate-limiting-example=> (frobnitzer 10)
"2024-09-13T19:34:00.198346Z" "(2) 4558 ms"
"2024-09-13T19:34:00.606248Z" "(4) 4966 ms"
"2024-09-13T19:34:00.664063Z" "(1) 5024 ms"
"2024-09-13T19:34:00.714072Z" "(0) 5074 ms"
"2024-09-13T19:34:00.939139Z" "(3) 5299 ms"
"2024-09-13T19:34:04.851510Z" "(5) 4646 ms"
"2024-09-13T19:34:05.360741Z" "(6) 4748 ms"
"2024-09-13T19:34:05.363133Z" "(8) 4643 ms"
"2024-09-13T19:34:05.624080Z" "(9) 4679 ms"
"2024-09-13T19:34:05.703215Z" "(7) 5038 ms"
(nil nil nil nil nil nil nil nil nil nil)
These techniques seem very handy for manual data creation scripts where you have to be making a lot of requests to an API, or making a cronlike service that needs to be polling an API periodically, or for clientside code (you should use jittering for things like exponential backoff or retrying).
Of course, one should never forget about Little's law when it comes to calculating proper limitations.
P.s. While doing this, I had to make sure that I didn't hit the AWS lambda concurrency limit which is 1000 invocations by default. I set a max concurrency of 500. Using average runtime durations from previous lambda runs as a value for Thread/sleep seemed to work. But of course the lambdas had to be warmed before the actual run.