Ad

What Is A Concise Way Of Counting How Many Requests Were Sent?

I am wondering how to count the number of sent requests concisely. There is an API that allows you to send maximum 50 requests in a minute to any endpoint of that API. Every 60 seconds they reset the number of requests that I sent to zero again, after the first request were sent. I need to do a lot of requests and I wonder, how I can keep track of them correctly.

async dataCollector(){
    let urlList = this.urlList; // 100+ urls
    for(let url in urlList){
      await this.getResource(url).then( data =>
        // do something with data
      )
}

Now I have to somehow check in getResource() how many requests I sent and so on such that I just wait until more requests are available. What would be the best way to check them?

getResource(url){
    return this.http.get(url);
}

I appreciate any advise.

Ad

Answer

You could keep a count of the requests sent, and the time of the first request. You also need a way to wait for the previous requests, that can be done with a Promise chain:

 chain = Promise.resolve();
 count = 0;
 first = undefined;

 aquireSlot() {
    return this.chain = this.chain.then(async () => {
       if(this.count >= 20) {
         await timer(60_000 - (Date.now() - this.first));
         this.count = 0;
         this.first = undefined;
       }
       if(!this.first) this.first = Date.now();
       this.count += 1;
    });
 }

 async getResource(url){
   await this.aquireSlot();
   return await this.http.get(url);
 }

  // the implementation of timer is left to the reader

With this code, every getResource call kind of ends in a queue (a promise chain). If the queue only has less than 20 elements, the chain simply continues and the calls are done. If there are 20 elements, the queue is paused till at least 60 seconds passed since the first call. Then the count is reset, and the whole thing repeats.

Ad
source: stackoverflow.com
Ad