Ad

MongoDB - Slow Concurrent Responses And Huge Difference In Term Of Speed For Different Response Sizes

I was trying to mock the real use case by creating hundreds of promises to query MongoDB with the same queries (not real though since the queries in real would be different at the same time) and then run a Promise.all. I have noticed 2 interesting behaviors:

  1. The first return might be much faster than the last return (usually double the time for the last one). For this one, does MongoDB lock the collection so the queries had to wait in a queue?
  2. If I reduce the return size from about 40 fields to 1 field, the query run 10 times faster.

Can anyone explain those 2 behaviors? Thanks.

Ad

Answer

Mongo does not queue and block queries.

try looking at your machine's performances, Promise.all/map are supposed to be used to make a process more efficient but you are obviously capped by your CPU/RAM availability, i suspect the answer to both your questions lies here.

you should try and find the sweet spot with Promise.map and {concurrency:x} if the queries are too heavy, i found my machine not to be able to deal with hundreds of complex queries queue'd together at the same time, however when i piplined them with a smaller concurrency i could efficiently use my RAM/CPU availability without overloading it.

Ad
source: stackoverflow.com
Ad