How To Fetch Request Body Of Waiting Or Queued Requests On Nginx?

- 1 answer

I am using NGINX as a load balancer and proxy server for a python application hosted by gunicorn. The application server(gunicorn) takes longer duration of time to give response and there is single worker of it. The requests hitting the nginx server at the time when application server is not free then those requests stay in processing queue waiting to be processed. Here I need a record of the requests siting in the waiting queue along with there request body to get there status.

I tried using nginx logs and also some external third party tools to do so but was unsuccessful in fulfilling my requirement.

My request body looks like this:

  "BatchNbr": "Batch_80",
  "InputPath": "TestPath/pdfs/20190516",

And I maintain batch status in the following format:

Batch Number  StartTime (IST)         EndTime (IST)      Status
Batch_80    2019-10-16 14:16:39  2019-10-16 14:16:39       QUEUED
Batch_70    2019-10-16 14:13:04  2019-10-16 14:13:04       QUEUED
Batch_71    2019-10-16 14:13:04  2019-10-16 14:13:06       FAILURE
batch_test1 2019-10-16 14:09:22  2019-10-16 14:09:22       SUCCESS

I need Batch Status as queued when the Batch request is sitting in the waiting queue of the nginx server, the status changes to RUNNING when request reaches to application server, and when the request's processing is complete the status is changed to SUCCESS. Any leads would be of great help.



There is possibly no way to fetch waiting requests data from nginx. We can only get number of hits, number of waiting requests or number of active requests but we can't fetch request data from it. So I tried another way to do my task. I fetched data from tcp dumps of the machine over a port on which nginx was running, this gave me whole request data as soon as the user hit the api, then parsed that data in python to get details what I required. Here is the code to fetch data from tcp layer:

# sudo tcpdump -nn -A -s1500 -l -i ens4 port 5000
while True:
    p = sub.Popen(('sudo', 'tcpdump', '-nn', '-A', '-s1500', '-l', '-i', 'ens4', 'port', '5000'), stdout=sub.PIPE)
    for row in iter(p.stdout.readline, b''):
        val = row.rstrip()
        # process here

This code is watching tcp dumps on port 5000 You can also check that by executing this command on linux terminal: sudo tcpdump -nn -A -s1500 -l -i ens4 port 5000