Python requests timeout in AWS Lambda

Question

Python requests is timing out from AWS Lambda for below url's.

https://www.nseindia.com/get-quotes/derivatives?symbol=NIFTY https://www.nseindia.com/api/option-chain-indices?symbol=NIFTY

And its working for other urls.

Working url - https://support.oneskyapp.com/hc/en-us/article_attachments/202761627/example_1.json

Below is the code I am using.

import requests

def lambda_handler(event, lambda_context):
    headers = {"User-Agent":'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.87 Safari/537.36'}
    
    with requests.session() as rq:
        print("Loading cookies")
        rq.get("https://www.nseindia.com/get-quotes/derivatives?symbol=NIFTY", headers=headers)

        print("Execute get request")
        result = rq.get("https://www.nseindia.com/api/option-chain-indices?symbol=NIFTY", headers=headers)

My Lambda is using Python3.7 runtime, not associated with any VPC and timeout is set to 1 minute. I tried with Lambda execution timeout of 5 minutes still no luck.

Output

Response: { "errorMessage": "2020-11-17T04:58:34.039Z 0f69a07d-6139-4aab-b1cf-310b19243735 Task timed out after 60.06 seconds" }

Request ID: "0f69a07d-6139-4aab-b1cf-310b19243735"

Function logs: START RequestId: 0f69a07d-6139-4aab-b1cf-310b19243735 Version: $LATEST Starting request session... Loading cookies END RequestId: 0f69a07d-6139-4aab-b1cf-310b19243735 REPORT RequestId: 0f69a07d-6139-4aab-b1cf-310b19243735 Duration: 60060.48 ms Billed Duration: 60000 ms Memory Size: 128 MB Max Memory Used: 59 MB Init Duration: 383.95 ms 2020-11-17T04:58:34.039Z 0f69a07d-6139-4aab-b1cf-310b19243735 Task timed out after 60.06 seconds

I am guessing this would be related to User-Agent headers. Any help would be highly appreciated.

Any break-through on this issue, @LG555 ???

TheCodeCache
– TheCodeCache

2022-01-15 20:39:15 +00:00
Commented Jan 15, 2022 at 20:39 — TheCodeCache
– TheCodeCache, Commented Jan 15, 2022 at 20:39
@LG555 ,any solutions?

Divyank
– Divyank

2024-07-03 12:23:11 +00:00
Commented Jul 3, 2024 at 12:23 — Divyank
– Divyank, Commented Jul 3, 2024 at 12:23

ivica · Accepted Answer · 2020-11-17 08:47:20Z

1

It could be that https://www.nseindia.com/ is blocking AWS IPs because they don't want to be scraped?

This is what I got when trying to open it from one of my EC2 machines

[ec2-user@ip-10-2-120-90 ~]$ curl https://www.nseindia.com/get-quotes/derivatives?symbol=NIFTY
<HTML><HEAD>
<TITLE>Access Denied</TITLE>
</HEAD><BODY>
<H1>Access Denied</H1>

You don't have permission to access "http&#58;&#47;&#47;www&#46;nseindia&#46;com&#47;get&#45;quotes&#47;derivatives&#63;" on this server.<P>
Reference&#32;&#35;18&#46;d6d3dead&#46;1605602802&#46;2598bb12
</BODY>
</HTML>

answered Nov 17, 2020 at 8:47

ivica

1,4282 gold badges24 silver badges38 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

LG555 Over a year ago

When I do curl from my local machine, I am getting the same permission issue. But it's getting response from Python console in local machine.

ivica Over a year ago

I'll what i'm getting with the requests module from EC2 tomorrow.

LG555 Over a year ago

I did not understood your above comment.

ivica Over a year ago

I tried running your code from one of my EC2 instances and got a timeout as well.

>>> headers = {"User-Agent":'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.87 Safari/537.36'} >>> resp = requests.get("https://www.nseindia.com/get-quotes/derivatives?symbol=NIFTY", headers=headers, timeout=30) requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='www.nseindia.com', port=443): Read timed out. (read timeout=30)

as you can see, this is my outgoing IP, from the AWS range. $ curl ifconfig.so 34.252.34.74

Collectives™ on Stack Overflow

Python requests timeout in AWS Lambda

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related