1

I am trying to execute request.get( ) method for few web urls. And the code is executing perfectly in my local system pycharm, jupyter notebook, google colab notebook etc. Only when I am executing the code in AWS Lambda function, it's working for few urls and throwing timeout exception for other urls. the list of urls are:

  1. 'https://www.google.com/'
  2. 'https://stackoverflow.com/'
  3. 'https://www.nseindia.com/'
  4. 'https://www.makemytrip.com/'

The code I've developed is

import json
import requests

def lambda_handler(event, context):
    url1 = 'https://www.google.com/'
    url2 = 'https://stackoverflow.com/'
    url3 = 'https://www.nseindia.com/'
    url4 = 'https://www.makemytrip.com/'
    header = {
        'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36'
    }
    response = requests.get(url=url3, headers=header,allow_redirects=False,timeout=60) 
    for i in response.cookies: 
        print(i.name +"==>>", i.value)
    return

The code is working for url1 and url2 but not working for url3 and url4 in AWS Lambda python shell only, but all the urls are working in google colab or jupyter notebook.

Improve this question
8
  • 1
    It could be that the site you're accessing is blocking AWS IP. See the answer here: stackoverflow.com/a/64871941/533711 Commented Oct 16, 2023 at 12:09
  • I read the link you shared. If the website is blocking AWS Ip, then how can I build a solution in AWS where requests.get() works. Commented Oct 16, 2023 at 12:23
  • "If the website is blocking AWS Ip, then how can I build a solution in AWS where requests.get() works." You can't. That's like asking how to get in somebody's house if they locked the door because they don't want you in their house. The owners of those websites have every right to block you from accessing the website if they don't want you accessing it through some automated tool like AWS Lambda. Commented Oct 16, 2023 at 13:48
  • Understood. But is there any way, may be by using some api, so that I can fetch the stock data in AWS? Commented Oct 16, 2023 at 17:28
  • @SaptarsiBasu: the NSE terms of use include this clause: "You should not conduct any systematic or automated data collection activities (including scraping, data mining, data extraction and data harvesting) on or in relation to our website.". The website is using technical measures to enforce this clause. Are you asking us to help you bypass these measures and get unauthorized access to the website? Commented Oct 17, 2023 at 3:17
Related questions
4
Python requests timeout in AWS Lambda
519
No Multiline Lambda in Python: Why not?
182
Using python Logging with AWS Lambda
Load 6 more related questions Show fewer related questions

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.