Python-Requests, extract url parameters from a string

Question

I am using this awesome library called requests to maintain python 2 & 3 compatibility and simplify my application requests management.

I have a case where I need to parse a url and replace one of it's parameter. E.g:

http://example.com?param1=a&token=TOKEN_TO_REPLACE&param2=c

And I want to get this:

http://example.com?param1=a&token=NEW_TOKEN&param2=c

With the urllib I can achieve it this way:

from urllib.parse import urlparse
from urllib.parse import parse_qs
from urllib.parse import urlencode

url = 'http://example.com?param1=a&token=TOKEN_TO_REPLACE&param2=c'

o = urlparse(url)
query = parse_qs(o.query)
if query.get('token'):
    query['token'] = ['NEW_TOKEN', ]
    new_query = urlencode(query, doseq=True)
    url.split('?')[0] + '?' + new_query

>>> http://example.com?param2=c&param1=a&token=NEW_TOKEN

How can you achieve the same using the requests library?

Martijn Pieters · Accepted Answer · 2015-02-04 18:52:54Z

73

You cannot use requests for this; the library builds such URLs if passed a Python structure for the parameters, but does not offer any tools to parse them. That's not a goal of the project.

Stick to the urllib.parse method to parse out the parameters. Once you have a dictionary or list of key-value tuples, just pass that to requests to build the URL again:

try:
    # Python 3
    from urllib.parse import urlparse, parse_qs
except ImportError:
    # Python 2
    from urlparse import urlparse, parse_qs

o = urlparse(url)
query = parse_qs(o.query)
# extract the URL without query parameters
url = o._replace(query=None).geturl()

if 'token' in query:
    query['token'] = 'NEW_TOKEN'

requests.get(url, params=query)

You can get both the urlparse and parse_qs functions in both Python 2 and 3, all you need to do is adjust the import location if you get an exception.

Demo on Python 3 (without the import exception guard) to demonstrate the URL having been built:

>>> from urllib.parse import urlparse, parse_qs
>>> url = "http://httpbin.org/get?token=TOKEN_TO_REPLACE&param2=c"
>>> o = urlparse(url)
>>> query = parse_qs(o.query)
>>> url = o._replace(query=None).geturl()
>>> if 'token' in query:
...     query['token'] = 'NEW_TOKEN'
... 
>>> response = requests.get(url, params=query)
>>> print(response.text)
{
  "args": {
    "param2": "c", 
    "token": "NEW_TOKEN"
  }, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.5.1 CPython/3.4.2 Darwin/14.1.0"
  }, 
  "origin": "188.29.165.245", 
  "url": "http://httpbin.org/get?token=NEW_TOKEN&param2=c"
}

edited Feb 4, 2015 at 18:52

answered Feb 4, 2015 at 18:40

Martijn Pieters

1.1m326 gold badges4.2k silver badges3.4k bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

Martijn Pieters Over a year ago

@Gab: I've updated the post to make it clear how you can maintain compatibility between Python 2 and 3; the same functionality exists in both versions, just in different locations. requests cannot help there.

Gab Over a year ago

effectively, I checked more in details the requests website and github folder it appears you are right. +1 for showing a simplified version of the code using the requests library in the end!

user3064538 Over a year ago

Martijn, you should update this answer. This function is available as requests.utils.urlparse (it's literally the same function, it comes from compat which comes from this file) and has been since 2017 I think.

Martijn Pieters Over a year ago

@Boris the requests.compat module is not part of the public API. It does what my answer does, essentially, but that doesn’t mean you can rely on it continuing to do so. It is only imported into requests.utils because the implementations in that module need it. In other words, the only reason it is there is as an implementation detail.

user3064538 Over a year ago

@MartijnPieters I'm not sure about that. None of the functions in that module are documented as far as I can tell, but it says "This module provides utility functions that are used within Requests that are also useful for external consumption". I would think if that was the idea they would be imported with an underscore import urlparse as _urlparse

|

vvvvv · Accepted Answer · 2019-09-16 15:03:28Z

22

Using requests only:

query = requests.utils.urlparse(url).query
params = dict(x.split('=') for x in query.split('&'))

if 'token' in params:
    params['token'] = 'NEW_TOKEN'

requests.get(url, params=params)

edited Sep 16, 2019 at 15:03

vvvvv

32.9k19 gold badges70 silver badges103 bronze badges

answered Apr 17, 2018 at 7:07

Dorik1972

2212 silver badges3 bronze badges

1 Comment

aliqandil Over a year ago

I think a requests.utils.unquote might also be needed, to prevent sending characters like Space, as "%20" and such.

Siruj · Accepted Answer · 2022-12-22 14:26:29Z

-3

This works for me:

params = request.args
my_param = params.get('my_param')

edited Dec 22, 2022 at 14:26

Siruj

4703 silver badges15 bronze badges

answered Dec 21, 2022 at 7:12

Lesha Neat

1

1 Comment

sourcream Over a year ago

Where did request come from?

Collectives™ on Stack Overflow

Python-Requests, extract url parameters from a string

3 Answers 3

7 Comments

1 Comment

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

7 Comments

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related