How to download a web page?¶

The httpx library sends HTTP requests to web pages and allows you to read their content. Most standard tasks are much easier than the standard module urllib and implemented cleaner than the outdated requests library. Sending data to web forms via HTTP GET and POST, submitting files and managing cookies are features of this useful library.

Install httpx with:

pip install httpx

The following code reads a static web page:

import httpx

r = httpx.get('https://www.python.org')
print(r.status_code)
print(r.text[:1000])

How to submit a HTTP GET request?¶

Here is a search with HTTP GET parameters:

import httpx

# Search scientific articles on PubMed:
url = "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi"
param_dict = {'db':'pubmed', 'term':'escherichia', 'rettype':'uilist'}

r = httpx.get(url, params=param_dict)
print(r.text)