Requests

What it is and why it exists

Requests is a third-party synchronous HTTP client for Python. It provides one consistent API for URL encoding, headers, cookies, authentication, redirects, TLS certificate verification, and response decoding, leaving application code to focus on the HTTP exchange instead of sockets.

Each call involves two objects worth distinguishing. A PreparedRequest is an encoded request ready to send; a Response holds the returned status, headers, content, and the request that was actually sent. That distinction helps when diagnosing signature failures, incorrect parameter encoding, or redirects.

Requests fits command-line tools, background jobs, service-to-service calls, and synchronous requests in tests. Its I/O blocks the current thread, and using a Session does not make it asynchronous. An async application waiting on many concurrent requests usually needs a client built for that execution model instead of scattered synchronous calls inside its event loop.

The library only handles client-side HTTP mechanics. It cannot decide whether a particular 404 means None or a domain error, nor whether a POST can be retried. Status interpretation, replay safety, response-size limits, and trust in the target URL remain part of the application contract.

How it works

Convenience functions such as requests.get() ultimately create, prepare, and send a request through a transport adapter. Preparation encodes params, data, json, and files, adds headers that can be derived, and produces a PreparedRequest. Sending handles connections, TLS, proxies, redirects, and response reads.

Follow one call from input to result in this order:

1. Choose an HTTP method and target URL.
2. Give Requests the query parameters, headers, cookies, authentication, and body.
3. Let the Session merge session-level configuration and create a PreparedRequest.
4. Let the mounted HTTPAdapter take a connection from its pool and send the bytes.
5. Receive response headers and, depending on stream, either read or defer the response body.
6. Check the status, parse the content, and close the response or consume its body completely.

The parameters represent different parts of HTTP and are not interchangeable:

A Session retains default headers, cookies, authentication, and adapter configuration, and it uses urllib3 to reuse underlying connections to the same host. A connection pool can reclaim a connection only after the response body has been consumed or the response has been closed. A session therefore owns configuration state and network-resource lifetimes.

Response.content returns raw bytes, Response.text decodes a string with the selected encoding, and Response.json() decodes the body as JSON. None of these interfaces proves that the operation succeeded. Check the status and media type against the API contract before choosing a parser.

Exceptions also occur at different layers. Connection failures and timeouts are transport problems; raise_for_status() turns 4xx and 5xx responses into HTTPError; malformed JSON raises requests.exceptions.JSONDecodeError. Do not erase those distinctions with one broad except Exception.

Examples

These four examples avoid public test services. The first only prepares a request, while the other three start temporary HTTP servers bound to the loopback interface, so their output does not depend on an external API or network state.

Inspecting the request that will be sent

Passing parameters to Requests is safer than assembling a URL and JSON by hand. A prepared request exposes the encoded result without sending network traffic.

The list of key-value pairs preserves both include parameters. The json= argument serializes the body and sets Content-Type; take explicit control of serialization only when the server requires a different JSON representation.

A PreparedRequest may contain access tokens, cookies, or personal data. Log only fields that may be disclosed, and redact headers and bodies rather than dumping the entire object during diagnosis.

Reading a successful JSON response

This local service echoes a search term. The client sets one-second connect and read timeouts, then checks the status before calling json().

import json
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from threading import Thread
from urllib.parse import parse_qs, urlsplit

import requests

class CatalogHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        query = parse_qs(urlsplit(self.path).query)
        body = json.dumps({"query": query["q"][0], "count": 2}).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json; charset=utf-8")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, format, *args):
        pass

server = ThreadingHTTPServer(("127.0.0.1", 0), CatalogHandler)
worker = Thread(target=server.serve_forever, daemon=True)
worker.start()

try:
    url = f"http://127.0.0.1:{server.server_port}/search"
    response = requests.get(
        url,
        params={"q": "blue mug", "limit": 2},
        headers={"Accept": "application/json"},
        timeout=(1, 1),
    )
    response.raise_for_status()
    print(response.request.path_url)
    print(response.headers["Content-Type"])
    print(response.json())
finally:
    server.shutdown()
    server.server_close()

/search?q=blue+mug&limit=2
application/json; charset=utf-8
{'query': 'blue mug', 'count': 2}

response.request.path_url shows that the space became +; the input dictionary needed no manual escaping. The server declares both JSON and a character set. A real client should still validate the media type promised by its contract instead of accepting any body that happens to parse.

When a body may be large, do not call response.text or response.content merely to log it because either loads the full content into memory. With stream=True, consume chunks through iter_content() inside a with block.

Separating HTTP errors from timeouts

The domain function maps a found order and a 404 to two explicit values. A transport timeout takes another path, so callers do not confuse “the server confirmed absence” with “no result arrived.”

import json
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from threading import Thread
import requests

class OrderHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        status = 200 if self.path == "/orders/42" else 404
        payload = {"id": 42, "state": "paid"} if status == 200 else {"error": "order not found"}
        body = json.dumps(payload).encode()
        self.send_response(status)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, format, *args):
        pass

def fetch_order(session, base_url, order_id):
    try:
        response = session.get(f"{base_url}/orders/{order_id}", timeout=(1, 1))
        response.raise_for_status()
        return response.json()
    except requests.HTTPError as error:
        detail = error.response.json().get("error", "unknown error")
        return {"status": error.response.status_code, "error": detail}
    except requests.Timeout:
        return {"error": "request timed out"}

server = ThreadingHTTPServer(("127.0.0.1", 0), OrderHandler)
Thread(target=server.serve_forever, daemon=True).start()
try:
    base_url = f"http://127.0.0.1:{server.server_port}"
    with requests.Session() as session:
        print(fetch_order(session, base_url, 42))
        print(fetch_order(session, base_url, 7))
finally:
    server.shutdown()
    server.server_close()

{'id': 42, 'state': 'paid'}
{'status': 404, 'error': 'order not found'}

The example assumes that error responses are JSON because its local server makes that contract explicit. When a real client cannot rely on this, handle media type and JSONDecodeError separately on the error path, and cap any response excerpt retained for logs.

It catches Timeout because the caller maps connection and read timeouts to the same result. Catch ConnectTimeout and ReadTimeout separately when recovery differs: the former commonly occurs before delivery, while the latter can leave the server-side result unknown.

Retrying only replayable calls

The final local server returns 503 twice, then succeeds. The adapter retries only GET and permits two retries, producing three total attempts.

from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from threading import Thread

import requests
from requests.adapters import HTTPAdapter
from urllib3.util import Retry

class BusyHandler(BaseHTTPRequestHandler):
    attempts = 0

    def do_GET(self):
        BusyHandler.attempts += 1
        status = 503 if BusyHandler.attempts < 3 else 200
        body = b"busy" if status == 503 else b"ready"
        self.send_response(status)
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, format, *args):
        pass

retry = Retry(
    total=2,
    status_forcelist={503},
    allowed_methods={"GET"},
    backoff_factor=0,
)
server = ThreadingHTTPServer(("127.0.0.1", 0), BusyHandler)
Thread(target=server.serve_forever, daemon=True).start()
try:
    with requests.Session() as session:
        session.mount("http://", HTTPAdapter(max_retries=retry))
        url = f"http://127.0.0.1:{server.server_port}/health"
        response = session.get(url, timeout=(1, 1))
        print(response.status_code, response.text)
        print("attempts:", BusyHandler.attempts)
finally:
    server.shutdown()
    server.server_close()

200 ready
attempts: 3

A production configuration usually needs nonzero backoff and jitter, plus contract-specific handling of Retry-After. This example uses zero backoff only to finish the local run immediately; it is not a production policy to copy.

Retry.total caps retries after the first attempt rather than total calls. Even when an HTTP method is idempotent by specification, the client must confirm that the server implementation, body, and preconditions really permit replay.

Pitfalls

Omitting a timeout or treating it as a total deadline

Fix: pass an explicit (connect_timeout, read_timeout) to every external call and enforce the operation’s overall deadline at a higher layer. Count retries, backoff, DNS, connection setup, and reads against one call budget.

Parsing JSON before checking status

Fix: interpret the status against the contract first, then check the media type and parse the corresponding schema. Give error responses their own schema, and retain only bounded, redacted diagnostics when parsing fails.

Retrying writes unconditionally

Fix: retry only methods explicitly known to be replayable. When a create operation must be retried, establish a real idempotency protocol, such as a caller-stable key with atomic server-side deduplication and replay of the stored result.

Disabling TLS verification to silence an error

Fix: keep verification enabled. Configure a controlled CA bundle for a private certificate authority; repair name, validity, or chain failures instead of bypassing them at the call site.

Leaking session state or streamed responses

Fix: create sessions at a clear authentication and ownership boundary, then close them with a context manager. Put streamed reads in with too, enforce a maximum size, and either consume or explicitly close every response.

Fetching an untrusted URL

Fix: prefer a fixed origin and controlled paths. If target addresses are required, restrict schemes and ports, resolve and validate hosts and IPs, revalidate every redirect, and use an outbound network policy as a second boundary.

Further reading

• Requests quickstart
• Requests advanced usage
• Requests developer interface
• urllib3 Retry reference