Python excels at network programming — from low-level sockets to high-level HTTP clients and web scraping.

HTTP Requests with requests

The most common way to interact with web APIs:

  import requests

# GET request
response = requests.get("https://api.github.com/users/python")
response.raise_for_status()
data = response.json()
print(data["public_repos"])

# POST with JSON body
payload = {"title": "My Post", "body": "Hello"}
response = requests.post(
    "https://jsonplaceholder.typicode.com/posts",
    json=payload,
    timeout=10,
)
print(response.status_code, response.json())

# Headers and authentication
headers = {"Authorization": "Bearer YOUR_TOKEN"}
response = requests.get("https://api.example.com/data", headers=headers)

Session Objects

Reuse connections for multiple requests:

  session = requests.Session()
session.headers.update({"User-Agent": "MyApp/1.0"})

for page in range(1, 6):
    resp = session.get(f"https://api.example.com/items?page={page}")
    process(resp.json())

Async HTTP with httpx

For async applications, use httpx:

  import httpx
import asyncio

async def fetch_users():
    async with httpx.AsyncClient() as client:
        response = await client.get("https://api.github.com/users/python")
        return response.json()

users = asyncio.run(fetch_users())

Raw Sockets

For custom protocols or learning how networks work:

  import socket

# TCP client
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
    s.connect(("example.com", 80))
    s.sendall(b"GET / HTTP/1.1\r\nHost: example.com\r\n\r\n")
    data = s.recv(4096)
    print(data.decode())

Web Scraping with BeautifulSoup

Extract data from HTML pages:

  import requests
from bs4 import BeautifulSoup

url = "https://quotes.toscrape.com"
response = requests.get(url)
soup = BeautifulSoup(response.text, "html.parser")

for quote in soup.select("div.quote"):
    text = quote.select_one("span.text").get_text()
    author = quote.select_one("small.author").get_text()
    print(f'"{text}" — {author}')

Respectful Scraping

  • Check robots.txt before scraping
  • Add delays between requests (time.sleep(1))
  • Set a descriptive User-Agent header
  • Prefer official APIs when available

Building a Simple HTTP Server 

  from http.server import HTTPServer, BaseHTTPRequestHandler
import json

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(json.dumps({"message": "Hello"}).encode())

HTTPServer(("localhost", 8080), Handler).serve_forever()

For production, use FastAPI, Flask, or Django instead.

REST API Best Practices

When consuming APIs in Python:

  1. Always set timeouts — requests.get(url, timeout=10)
  2. Handle errors — check response.status_code or use raise_for_status()
  3. Use retries — for transient failures with exponential backoff
  4. Cache responses — when data doesn’t change frequently
  5. Respect rate limits — implement backoff when receiving 429 responses

Retries with Exponential Backoff

Transient network failures are common. Retry with increasing delays:

  import time
import requests

def fetch_with_retry(url, max_retries=3):
    for attempt in range(max_retries):
        try:
            response = requests.get(url, timeout=10)
            response.raise_for_status()
            return response.json()
        except requests.RequestException as e:
            if attempt == max_retries - 1:
                raise
            wait = 2 ** attempt  # 1s, 2s, 4s
            print(f"Retry {attempt + 1} after {wait}s: {e}")
            time.sleep(wait)

Use tenacity for production-grade retry logic:

  from tenacity import retry, stop_after_attempt, wait_exponential

@retry(stop=stop_after_attempt(3), wait=wait_exponential(min=1, max=10))
def fetch_data(url):
    return requests.get(url, timeout=10).json()

Handling Rate Limits (429) 

  def fetch_respecting_limits(url, session):
    while True:
        response = session.get(url)
        if response.status_code == 429:
            retry_after = int(response.headers.get("Retry-After", 60))
            time.sleep(retry_after)
            continue
        response.raise_for_status()
        return response.json()

Downloading Files 

  url = "https://example.com/data.zip"
with requests.get(url, stream=True, timeout=30) as r:
    r.raise_for_status()
    with open("data.zip", "wb") as f:
        for chunk in r.iter_content(chunk_size=8192):
            f.write(chunk)

stream=True avoids loading large files entirely into memory.

URL Parsing 

  from urllib.parse import urljoin, urlparse, parse_qs

base = "https://api.example.com/v1/"
endpoint = urljoin(base, "users?page=2")

parsed = urlparse("https://example.com/search?q=python&page=1")
print(parsed.netloc)           # example.com
print(parse_qs(parsed.query))  # {'q': ['python'], 'page': ['1']}

WebSocket Client

For real-time APIs, use the websockets library:

  import asyncio
import websockets

async def listen():
    async with websockets.connect("wss://echo.websocket.org") as ws:
        await ws.send("Hello!")
        reply = await ws.recv()
        print(reply)

asyncio.run(listen())

See the WebSocket Chat Project for a full server implementation.

Testing Network Code

Mock HTTP calls in tests — never depend on live APIs:

  from unittest.mock import patch, MagicMock

@patch("myapp.api.requests.get")
def test_fetch_users(mock_get):
    mock_get.return_value = MagicMock(
        status_code=200,
        json=lambda: [{"id": 1, "name": "Alice"}],
    )
    users = fetch_users()
    assert len(users) == 1

Network programming connects your Python applications to the entire web.