HTTP/HTTPS Learning Manual (1) Overview of HTTP

In the TCP/IP reference model of computer networking, there are a total of four layers, arranged from bottom to top as follows: Network Interface Layer, Internet Layer, Transport Layer, and Application Layer.

The Network Interface Layer corresponds to the physical layer and data link layer in the OSI model. The physical layer ensures the transmission of raw bit streams over physical media, guaranteeing that data is transferred as a stream of bits from one device to another. The data link layer manages data transfer between directly connected devices, defining and managing data transmission through frames.

The Internet Layer primarily uses the Internet Protocol (IP) for forwarding packets from source hosts to destination hosts, passing through various routers or other nodes that use routing algorithms to determine the path for packet delivery.

The Transport Layer is responsible for end-to-end reliable communication between hosts. This layer utilizes two protocols to support different methods of data transmission: one is a reliable, connection-oriented data transfer service, and the other is an unreliable, connectionless data service, supported respectively by the TCP and UDP protocols. While the Internet Layer does not ensure reliable data transfer, the Transport Layer uses TCP over the Internet Layer to achieve reliable data transmission and provides additional network functionalities such as traffic control and congestion control to enhance overall network performance.

The Application Layer is the topmost layer in the TCP/IP reference model, where various protocols operate to address different usage scenarios. One such protocol introduced in this article is HTTP/HTTPS, which resides in this layer. HTTP (HyperText Transfer Protocol) is used extensively for distributed, collaborative, and hypermedia information systems. It defines the formats for requests and responses between clients (e.g., browsers) and servers. HTTP forms the basis of communication on the World Wide Web, enabling clients to request resources (e.g., HTML files, image files) from servers, which respond with the requested resources or process outcomes accordingly.

HTTP Introduction

HTTP is designed for distributed, collaborative, and hypermedia information systems. It is an application-layer protocol for transmitting hypermedia documents, such as HTML files, over the Internet. HTTP defines how clients (such as web browsers) request resources from a server and how servers respond to such requests. HTTP is fundamental to communication on the World Wide Web, allowing clients to request resources from servers and servers to respond with content or process outcomes based on the request.

Web Clients Use HTTP to Communicate with Servers

Various web servers worldwide store a wide range of resources, including images (e.g., jpg, png), HTML files, videos, music, and movies. When users browse a webpage using a browser, the browser establishes a connection with the web server and transmits data using HTTP. The client browser sends an HTTP request to the web server, which responds with the requested data. The client browser then parses and renders this data for user presentation.

URI

To facilitate resource requests on web servers, each resource needs a specific identifier. The Uniform Resource Identifier (URI) was created for this purpose, uniquely identifying resources globally. For example, https://www.jackey-song.com/img/boyWithCat.webp identifies an image resource on a server. URI includes both URL and URN.

URL

A Uniform Resource Locator (URL) is the most widely used type of URI. It specifies the precise location of a resource for browsers to access.

URL comprises three main parts:

The scheme (http:// or https://).
The Internet address of the server, such as jackey-song.com or an IP address.
Additional details specifying the resource’s location on the server or other parameters.

URN

A Uniform Resource Name (URN) is a unique name for a specific content type, supporting access through multiple network protocols. As of July 2024, URN implementation is still in the experimental stage.

Transactions

An HTTP transaction consists of an HTTP request and an HTTP response. Communication occurs through HTTP messages.

Methods

HTTP supports multiple methods to achieve various purposes:

GET: Requests a resource from the server.
PUT: Stores data sent by the client in a server resource.
DELETE: Removes a resource from the server.
POST: Sends client data to a gateway application on the server.
HEAD: Similar to GET but without the message body, used primarily for testing.

Status Codes

In daily use, you have likely encountered “404 NOT FOUND”, a status code indicating that the requested resource was not found.

Common status codes include:

200 OK
302 Redirect
404 Not Found

Resources in HTML Pages

Those who have done front-end development and written HTML know that HTML pages contain various resources. Without these resources, web pages would consist only of plain text. When you visit a webpage and right-click to inspect or press F12 to enter debugging mode, you can view the page source code and its included resources.

Messages

A simple HTTP request message:

GET /test/hi-here.txt HTTP/1.0   | Start Line
--------------------------------
Accept: text/*                   | Header Fields
Accept-Language: en,fr
                                 | Headers end with an empty line

A simple HTTP response message:

HTTP/1.0 200 OK                  | Start Line
--------------------------------
Content-type: text/plain         | Header Fields
Content-length: 19
                                 | Headers end with an empty line
--------------------------------
Hello world! I am a message!     | Body

Start Line: Specifies the action in a request or the outcome in a response.
Header Fields: Consist of multiple lines, each containing a name and value, providing information. Headers end with an empty line.
Body: Can be empty or contain various data types.

Connection

HTTP operates at the application layer. Below it are the transport layer (TCP protocol) and the internet layer (IP protocol). HTTP leverages TCP/IP, which provides a reliable connection. Therefore, HTTP focuses on application-level concerns without needing to manage lower-level details. TCP ensures error-free data transfer, sequential delivery, and an unsegmented data stream.

Before exchanging HTTP messages between clients and servers, a TCP/IP connection is established using an Internet Protocol address (IP address) and port number. An IP address is akin to a country’s name, and a port number is like a country’s port. Communication between two hosts on the network is similar to shipping goods between countries. A host may run multiple network programs, each requiring a port number to ensure data is sent to the correct program. HTTP typically uses port number 80, while HTTPS (HTTP Secure) uses port number 443.

In daily usage, IP addresses are often unseen because they consist of numeric strings that are difficult for humans to remember. Instead, users frequently encounter domain names like jackey-song.com or baidu.com. Computers cannot interpret domain names directly, so the Internet features a Domain Name System (DNS) for mapping domain names to corresponding IP addresses. DNS is a large, distributed database system found on numerous servers worldwide, handling vast volumes of DNS resolution tasks daily. When accessing a website via its domain name on the Internet, a computer’s initial task is to query DNS to obtain the corresponding IP address. For those interested in the details of DNS implementation, further research is recommended.

When a browser accesses a resource:

The browser extracts the server hostname from the URL.
The browser queries DNS for the IP address corresponding to the hostname.
The port number is extracted from the URL.
A TCP connection is established with the HTTP server (web server).
The browser sends an HTTP request message to the server.
The server sends an HTTP response message to the browser.
The connection is closed, and the browser displays the resource.

TCP Three-Way Handshake and Four-Way Handshake

Three-Way Handshake:

The process of establishing a TCP connection is called “three-way handshake,” ensuring that both communication parties can interact properly. The detailed process is as follows:

First handshake: The client sends a SYN (Synchronize) segment to the server, which includes only TCP header information, with the SYN flag set to 1 to signify a connection request. The client also selects an initial sequence number seq=x, placing it in the TCP segment’s sequence number field. After sending, the client enters the SYN_SENT state.
Second handshake: Upon receiving the SYN segment from the client, if the server agrees to establish a connection, it responds with a SYN+ACK segment. This segment acknowledges the client’s SYN segment with the ACK flag set to 1, confirming the client’s SYN segment and setting the acknowledgment number to x+1 (the initial sequence number of the client plus one). The server also selects an initial sequence number seq=y, placing it in the TCP segment’s sequence number field. After sending, the server enters the SYN_RCVD state.
Third handshake: Upon receiving the SYN+ACK segment from the server, the client responds with an ACK segment. This segment acknowledges the server’s SYN segment with the ACK flag set to 1 and the acknowledgment number set to y+1 (the initial sequence number of the server plus one), confirming the server’s SYN segment. At this point, the TCP connection is established, and both client and server enter the ESTABLISHED state, enabling data transmission.

TCP’s Four-Way Handshake

The process of terminating a TCP connection is called the “four-way handshake,” ensuring both sides can safely close the connection. Here’s a detailed breakdown:

First Handshake (FIN from client):

The client (or data sender) sends a FIN segment to the server (or data receiver), indicating it wants to close the connection. After sending this, the client enters the FIN_WAIT_1 state.
Second Handshake (ACK from server):

Upon receiving the FIN segment from the client, the server responds with an ACK segment, acknowledging the receipt of the FIN. The acknowledgment number is set to the sequence number of the received FIN segment plus one. This ACK indicates the server has received the client’s request to close the connection but may still have data to send. The server enters the CLOSE_WAIT state.
Third Handshake (Optional):

If the server indeed has pending data to transmit to the client, it sends a FIN segment to the client after transmitting all data. This indicates the server also wants to close the connection. After sending this FIN, the server enters the LAST_ACK state.
Fourth Handshake (ACK from client):

Upon receiving the FIN segment from the server, the client responds with an ACK segment, acknowledging the receipt of the server’s FIN. The acknowledgment number is set to the sequence number of the received FIN plus one. This ACK confirms the client has received the server’s request to close the connection and agrees to close it. After sending this ACK, the client enters the TIME_WAIT state, waiting for a period (usually twice the maximum segment lifetime, MSL) to ensure the server receives the ACK. If no retransmission of the FIN from the server occurs during this time, the client closes the connection and enters the CLOSED state. The server also closes the connection upon receiving the client’s ACK and enters the CLOSED state.

HTTP Versions

HTTP 0.9
- Release Year: 1989
- Features:
  - The first version of the HTTP protocol, inventing the World Wide Web and creating the world’s first web browser.
  - Supports only the GET request method, and servers can only respond with HTTP text data.
  - Does not include HTTP headers, status codes, or error codes; uses a single line of request information.
  - After receiving data, the client closes the connection via the four-way handshake, hence also known as a single-line protocol.
HTTP 1.0
- Release Year: 1996
- Features:
  - Enriched the interactive response file types, enabling web pages to play music before this time.
  - Introduced HTTP protocol headers; responses added status codes and types of transferable content (MIME media types).
  - Remained a single-line protocol where one connection could handle just one request and response, with high costs for establishing physical connections.
  - Servers close after responding, with no persistent connections support.
HTTP 1.1
- Release Year: 1997
- Features:
  - Introduced persistent connections (keep-alive), allowing multiple requests and responses on one connection, increasing transmission efficiency.
  - Introduced pipelining, allowing clients to send multiple requests before receiving responses, further enhancing efficiency.
  - Increased caching control, supporting more complex caching mechanisms.
  - Added more request methods like PUT, DELETE, enhancing HTTP functionality.
HTTP/2 (Originally HTTP 2.0)
- Release Year: 2015
- Features:
  - Developed based on the SPDY protocol, the first major update to HTTP since HTTP 1.1 in 1999.
  - Uses binary frame layers (not plain text) to transfer HTTP headers and data, improving transfer efficiency and compression.
  - Supports multiplexing, allowing multiple requests and responses to be sent concurrently on a single TCP connection, resolving head-of-line blocking issues in HTTP 1.x.
  - Introduces HPACK algorithm for compressing HTTP headers, further reducing transmitted data volume.
  - Supports server push, enabling servers to proactively send resources to clients, reducing client request times.
HTTP/3
- Release Year: June 2022
- Features:
  - Replaces TCP with the QUIC protocol as a general-purpose transport protocol in the application layer. QUIC is a multiplexed transport protocol built on UDP, improving connection speed and user experience.
  - Addresses TCP performance degradation issues during mobile device network switching.
  - Reduces latency, supports zero round-trip time (0-RTT) connection establishment, allowing clients to skip handshake processes with connected servers.
  - Provides comprehensive encryption, setting up encrypted connections at the transport layer by default, enhancing security.

HTTP continues to evolve and develop.

Proxy

A proxy is an HTTP entity located between a client and a server, receiving HTTP requests from the client and acting on behalf of the client to access the server (potentially modifying requests from the client). It returns the response to the client. Proxies can filter both requests and responses.

Cache

Web cache, also known as proxy cache, is a specialized HTTP proxy server that stores resources from web servers. When users revisit the same resources, they can retrieve data directly from the web cache, balancing the load on web servers and avoiding network congestion.

Gateway

A gateway is a specialized server, typically an intermediary entity between clients and servers. It is often used to convert HTTP traffic into other protocols.

Tunnel

In HTTP, a tunnel refers to a special technique that encapsulates non-HTTP protocol data using HTTP or HTTPS. This allows such protocol data to be transmitted through restricted network environments. Specifically, an HTTP tunnel establishes a virtual data channel via HTTP or HTTPS connections, ensuring that non-HTTP or HTTPS protocol data can be transmitted without interception or blocking by intermediate network devices such as firewalls or NATs.

Agent Proxy

An agent proxy represents a client program initiating HTTP requests on behalf of a user. A web browser serves as a type of user agent proxy. Other types of proxies include web spiders or web robots. For instance, search engines function as web spiders, constantly browsing the internet, gathering information, and compiling it into searchable databases. When you use a search engine to search for a keyword, it retrieves the relevant content from its database and presents it to you using algorithms.

Jackey Song's Blog