Intro

Let’s talk about HTTP. I was not planning to write about it, but things changed when I started writing for my API design post. HTTP is a fundamental building block of the internet. As engineers, we craft web APIs, a lot of which rely on HTTP, so I think there’s a lot of value in understanding the high level concepts of HTTP.

I’ll only cover the basics here. Of course, there’s a lot of things to learn about HTTP, so I’ll leave some resources (towards the end) that I personally find useful.

Let’s get started.

What is HTTP?

Definition

Taken from here:

The Hypertext Transfer Protocol (HTTP) is an application protocol for distributed, collaborative, hypermedia information systems. HTTP is the foundation of data communication for the World Wide Web, where hypertext documents include hyperlinks to other resources that the user can easily access, for example by a mouse click or by tapping the screen in a web browser.

There are some words above that not everyone would know about, but that’s okay. The point is that HTTP is a communication protocol that’s generally used when two machines have to talk over a network.

To be a bit more specific, HTTP is an application layer protocol. There are other layers as well.

HTTP is not the only application layer protocol. Some other popular examples are: Telnet, FTP, SMTP. Each of them were created to serve a certain purpose. HTTP’s was to facilitate the communication between two machines on the internet.

Properties

These are some of the characteristics of HTTP:

• It follows a client-server model. Clients makes a HTTP request, and server responds by sending a HTTP response.
• It’s a stateless protocol. This means that from a protocol perspective, two HTTP requests are independent. Having said that, HTTP is not sessionless. We’ll discuss this briefly later on under “Cookies”.

Terms and Concepts

General

Domain name

From here:

In an HTTP URL, the first substring that follows the initial http:// or https:// is called the domain name. This domain name is hosted on a server where the document resides.

A server isn’t necessarily a physical machine: several servers can reside on the same physical machine. Or, one server can be handled by several machines, cooperating to produce the answer or balancing the load of the requests between them. The key point is that semantically one domain name represents one single server.

So when making HTTP requests, how do we know where the request should go? Domain name tells us that¹.

For example when you enter http://google.com in your browser, google.com is the domain name.

You can read more about this here.

Resource

Earlier we discussed that HTTP follows a client-server model using requests and responses.

A resource is defined as the target of a HTTP request. It’s the “thing” that the client is requesting from the server. It can be a document, photo, or really anything else. A lot of times, your browser would request HTML and CSS files from the server. In this case, these files would be considered as resources in the HTTP protocol.

MIME type

MIME is short for “Multipurpose internet mail extensions”. From here:

MIME is a standard to describe documents in other forms beside ASCII text, e.g. audio, video and images. Initially used for E-Mail attachments, it has become the de facto standard to define types of documents anywhere.

In the context of HTTP, we’re mostly interested in MIME type which is also knows as “media type” or “content type”.

So why is MIME type important? Well it can be sent in requests and responses for communicating the “type of resource”. For example, if the server is sending a PDF file back to the client, it can indicate by setting content type (a HTTP header which we’ll discuss in a bit) to application/pdf.

URL, URI, URN

Okay, so now we have:

• Domain: tells us where to go
• Resource: what to fetch or send back. MIME type also tells us the type of the resource.

How’s the resource identified though? There can be multiple resources we’re interested in, so it’d be nice if we have a standard format that uniquely identifies a resource. There’s a lot to talk about here, so I’ll recommend reading through this article. Let’s summarize this though:

• URI is short for Uniform Resource Identifier. A URI identifies a resource using a short string. You can find the details of URI syntax in IETF RFC 3986.
• URL is short for Uniform Resource Locator. A URL locates (and thus also identify) a resource. Locate here means what protocol to use to reach, and the domain name (address of the machine to talk to). URL also identify the resource though, so they’ll also have the resource URI embedded. Whatever you enter in your browser is the URL.

In other words, URL is just a type of URI. In fact, it’s a stricter form of URI: one that has the location information (protocol and domain name). This implies that all URLs are URIs, but all URIs are not URLs.

Here’s an image (taken from here) illustrating it:

What is an URN now? URN is short for Uniform Resource Name. Just like URL, URN is again a type of URI. However, URN does not help locate the resource like URL does. URN just identifies the resource by a “name”. The key here is that the name has to follow a certain URN format. This format is as follows:

urn:<NID>:<NSS>

NID is a Namespace identifier, and NSS is the Namespace specific string. Fancy terms, right? Don’t worry. If you’re really curious, you can read more about them in the IETF RFC linked below. But let’s take an example. urn:fruit:12 represents a fruit resource with an id 12.

To know more about the URN format, you can go through the IETF RFC 8141.

Okay, now hopefully you’ve a better idea of what each of these terms mean. There’s a lot of confusion around these, especially on what the differences are between them. Here are some good readings if you are interested:

• Confusion around URL and URI
• Stack overflow discussion

HTTP specific

Version

Like any other piece of software, the HTTP protocol has evolved over time as well. So now we’ve different versions of HTTP: 0.9, 1, 1.1, 2, and 3. There’s a lot of history here that I won’t go over in this post. But I find it interesting. If you are curious, read this article on Evolution of HTTP.

As of the writing of this post, HTTP 1.1 has been prominent over the internet (and that too for a long time). HTTP 2 is more of a “modern web” thing, that’s still being adopted slowly. HTTP 3 on the other hand is “the future”; we’re far from it.

Header

From here:

HTTP headers let the client and the server pass additional information with an HTTP request or response. An HTTP header consists of its case-insensitive name followed by a colon (:), then by its value. Whitespace before the value is ignored.

You can read about all the HTTP headers here. But let’s see some of the common HTTP headers:

• Connection header: General (both in response and request) header that tells whether the connection should remain open after the request/response is sent.
• Accept header: Informs the server about what MIME types are acceptable.
• Content-Type header: If client or server is sending a resource, this header tells the MIME type of that resource.
• Content-Length header: The size of the resource, in decimal number of bytes.
• Authorization header: Contains the credentials to authenticate a client with a server
• Cache-Control header: Has instructions for caching in both requests and responses.

Method

From here:

HTTP defines a set of request methods to indicate the desired action to be performed for a given resource. Although they can also be nouns, these request methods are sometimes referred to as HTTP verbs. Each of them implements a different semantic, but some common features are shared by a group of them: e.g. a request method can be safe, idempotent, or cacheable.

You can find more HTTP methods here, but I’m referencing some of the common ones (taken from the same site) here:

• The GET method requests a representation of the specified resource. Requests using GET should only retrieve data.
• The HEAD method asks for a response identical to that of a GET request, but without the response body.
• The POST method is used to submit an entity to the specified resource, often causing a change in state or side effects on the server.
• The PUT method replaces all current representations of the target resource with the request payload.
• The DELETE method deletes the specified resource.
• The PATCH method is used to apply partial modifications to a resource.

Response status code

HTTP response code is sent by the server to the client indicating the status of the response. From here:

HTTP response status codes indicate whether a specific HTTP request has been successfully completed. Responses are grouped in five classes:

• Informational responses (100–199),
• Successful responses (200–299),
• Redirects (300–399),
• Client errors (400–499),
• and Server errors (500–599).

You can read more details here, but I’m referencing some common status codes here:

• 200 OK. The request succeeded. The exact success criteria can be read in the link above.
• 202 Accepted. The request has been received but not yet acted upon.
• 204 No Content. There is no content to send for this request.
• 307 Temporary Redirect. The server sends this response to direct the client to get the requested resource at another URI with same method that was used in the prior request.
• 308 Permanent Redirect. This means that the resource is now permanently located at another URI, specified by the Location HTTP Response header.
• 400 Bad Request. The server could not understand the request due to invalid syntax.
• 401 Unauthorized. Although the HTTP standard specifies “unauthorized”, semantically this response means “unauthenticated”. That is, the client must authenticate itself to get the requested response.
• 403 Forbidden. The client does not have access rights to the content; that is, it is unauthorized, so the server is refusing to give the requested resource. Unlike 401, the client’s identity is known to the server.
• 404 Not Found. The server can not find the requested resource. In the browser, this means the URL is not recognized. In an API, this can also mean that the endpoint is valid but the resource itself does not exist. Servers may also send this response instead of 403 to hide the existence of a resource from an unauthorized client. This response code is probably the most famous one due to its frequent occurrence on the web.
• 500 Internal Server Error. The server has encountered a situation it doesn’t know how to handle.

HTTP flow

To get an idea of what steps are taken between a typical client-server communication, read this.

HTTP request and response formats

We’ve discussed HTTP headers, methods, status codes, URL. These are basic building blocks that form a HTTP request or response. Both the request and response follow a format that we’ll discuss in this section.

This entire section references this page. For convenience, I’m quoting the content here directly.

HTTP messages, as defined in HTTP/1.1 and earlier, are human-readable. In HTTP/2, these messages are embedded into a binary structure, a frame, allowing optimizations like compression of headers and multiplexing. Even if only part of the original HTTP message is sent in this version of HTTP, the semantics of each message is unchanged and the client reconstitutes (virtually) the original HTTP/1.1 request. It is therefore useful to comprehend HTTP/2 messages in the HTTP/1.1 format.

There are two types of HTTP messages, requests and responses, each with its own format.

Request format

Here’s the format of a HTTP request:

Requests consists of the following elements:

• An HTTP method, usually a verb like GET, POST or a noun like OPTIONS or HEAD that defines the operation the client wants to perform. Typically, a client wants to fetch a resource (using GET) or post the value of an HTML form (using POST), though more operations may be needed in other cases.
• The path of the resource to fetch; the URL of the resource stripped from elements that are obvious from the context, for example without the protocol (http://), the domain (here, developer.mozilla.org), or the TCP port (here, 80).
• The version of the HTTP protocol.
• Optional headers that convey additional information for the servers.
• Or a body, for some methods like POST, similar to those in responses, which contain the resource sent.

Response format

Here’s an example response:

Responses consist of the following elements:

• The version of the HTTP protocol they follow.
• A status code, indicating if the request was successful, or not, and why.
• A status message, a non-authoritative short description of the status code.
• HTTP headers, like those for requests.
• Optionally, a body containing the fetched resource.

Example

There are multiple ways to make HTTP calls:

• Browser makes HTTP calls. For Chrome, you can see these in the dev tools.
• Browser extensions e.g. Postman.
• Web apps e.g. httpbin.
• CLI tools e.g. curl, httpie. You can also use this online converter to convert curl requests to httpie requests.

I love httpie, so let’s use that to make a HTTP request to a server, and see what the server responds with:

Advance HTTP topics

• Caching
• Sessions, Cookies
• Authentication
• Connection management
• Query parameters
• URL encoding

More resources

If you’re interested to learn more about HTTP, or web development in general, I highly encourage you to read through Mozilla’s fantastic docs. The HTTP section is here.

Another good resource is HTTP Made Really Easy.

---

1. You may ask computer addresses are not a bunch of characters, but some numbers forming an IP Address. So how does HTTP request know which machine to reach? Well, there’s this system called DNS, which is responsible for mapping domain names to IP addresses. ↩︎