Understanding HTTP client using node:http module

What is HTTP ?

HTTP (Hypertext Transfer Protocol) is a protocol designed to transfer file (hypermedia) or information (hypertext) over a network.

But, do you really know how does it work? How does it sends information?

Lets address some of the key concepts of http and then we can dive deep into more details.

What is hypertext?
Hypertext is a system of linking text to other text, enabling users to navigate between different parts of a document or different documents altogether—just like clicking on a hyperlink on a webpage.

What is hypermedia?
Hypermedia extends this concept beyond text to include images, videos, and other multimedia elements.

What is protocol?
Protocol is a set of predefined instructions or set of rules.

Why we need a protocol here?
To standardize the internet. Protocol is must which governs how the data or information will be transferred over a network.

and hope you know about the transfer ;)

Okay, we just understood the meaning of HTTP but what are the rules?

HTTP is stateless, meaning that each request from a client to a server is treated as independent. The server does not retain any information from previous requests. This means that even if you log in once, the server does not inherently "remember" you. To maintain user sessions, the web uses mechanisms like cookies, session storage, and authentication tokens.

Lets simplify

What is a state?
In this context state is nothing but a remembered information over time. We call it stateful (Maintains a state over time) in computer terminology.

Why do we need HTTP?

The internet is a massive network where computers communicate with each other. Without a standardized set of rules (protocol), devices wouldn’t be able to understand each other. HTTP provides a structured way to transfer data, ensuring compatibility between different systems.

Okay, Lets understand how HTTP works now

HTTP uses TCP (Transfer layer protocol) under the hood, What is TCP now? another protocol?

Yes, HTTP is relied on TCP although now with HTTP/3 you can use UDP as well, but that discussion is for another day.

Do i need to understand TCP?
Yes, Its better that you know TCP to understand HTTP.

Go read about it if you are not familiar with TCP

But in simpler terms
Before any HTTP request a TCP connection is created using TCP 3 way handshake

TCP sends SYN (seq=x) then server responds back with SYN_ACK (seq=y, ack=x+1) means i acknowledge for your SYN (Sync) then client responds back with ACK (seq=x+1, ack=y+1) saying i received your response (Acknowledge).

3 Way handshake is a very interesting part. For now remember the these sequence numbers are important and each request has its own sequence and acknowledgment to maintain the order of the data.

Client → Server: SYN, seq=x (Client sends a sync request with an initial sequence number)
Server → Client: SYN, seq=y, ACK, ack=x+1 (Server responds with its own sync request and acknowledges the client's sequence)
Client → Server: ACK, seq=x+1, ack=y+1 (Client acknowledges the server's sequence, completing the handshake)

TCP Connection is a duplex stream so its not optimal or efficient to open TCP connection for each request. So http's default behavior to use same TCP connection for multiple requests by informing to server that keep connection alive, You will see in our example how to keep it alive.

What is Stream now?
To move something in continuous flow
Its beautiful way to flow the information, And we will learn this in next chapter.

There are ways you can inform server that you want to keep connection alive for 8 seconds or 4 seconds, so the TCP connection will wait for provided number of seconds and if no new requests then it will terminate.

Now, lets understand HTTP

How Does an HTTP Request Work?

Once a TCP connection is established, the client sends an HTTP request to the server.

Here’s an example of how a Node.js client can send an HTTP request:

const http = require('http');

const agent = new http.Agent({
  keepAlive: true,  // Keep the connection alive
});

const options = {
  hostname: 'example.com', // Server address
  port: 80, // HTTP default port
  path: '/', // The specific endpoint to access
  method: 'GET', // The type of request (GET, POST, etc.)
  headers: {
    'User-Agent': 'Node.js Client',
    'Accept': 'text/html',  
  },
  agent: agent,  // Attach the agent to the request options
};

const req = http.request(options, (res) => {
  let data = '';

  // Collect response data
  res.on('data', (chunk) => {
    data += chunk;
  });

  // Response received completely
  res.on('end', () => {
    console.log('Response:', data);
  });
});

req.on('error', (err) => {
  console.error('Request Error:', err);
});

req.end();

Breaking Down the Request

Each HTTP request consists of several key components:

Request Line
Method (e.g., GET, POST, PUT, DELETE): Specifies the type of request.
Path: The resource being requested (e.g., '/' for the homepage).
HTTP Version: Specifies the HTTP version (e.g., HTTP/1.1 or HTTP/2).

Headers (Metadata)
Host: The target server address (e.g., example.com).
User-Agent: Identifies the client (e.g., browser or program).
Accept: Specifies the content type the client can handle.
Content-Type: Specifies the format of the request body (e.g., application/json, text/html).
Other headers may include Authorization, Cache-Control, and more.

After sending the request, the server processes it and responds with an HTTP response, which includes:

Status Code (e.g., 200 OK, 404 Not Found, 500 Internal Server Error)
Response Headers (like Content-Type, Content-Length, etc.)
Response Body (e.g., HTML page, JSON data, or other content)

Here, req.end() does not terminate the TCP connection, It only informs the server that i am done sending information for this request and now waiting for response.

What happens if you dont end the request or forgot to call req.end()?

The request will never be complete, and connection will remain open. The server will wait for the remaining data to be sent and client will be left hanging. This will lead to resource leakage and poor performance.

Cool, Hope you understand basics of HTTP, How does it work what does it uses to send the information and handhsake.

We will learn TCP and Stream in coming chapter stay tuned.

Happy Learning :)