HTTP headers serve an important function; they enable both the client and server to transmit more details that are contained within the sent request or the response.

HTTP stands for Hypertext Transfer Protocol and it manages the structure and transfer of information on the internet, and the specific ways web servers and browsers should respond to various requests.

Sent requests include a header and the HTTP headers contain additional information as may be required by the webserver. The web server then responds to this by sending specific data back to the client. The returned data is well structured depending on the software specifications as contained in the header.

HTTP Headers According to Context 

Categorized according to their context, HTTP headers can be grouped in the following ways:

HTTP Response Header

Response headers are sent by the web server in response to requests in an HTTP transaction. These headers contain information on the status of the initial request; if it went through or not, the type of connection used, encoding, and others. HTTP response headers would send an error code in the event where the request doesn’t go through.

HTTP header error codes are divided into the following:

• 1xx – informational
• 2xx – success
• 3xx – redirection
• 4xx – client error
• 5xx – server error

HTTP Status Codes

1xx Informational

• 100 Continue
• 101 Switching Protocols
• 102 Processing (WebDAV)

2xx Success

• 200 OK
• 201 Created
• 202 Accepted
• 203 Non-Authoritative Information
• 204 No Content
• 205 Reset Content
• 206 Partial Content
• 207 Multi-Status (WebDAV)
• 208 Already Reported (WebDAV)
• 226 IM Used

3xx Redirection

• 300 Multiple Choices
• 301 Moved Permanently
• 302 Found
• 303 See Other
• 304 Not Modified
• 305 Use Proxy
• 306 (Unused)
• 307 Temporary Redirect
• 308 Permanent Redirect (experimental)

4xx Client Error

• 400 Bad Request
• 401 Unauthorized
• 402 Payment Required
• 403 Forbidden
• 404 Not Found
• 405 Method Not Allowed
• 406 Not Acceptable
• 407 Proxy Authentication Required
• 408 Request Timeout
• 409 Conflict
• 410 Gone
• 411 Length Required
• 412 Precondition Failed
• 413 Request Entity Too Large
• 414 Request-URI Too Long
• 415 Unsupported Media Type
• 416 Requested Range Not Satisfiable
• 417 Expectation Failed
• 418 I'm a teapot (RFC 2324)
• 420 Enhance Your Calm (Twitter)
• 422 Unprocessable Entity (WebDAV)
• 423 Locked (WebDAV)
• 424 Failed Dependency (WebDAV)
• 425 Reserved for WebDAV
• 426 Upgrade Required
• 428 Precondition Required
• 429 Too Many Requests
• 431 Request Header Fields Too Large
• 444 No Response (Nginx)
• 449 Retry With (Microsoft)
• 450 Blocked by Windows Parental Controls (Microsoft)
• 451 Unavailable For Legal Reasons
• 499 Client Closed Request (Nginx)

5xx Server Error

• 500 Internal Server Error
• 501 Not Implemented
• 502 Bad Gateway
• 503 Service Unavailable
• 504 Gateway Timeout
• 505 HTTP Version Not Supported
• 506 Variant Also Negotiates (Experimental)
• 507 Insufficient Storage (WebDAV)
• 508 Loop Detected (WebDAV)
• 509 Bandwidth Limit Exceeded (Apache)
• 510 Not Extended
• 511 Network Authentication Required

HTTP Request Header

This header is sent in an HTTP transaction by the client to the browser. These headers contain details about the source of the request such as the type of browser being used and the version.

HTTP request headers are important where HTTP communication is being used and one of its importance is helping proper display of information. Since websites give feedback in terms of layout and design based on the type of operating system, machine, and application that is sending the request, the information provided by the request header would come in handy.

The information on the software and hardware of the request source is sometimes called the user agent and it's important otherwise the data sent back by the website’s server may be displayed incorrectly.

In some cases where the user agent is not recognized by the website’s server, it may either block the request entirely or display a default HTML version of the page that has been prepared for cases like this.

HTTP Entity Header

HTTP entity-headers contain information about the body of the resource and each entity-tag comes as a pair. For example, Content-Language, Content-Length, and so on.

General HTTP Header

General HTTP headers apply to both the requests and responses but they don’t apply to the content of both request and response itself. You can find these headers in an HTTP message and the most common are Connection, Cache-Control, or Date.

HTTP Headers According to Proxy Interaction

Proxy-Authenticate

Proxy authenticate is a response header that defines the authentication method to have access to a resource behind a proxy. It authenticates your request to the proxy and allows the request to go from there to the target site.

Connection

Connection is a general header that controls the state of a network connection; whether it would stay open or not after the current transaction finishes.

Proxy-Authorization

This is a request header that has the authentication credentials for a user agent to a proxy server.

TE

TE is a request header that specifies the transfer encodings that are acceptable to the user agent.

Transfer-Encoding

This is specific to the type of encoding that is used to safely transfer the payload body to the sender. It isn’t applied to the resource itself, but the message shared between nodes.

 Keep-Alive

This header allows the client to set the maximum number of requests and a timeout after that in the connection. The connection header must be set to Keep-Alive for this header to be valid.

Trailer

Trailer is a response header that allows the sender to include additional fields at the end of a message. An example is a digital signature.

These examples of HTTP headers are a few of the many that exist and it’s almost impossible to list out all the available HTTP headers. Note that HTTP headers can send out many types of requests or ask for specific details like language or encoding.

Upgrade

The upgrade header just as its name implies is used to modify an already established connection by upgrading it. This modification takes place over the connection to the same protocol but over a different protocol.

Clients may use the upgrade header to get a server to switch protocol to another of the listed ones. The server may ignore the tray and when this happens, it would be as if the request wasn’t sent at all. 

Authentication

Authorization 

It contains the needed credentials for authentication to be successful between a user agent and a server. 

Proxy-authenticate

This is the authentication method that should be used when a resource behind a proxy server is to be accessed. 

WWW- Authenticate

This is the authentication method that is used when a resource is to be accessed. 

Proxy-authorization 

This authorization contains the necessary credentials a proxy server would need to authenticate a user agent. 

Caching

Age

It’s the time duration in seconds that an object has been in the proxy cache

Cache-control

This is in charge of the caching mechanisms

Clear-site-data

This clears the browsing data that is associated with the requesting server

Expires

The date and time that a response is no longer valid

Warning 

This is information about possible problems that may be encountered

Connection management

Connection 

It determines whether the network connection would stay open after the current transaction is over. 

Keep-alive

It controls the time a connection should stay open. 

Content negotiation

Agent

It provides information to the server about data that can be sent in response to a request

Accept-charset

It provides information about the character encoding that the client would understand 

Accept-encoding 

The accept-encoding is used in the response resource 

Accept-language 

This provides the server with the acceptable language the user would understand.

Controls

Expect

It shows what the server should do to handle the request properly 

Cookies

Cookie

It contains HTTP cookies that have previously been stored and sent by the server. 

Set-cookie

It sends cookies from the server to the user agent

Do not track

DNT

This shows the tracking preference of the user. 

TK

It shows the tracking status of the response 

Downloads

Content-disposition 

This shows if the transmitted resource should be handled as a download and the browser display a “save as” dialogue, or if it should be displayed without the header. 

Message body information

Content-encoding

This is used to specify the compression algorithm to be used. 

Content-length 

This is the size of the resource in bytes

Content-type

It shows the type of resource as a media

Proxies

Forwarded 

It has information from the client side of proxies that are changed when the proxy gets involved with requests. 

X-forwarded-for

It identifies the IP of the source of the request when a client tries to connect to the web. 

X-forwarded-host

It identifies the original host that the client used to connect to the proxy. 

X-forwarded-proto

It identifies the protocol that is used when connecting to a proxy. 

Redirect

Location

It gives the URL that a page should be redirected to. 

Request context

From

It contains the email address for a human user to control the requesting user agent. 

Host

It specifies the domain name of the server and TCP ports through which the server is listening. 

Referer 

This is the web address of the page that contains a link to which the current page was reached. 

User-agent

It has a characteristic string that gives the network protocol the ability to identify some data like the OS, type of app, and software version. 

Response context

Allow

This is a list of HTTP request methods that the resource supports. 

Server

It contains information about the software used to handle a request.

Even though HTTP headers can be used in web scraping to avoid blocks on your IP address, they can also be used for web security. The HTTP security headers are a special contract between the web browser and the developer and are defined by the HTTP response headers that are responsible for setting the website’s security level.

Below are some of the most popular HTTP headers that allow for secure web applications:

Content Security Policy Header

This header gives an additional security layer and helps to keep the web server safe from various attacks such as Cross-Site Scripting (XSS). It defines the sources of content which are then approved and loaded by the browser.

Referrer-Policy Header

This header controls the amount of referrer information sent through the referrer header that should be included in the requests.

X-Frame-Options Header

This header serves to protect the website’s visitors from clickjacking attacks

X-XSS-Protection Header

This header is used in the configuration of XSS protection. You can find it in chrome, firefox, and Safari browsers.

Feature Policy Header

This header either allows or denies the use of a web browser in its frame and also in the content within elements.

X-Content-Type-Options Response Header

This header serves as a marker used by the webserver. It indicates that no changes should be made to the MIME types advertised in content headers but rather they should be followed.

It’s easy to check your HTTP header security online. Some tools can provide you with this feature on your website and all you need to have is a URL.

Also Read : How to Use a Proxy in Internet Explorer

There are two major reasons why you should optimize HTTP headers. They are:

• To ensure that the quality of data that is retrieved is of high quality for proper analysis and accurate results
• To reduce the probability of a web scraper getting blocked by the target server while scraping

So you can see that using HTTP headers would have a direct impact on the type of data you extract, and the quality of data that is gotten from the web. HTTP headers when used properly would also reduce the chances of your IP address getting blocked by target servers as you scrape the web.

Most website owners have come to terms with the fact that their data would be scraped even if they don’t accept this action.

Apart from having to share their data against their will, web scrapers also slow down websites with their multiple requests and so all these make website owners use every tool they can lay hands on to prevent web scraping.

One of the techniques they may use is to automatically block fake user agents that are detected. Some web servers may even be programmed to display incorrect information if a fake user agent is detected and this would have severe consequences.

Since HTTP headers also carry information to the web servers, you can make your internet request seem organic by optimizing the content of the message it carries. Doing this makes it less likely for your requests to the webserver to get blocked.