Email Protocols - SMTP, POP and IMAP

Email is one of the primary services running on the internet. Here in this article, we will discuss the email that how email is working end devices. What application, protocol and services are required for email? Email messages are stored in a database on the email server. The email is using store-and-forward method for sending and storing the messages. The email clients communicate with the servers running mail services to send and receive email. The client connected server communicates with other mail servers to transport messages from one domain to another. The client does not communicate directly with another email client when sending an email. But, both mail clients rely on the mail server to transport messages.

There is three types of protocol which are used for email process: SMTP (Simple Mail Transfer Protocol), POP(Post Office Protocol), and IMAP(internet messaging Access Protocol). The application layer process that sends mail uses SMTP. But a client retrieves email using POP or IMAP.

Simple Mail Transfer Protocol (SMTP) Operation

The SMTP message formats required a message body with message header. The body of the message can hold any amount of text, the message header must have a properly formatted recipient email address and a sender address.

When a client sends an email message, the client SMTP process connects with a server SMTP process on port 25. When the connection is established, then the client tries to send the email message to the server. After the server receives the email message, it either places the message in a local account in case of the local recipient or forwards the message to another mail server for delivery. If the destination email server is busy or not online then the SMTP spools message to be sent at the later time. The server checks for the queue periodically and attempts to send them again. If the message expiration time is over and the message is still in the queue, it is returned to the sender as an undeliverable message.

 

The figure below illustrates the technique of message sending. The client sends an email message to admin@fschub.com. The SMTP / POP server-1 will receive the message. Server-1 will check the recipient's list of local recipients. If found the message will be placed on the local account, if not found, the message will be forwarded to SMTP / POP server-2.

Post Office Protocol (POP) Operation

The POP server passively listening on TCP port 110 for client connection requests. When a client needs to make use of the POP service; it sends a request to start a TCP connection with the server. On establishing a connection the POP server sends a welcome to the client.  After connection establish both client and POP server exchange commands and responses until the connection is closed or aborted.

With POP, incoming email messages are downloaded to the client and then removed from the server. The POP server works as a temporary holding area for mail until it is downloaded by the mail client. So there is no central place where email messages are kept. Because of no centralized storage for email messages; it is no an attractive choice for a small business that needs a centralized storage for backup.

Internet Messaging Access Protocol (IMAP) Operation

The Internet Message Access Protocol (commonly known as IMAP is another protocol that describes a technique to retrieve email messages from the remote mail server. An IMAP server usually listens on port 143 and IMAP over SSL is assigned port number 993. Unlike POP, when the user connects to an IMAP- server, copies of the mail are downloaded to the client application. The original messages are held in reserve on the server until the user explicitly deletes them. Users view copies of the messages in their email client software.

Incoming email messages are stored on the email server that in the recipient's email box. The user retrieves the messages with an email client that uses one of a number of email retrieval protocols. The majority of clients support the standard protocols, SMTP for sending an e-mail message,  POP and IMAP for retrieving email.

The IMAP client can make a file hierarchy on the server to organize and store emails. When a user wants to delete a mail; the server synchronizes that command and deletes the message from the mail server.

HTTP and HTML

When an address is typed into a browser, the browser establishes a connection to the web service running on the server. The protocol for establishing a connection is HTTP. HTTP means HyperText Transfer Protocol. Hyper Text Transfer Protocol is the basic protocol used by the World Wide Web. This protocol defines how messages are formatted and transmitted, and what actions should take by Web servers and web browsers in response to various commands.

The URL(Uniform Resource Locator) and URI(Uniform Resource Identifier)  are the names the majority people used with the web addresses. If we want to open a web address http://fschub.com/ccna-study-guide.html   we can examine how an address is opened in the browser.

1. Protocol - HTTP


2. Server Name - fschub.com


3. The Specific filename which is requested ccna-study-guide.html

As shown in Figure, entering the mentioned URL in the browser, the browser then checks with a name server to convert fschub.com into a numeric IP address, which it uses to connect to the server. The browser then sends a GET request to the server using HTTP and asks for the ccna-study-guide.html file. The server then sends the HTML code of this particular page to the browser. In conclusion, the browser read the HTML code and formats the page for the browser window and show it to the user. The HTML is the main standard that controls how the World Wide Web works. It covers how the Web pages are formatted and displayed at the user screen.

HTTP and HTTPS

HTTP is a request/response protocol. When a client, sends a request to a web server, the protocol which specifies the message type is HTTP. There are three common message types are GET, POST, and PUT.

GET - A host request for data, generally a webpage request

POST - Uploads data files to the web server

PUT - Uploads resources or content to the web server such as an image, video, and audio.

Hypertext transfer protocol is extraordinarily protocol but it is not secure. The HTTP send request messages to the server in plain text that can be intercepted and read anywhere in the way. The responded HTML pages are also in unencrypted and unsecured pages.

To secure communication across the internet the HTTPS protocol is used which is too secure from Hypertext transfer protocol. This protocol uses authentication and encryption to secure data traveling between the client and server. it uses the same client request-server response process as HTTP; but the data travel between client and server is encrypted with SSL (Secure Socket Layer)

Network Model - Client Server and P2P

Client-Server Network Model

In the client-server network model, the device who is requesting the information is called a client and the device who is responding to the request is called a server. The Client and server processes are working in the application layer. The client device starts the connection by requesting data from the server, the server can either accept or reject the connection. If the connection is accepted, the server establishes and maintains a connection with the client over a specific protocol.

The protocols of application layer explain the format of data exchange between clients and servers. The data exchange between server and client may also require user authentication and the identification of a data file to be transferred.

The email server is one of the best examples of client/server model which send, receive and store email. The client on a remote location issues a request to the email server for any mail to read. The server then replied by sending the requested email to the client. The data stream from the client to server is called upload and the data stream from server to client is called download. The figure below illustrates the email client/server Model

Other examples of servers are web servers, FTP server, TFTP servers and Online multiplayer gaming server. Every one of these servers provides resources to the client. Most servers have a one-to-many relationship with clients, meaning a single server can provide resources to multiple clients at one time.

Peer-to-Peer Network Model

Like Client to Server Model, the peer-to-peer network model has no dedicated server; the data is directly accessed from a peer device without the use of a server. The P2P network model has the part: P2P networks and P2P applications. Both have same features, but in practice, there are little different.

In this model, two or more hosts are connected using a network and be capable of share resources such as printers and files without having a dedicated server. Each connected end device is known as a peer. The peer can work both as a server and a client. One host might suppose the role of server for one transaction at the same time as serving as a client for another. In P2P networking model, the roles of client and server are set on a per request basis.

Peer-to-Peer(P2P) Applications

Due to P2P application devices in this model act both as a client and a server within the same communication; every client is a server and every server a client. The P2P applications need that each end device provides a user interface and run background P2P services.

Various P2P applications make use of a hybrid system where resource sharing is decentralized; but the indexes database that addresses to resource locations are stored in a centralized directory server. Each peer accesses an index server to get the location of a resource stored on another peer.

Common Peer-to-Peer (P2P) Applications

Every computer in the network running the P2P application can act both as a client and server for other computers in the network running the P2P application. Common P2P networks are following:

1. BitTorrent


2. Utorrent


3. eDonkey


4. G2


5. Bitcoin


6. Soulseek


7. eMule


8. KCeasy


9. Ares Galaxy


10. Gnutella

Gnutella protocol also used in some P2P applications, where all user shares entire files with all other users. There are many Gnutella client applications are available, as well as gtk-Gnutella, WireShare, Shareaza, and Bearshare.

A lot of P2P applications permit users to share pieces of many files with each other at the same time. Clients of this application use a small file called a torrent file to locate other users who have pieces that they need so that they can connect directly to them. This torrent also contains information about tracker computers that remain track of which users computer have what files. The torrent clients inquire for pieces from multiple users at the same time, recognized as a swarm. This technology is called BitTorrent. There are many BitTorrent clients as well as BitTorrent, uTorrent, Frostwire, and BitTorrent.

Whit the help of these P2P any type of file can be shared between users. A lot of these files are copyrighted. Usage and distribution of these file without permission from the copyright holder is against the law. Copyright violation is on offense and results in criminal charges and civil lawsuits.

 

Application Layer

The application layer is the topmost layer of the OSI Model. As shown in the figure below, the upper three layers of the OSI model (application, presentation, and session) define functions of the single TCP/IP application layer. The application layer enables the human or software to get access to the network. It serves as the source and destination of communications across data networks. The application layer applications, services, and protocols enable humans to interact with the data network in a way that is useful. The applications are computer software programs with which the user interacts and start the data transfer process at the request. The services are programs which run in the background and give the link between the application layer and the lower layers.

The Protocols give a structure of rules that make sure services running on a particular device can send and receive data from a range of different network devices. Data Packet delivery over the network should be requested by the client from a server. If there is a P2P network, the client/server affiliation is established according to which is the source device and which the destination device is at that time of establishes. The conversions are exchanged between the application layer services at both end devices in accordance with the terms of protocol to establish and utilize these relations.

  

TCP/IP Application Layer Protocols

The Application Layer contains a protocol that is usually required by end users. The HTTP (hypertext transfer protocol) is one of the widely used application protocol, which supports the delivery of web pages to end devices. The HTTP is the base for the World Wide Web. When a browser requests a web page, the protocol sends the name of the required page to the server. The server then sends the requested page to a client.

SMTP (simple mail transfer protocol), IMAP(Internet messaging access protocol), and POP (post office protocol) maintain sending and receiving email. SMB(server message block),  FTP (file transfer protocol) and TFTP(trivial file transfer protocol)  allow clients to share files. P2P applications make it easier to share media in a distributed fashion. DNS (domain name system) resolves the IP address and name address for better human understanding. Clouds are remote locations that host application and store data so that end users do not need as many local resources, and the users can effortlessly access content from a different location. The TCP/IP application protocols identify the format and control information required for many general Internet communication functions.  Both source and destination devices are used by the application layer protocols during a communication session. The application layer enables hosts to work and play over the Internet.

Presentation and Session Layer

The Presentation Layer

There is three main function of the presentation layer.

1. Translation: Before being transmitted, formatting, or presenting, data at the source device into a compatible form for reception by the destination device. All data should be changed to bit streams. This layer is responsible for interoperability between encoding methods as different computers use different encoding methods. It translates data between the formats the network requires and the format of the computer.


2. Encryption: It carries Encrypting data for transmission and decrypting data at the receiver end.


3. Compression: This layer carries out data compression in a way that can be decompressed by the destination device. The role of compression is to decrease the number of bits to be transmitted. It is important in transmitting a big file.

The presentation layer formats data for the application layer, and it sets principles for file formats. Some well-known standards for video and graphics format are QuickTime, Motion Picture Experts Group (MPEG), Portable Network Graphics (PNG) format, Graphics Interchange Format (GIF) and Joint Photographic Experts Group (JPEG).

The Session Layer

The functions of the session layer are to create and maintain session between the source and the destination applications. This layer handles the following.

1. Dialog Control: The session layer allows two hosts to start communicating each other in half-duplex or full-duplex mode.


2. Token Management: The session layer prevents two hosts from attempting the same critical operation at the same time.


3. Synchronization: The session layer allows a process to add checkpoints which are measured as synchronization points into the stream of data.

UDP (User Datagram Protocol)

UDP (User Datagram Protocol) is an optional communications protocol to Transmission Control Protocol used mostly for establishing low-latency and loss tolerating connections between applications on the Internet. Both UDP and TCP are working together with IP and are sometimes referred to as UDP/IP or TCP/IP. Both protocols send short packets of data, called datagram.

UDP(User Datagram Protocol) Low Overhead vs Reliability

UDP(User Datagram Protocol) is a protocol that provides the basic transport layer functions. It sends the packets, with much lower bandwidth overhead and latency than TCP. UDP is not a connection-oriented protocol so it does not offer the sophisticated retransmission, flow control and sequencing mechanism for lost and out of order packets. So UDP is not providing a reliability like TCP.  But this does not mean that application that uses UDP are forever unreliable and substandard. It only means that these functions are not provided by the transport layer protocol and must be implemented in a different place if required.

Because of low overhead, UDP is the best protocol for network applications in which apparent latency is critical such as gaming, voice and video communications, which can bear some data loss without badly disturbing apparent quality. Like TCP, UDP does not establish a connection before sending data, it just starts sending data whenever required.

UDP Datagram Reassembly

UDP datagrams are received to the destination using different routes and therefore these datagrams arrive in the wrong order. The UDP does not follow sequence numbers just like TCP. It has no mechanism to reorder the datagrams into their transmission order.

So, the UDP reassembles the data in the same order that it was received and forwards it to the application. If the sequence is important to the application; the application should identify the right sequence number and determine how the data should be processed.

UDP(User Datagram Protocol) Server Processes and Requests

UDP-based server applications are also assigned well-known or registered port numbers just like TCP When these applications and processes are running on a server; they accept the data matched with the assigned port number. When UDP receives a datagram destined for one of these ports; it forwards the application data to the proper application based on its port number.

UDP(User Datagram Protocol) Client Processes

Communication between client and server will be initiated by a client application requesting a server process. The UDP client process selects a port number from the range of port numbers randomly. The destination port on the server is generally the well-known or registered port number assigned to the server process.

Once the client selects the source and destination ports; this selected pair of ports is used in the header of all datagrams. For returning the data from server to the client, the destination and source port in the datagram header are reversed.

TCP Flow Control - Window Size and Acknowledgment

TCP is the protocol that guarantees a reliable communication channel over an unreliable network. When someone sends data from a host to another, packets can be lost, they can arrive at the receiving host out of order, the network can be congested or the receiver node can be overloaded. When we are sending some application data, we usually don’t need to deal with this complexity, we just write data to a socket and TCP makes sure the packets are delivered correctly to the receiver node. The TCP provides important service which is called TCP Flow Control.

The quantity of data that the destination host can receive and process reliably. TCP flow control is the service which maintains the reliability of TCP transmission by adjusting the rate of data flow between the source host and destination host for an established session. To achieve this, the TCP header includes a 16-bit field called the window size.

The figure below illustrates an example of window size and its acknowledgments. The window size is the number of bytes that the destination device of a TCP session can accept and process a single time. In this example, host B’s initial window size for the TCP session is 1,000 bytes. Starting with the first byte, byte number 1; the final byte PC A can send without receiving acknowledgments is byte 1,000. This is known as PC A’s send window. The window size is included in every TCP segment so the receiver can adjust the window size at any time depending on buffer availability.

The figure illustrates, the source is transmitting 1,500 bytes of data within each TCP segment. This is known as the MSS (Maximum Segment Size).

The primary window size is settled upon when the TCP session is established during the three-way handshake. The source host must bound the number of bytes sent to the destination host based on the destination’s window size.

Only after the source host receives an acknowledgment that the all the bytes have been received at the destination host, can it continue sending more data for the session. Usually, the destination host will not wait for all the bytes for its window size to be received before replying with an acknowledgment. As the bytes are received and processed; the destination host will send acknowledgments to inform the source host that it can continue to send additional bytes.

Usually, host B will not wait until all 4,500 bytes have been received before sending an acknowledgment. This means host A can correct its send window as it receives acknowledgments from host B. As shown in the figure below, when host A receives an acknowledgment with the acknowledgment number 3,001, host A’s send window will increment another 4,500 bytes (the size of host B’s current window size) to 7,500. host A can now continue to send up to another 4,500 bytes to host B as long as it does not send past its new send window at 7,500. The process of the destination host sending acknowledgments as it processes bytes received and the continual adjustment of the source’s send window is known as sliding windows.

If the availability of the destination’s buffer space decreases; it may reduce its window size to inform the source to reduce the number of bytes it should send without receiving an acknowledgment. The window size determines the number of bytes that can be sent before expecting an acknowledgment. The acknowledgment number is the number of the next expected byte.

TCP Flow Control - Congestion Avoidance

When congestion occurs on a network, it results in packets being discarded by the overloaded router. When packets containing TCP segments don’t reach their destination, they are left unacknowledged. By determining the rate at which TCP segments are sent but not acknowledged; the source host can suppose a certain level of network congestion.

One of the main principles for congestion control is avoidance. TCP tries to sense symbols of congestion earlier than it happens and to reduce or increase the load into the network accordingly. The option of waiting for congestion and then reacting is not as good as because once a network saturates; it does so at an exponential growth rate and decreases on the whole throughput enormously. It takes a long time for the queues to consume, and then all senders host again repeat this phase. By taking a practical congestion avoidance approach; the pipe is kept as full as possible without the threat of network saturation. The key is for the sender host to recognize the state of the network and client and to control the amount of traffic injected into the system.

Whenever there is congestion, retransmission of lost segments from the source will take place. If the retransmission is not properly controlled, the extra retransmission of the TCP segments can make the congestion even worse. Not only are new packets with TCP segments introduced into the network; but the feedback effect of the retransmitted TCP segments that were lost will also add to the congestion. To avoid and control congestion, TCP employs several congestion management mechanisms, timers, and algorithms.

If the source host determines that the TCP segments are either not being acknowledged or not acknowledged in a timely. Then it can reduce the number of bytes it sends before receiving an acknowledgment. Note that it is the source host that is decreasing the number of unacknowledged bytes it sends and not the window size determined by the destination. The figure below illustrates the TCP congestion control. The acknowledgment number is for next expected byte not for the segment.

TCP Reliability - The Segment Ordered Delivery

Transmission Control Protocol accepts data from a stream, divides it into small chunks, and adds a TCP header creating a TCP segment. The TCP segment is then encapsulated into an Internet Protocol datagram (IP datagram) and exchanged with peers.

These TCP segments possibly will arrive at their destination out of order. For the original message to be understood by the receiver; the data in these out of order segments are reassembled into the original order. Sequence numbers are assigned in the header of each segment to get this goal. The sequence number represents the first data byte of the TCP segment.

During the established session, the first sequence number (ISN) is set. This ISN represents the opening value of the bytes for this session that is transmitted to the receiving side application. As data is transmitted during the established session, the sequence number is incremented by the number of bytes that have been transmitted. This data byte tracking enables every segment to be individually identified and acknowledged. Missing segments can then be identified and then reported. The ISN is effectively a random number. This is to avoid certain types of malicious attacks. For simplicity, we will use an ISN of 1 for the examples. sequence numbers indicate how to reassemble and reorder received segments, as shown in the figure.

The receiving TCP process places the data from a segment into a receiving buffer. Segments are placed in the proper sequence order and passed to the application layer when reassembled. Any segments that arrive with sequence numbers that are out of order are seized for later processing. in that case, when the segments with the missing bytes reach the destination, these segments are processed in proper order.