Design and Architecture - Multiplexing

(Page 2 of 7 )

The first strategy for handling multiple connections that we’ll discuss is multiplexing. Multiplexing is a way of handling multiple clients in a single server process. The application allows clients to connect to the server and adds them to a watch list. This watch list is just an array of socket descriptors. Then the operating system tells the application which clients (if any) need to be serviced or if a new client has established a connection.

As an example, think of a restaurant with only one waiter. The waiter is responsible for attending to all the tables at the same time. As customers come in and are seated, the waiter adds them to a mental list of tables to check on. Then, when a table needs attention, he attends to it. Of course, only one table may be serviced at a time, and the possibility exists of a single table using up all the waiter’s time.

The select() Function

select()is a system function that allows us to specify a set of descriptors (sockets, in this case) that we are interested in. It is worth noting thatselect()works with any descriptor, including files, pipes, FIFOs, etc. The system puts our program to sleep, polls the sockets for activity, and wakes the program when an event occurs at one of the sockets. This keeps us from writing a busy loop and wasting clock cycles. Theselect()function prototype looks like this:

#include <sys/select.h>
int select(int n, fd_set *readfds, fd_set *writefds,
          fd_set *exceptfds, struct timeval *timeout);

The first parameter specifies the highest numbered descriptor (plus 1) to watch in the three sets. It is important to remember that you must add 1 to the highest numbered descriptor in the sets. The reason is that the watch lists are linear arrays of bit values, with 1 bit for every available descriptor in the system. What we are really passing to the function is the number of descriptors in the array that it needs to copy. Since descriptors start at 0, the number we pass is the largest descriptor number plus 1.

Next, we provide three descriptor sets. The first set contains descriptors to be watched for read events, the second for write events, and the third for exceptions or error events. Finally, we provide atimevalthat specifies a timeout. If no event occurs in any of the sets before the timeout, thenselect()returns a 0. We can also specify a null pointer for thetimeoutparameter. In this case, the call will not return until an event happens on one of the watched descriptors. Otherwise, it returns the number of descriptors in the three sets.

It is important to note thatselect()does modify the descriptor sets that are passed to it. Upon return, the sets will contain only those descriptors that had some activity. To call select multiple times, we must retain a copy of the original sets. Other than a socket error, if any error occurs, then –1 is returned.

Four macros are provided to help deal with the descriptor sets. They areFD_CLR,FD_ISSET,FD_SET, andFD_ZERO. Each takes a pointer to a variable typefd_set. Except forFD_ZERO, each takes a descriptor as well. It is important to note that the behavior of these macros is undefined if you pass in a descriptor that is less than zero or greater than FD_SETSIZE. The macros are prototyped as follows:

1. void FD_SET(int fd, fd_set *set):FD_SETflags a descriptor to be watched.
   
2. void FD_CLR(int fd, fd_set *set):FD_CLRresets the flag set to a descriptor.
   
3. int FD_ISSET(int fd, fd_set *set): Afterselect()returns,FD_ISSETdetermines whether a descriptor is flagged or not.
   
4. void FD_ZERO(fd_set *set):FD_ZEROclears the set so that no descriptors are watched.
   

A flagged descriptor indicates activity at the socket.

Here is a code fragment example of usingselect():

  int sd; /* our socket descriptor */
  fd_set sockreadset;
  FD_ZERO(&sockreadset);
  FD_SET(sd, &sockreadset);
  select(FD_SETSIZE, sockreadset, NULL,  
NULL, NULL);
  if (FD_ISSET(sockreadset))
     printf("Socket ready for read.\n");

In this example, the program will wait indefinitely for a read event to occur on the descriptor whose value is specified insd.

A Multiplexing Server

In our example, the server usesselect()to listen for new connections, check for client disconnects, and read events on existing connections. If a read event occurs on the server’s listening socket, then a new connection is initiated and the server callsaccept()to get the new socket descriptor. The new descriptor is then added to the server’s watch set.

On the other hand, if a read event occurs on another socket, then the server callsrecvto retrieve any data sent by the client. If no data is received, then the client has disconnected, and the server removes the respective descriptor from the watch set. Otherwise, the data is read and echoed back to the client. Figure 5-1 shows the basic architecture of a multiplexing server.

Figure 5-1.  Basic architecture of a multiplexing server

Here is the program (server1.c) to implement the preceding example:

/* server1.c */
#include <stdio.h>
#include <sys/ioctl.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
int main(int argc, char *argv[])

Next, we set up the variables that we’ll need. Asselect()modifies the set passed to it, we use two variables: one to maintain our state and another to interact with theselect()function. We need to keep the master set separately:

{
    struct sockaddr_in sAddr;
    fd_set readset, testset;
    int listensock;
    int newsock;
    char buffer[25];
    int result;
    int nread;
    int x;
    int val;

Then we create the listening socket. This is the socket that will listen for incoming connections from the clients.

  listensock = socket(AF_INET, SOCK_STREAM, 
  IPPROTO_TCP);

Afterward, we set the socket optionSO_REUSEADDR. While debugging, you’ll be starting and stopping your server often. Linux tends to keep the address and port that was used by your program reserved. This option allows you to avoid the dreaded “address in use” error.

      val = 1;
      result = setsockopt(listensock, 
      SOL_SOCKET, SO_REUSEADDR, &val,
sizeof(val));
      if (result < 0) {
               perror("server1");
               return 0;
      }

Here, we bind the socket to the listening port. We use the special addressINADDR_ANYto specify that we’ll listen on all IP addresses associated with the server:

  sAddr.sin_family = AF_INET;
  sAddr.sin_port = htons(1972); 
  sAddr.sin_addr.s_addr = INADDR_ANY;
  result = bind(listensock, (struct sockaddr
*) &sAddr, sizeof(sAddr));
  if (result < 0) {
    perror("server1");
    return 0;
  }

We put the socket into “listen” mode so that we can accept incoming connections:

  result = listen(listensock, 5);
  if (result < 0) {
    perror("server1");
    return 0;
  }

We initialize our descriptor set usingFD_ZERO. Then we add the listening socket to the set so that the system will notify us when a client wishes to connect. Connection requests are treated as read events on the listening socket:

  FD_ZERO(&readset);
  FD_SET(listensock, &readset);

Notice that we assign our descriptor set to an alternate variable to be passed to theselect()function. As noted previously, this is becauseselect()will alter the set we pass, so that upon return, only those sockets with activity are flagged in the set. Our call toselect()signifies that we are interested only in read events. In a real-world application, we would need to be concerned with errors and possibly write events. We loop through the entire set of descriptors.FD_SETSIZEis a constant set in the kernel and is usually 1024. A more efficient server implementation would keep track of the highest numbered descriptor and not loop through the entire set.FD_ISSETis used to determine if the descriptor is flagged as having activity. It returns a nonzero value if the supplied descriptor is set as having had activity; otherwise, it returns 0.

  while (1){
    testset = readset;
    result = select(FD_SETSIZE, &testset, 
NULL, NULL, NULL);
    if (result < 1) {
      perror("server1");
      return 0;
    }
    for (x = 0; x < FD_SETSIZE; x++) {
      if (FD_ISSET(x, &testset)) {

If the activity is on the listening socket, then we accept the new connection and add its socket to our watch set. Otherwise, we read characters from the client. If the number of characters read is less than or equal to zero, then the client is assumed to have closed the connection. We close the connection on our side and remove the descriptor from our watch list. Otherwise, we echo the characters to the screen and back to the client.

        if (x == listensock){
          newsock = accept(listensock, NULL,NULL);
          FD_SET(newsock, &readset);
        } else {
          nread = recv(x, buffer, 25, 0);
          if (nread <= 0) {
            close(x);
            FD_CLR(x, &readset);
            printf("client on descriptor #%i disconnected\n", x);
          } else {
            buffer[nread] = '\0';
            printf("%s\n", buffer);
            send(x, buffer, nread, 0);
          }
        }
      }
    }
  }
}

The server can be compiled with a command similar to the example client. Figure 5-2 shows a sample of the output obtained on executing the program.

Figure 5-2.  Output from a multiplexing server

Notice that, for a brief time, all five clients are connected at the same time.

Next: Forking >>
 

More Administration Articles
More By Apress Publishing

Comments On: Design and Architecture

· This article is an excerpt from the book "Definitive Guide to Linux Network...   >>> Post your comment now!

Buy this book...

This book is excerpted from chapter five of the book The Definitive Guide to Linux Networking Programming, written by Keir Davis et. al. (Apress, 2004; ISBN: 1590593227). Check it out today at your favorite bookstore. Buy this book now.