Shared Memory Class

With each class I finish, I'm more convinced the ease which a class interface integrates into application designs, performs common actions and prevents common mistakes should take precedence over the internal class design. So, before I write an operating system class interface I use the underlying C facilities and other implementations of similar classes a while to recognise their weaknesses and then make up several candidate class designs. I try including each of these candidates in imaginary applications to select and hone the best of these interfaces, only then do I begin to write the class implementation.

Such was the case for this shared memory class. I needed one for a message queue in an operating system simulation. I read a few books on the subject, tried a few classes written by others but ended up writing my own. In this article, I present a this class facilitating the use of shared memory and a rudimentary integration of shared memory and the semaphores from the last issue. The flaws of contention management design will be readily apparent, I hope you'll think about them and send me your suggestions for improvement. With your help, I hope to present a more useful shared memory management scheme in the future. Your feedback will ensure that scheme will be generally useful. Without further ado, here's the shared memory class, common_memory_region.

#ifndef OS_SHAREDMEMORY_H 
#define OS_SHAREDMEMORY_H 
/***********************************************
* os_shm.h
* Will allocate a shared memory region and
* assign space for structures and variables
* listed in this header file. You must also
* indicate which processes will have access
* to the shared region, in what manner they
* will have access, and provide ptrs for
* those programs to access the data they are
* interested in.
#include <sys/types.h>
#include <sys/ipc.h>
// May need extern "C" declaration
#include <sys/shm.h>
// outside SYSV
enum cmr_perm
// Take from ipc.h and shm.h
{
  CMR_RW_ALL = (0400 | 0200),
  NEWCMR_RW_ALL = (IPC_CREAT | CMR_RW_ALL),
  RW_OWNER = (00400 | 00200),
  NEW_CMR_RW_OWNER = ( IPC_CREAT | RW_OWNER)
};
  
class common_memory_region
{
public:
  common_memory_region() ;
  virtual ~common_memory_region();
  int cmr_server(int cmr_id,
                 int size_of_block, 
                 cmr_perm access_rights); 
  void *cmr_client(int id,
                   cmr_perm access_rights);
  void *cmr_register_client(
                   cmr_perm access_rights);
  int set_cmr(int id); 
protected:
  virtual void shm_error (char *error_msg); 
private:
  int server_established; 
  int common_region_id; 
  char *client_address; 
  struct shmid_ds *cmr_information; 
  int file_handle;
}; 
#endif

Intro to Shared Memory

Shared memory is designed, as its name implies to share information via a region of memory addressable across process boundaries. While shared memory is allocated, like regular memory it is not easily extended. It is possible to create a sbrk equivalent how ever, since support for this type of operation is not built into the shared memory system call interface, extending the region necessitates either copying its entire contents to a new, larger region and adjusting external pointers or adding a second region and using a sophisticated memory manager to map storage operations to the proper region. Of the two, the second alternative seems superior and I may add such sophistication to the memory manager but it hasn't happened yet.

Design

Whenever you work with interprocess communication, a robust design becomes more important because run-time debugging will be difficult. A example of how a class interface can help prevent run-time errors, consider that the semaphore class provided with the shared memory class does not protect itself against race conditions. That is, its behaviour is undetermined if two processes attempt to become the server for a semaphore id simultaneously. The client-server design of the shared memory class attempts to prevent this problem.

To use this class, you must create a "server" and "client" for the shared memory. Only one server can delete the memory region. The server also designates the desired read write permissions clients may have. Clients, when created, can request permission levels up to that allowed by the server.

Servers cannot be removed until all the clients they serve have detached. Clients on the other hand, can switch the shared memory regions they address at run time by using the "set_cmr" method. After using set_cmr, you must call "cmr_register_client". When you cast the return of this call, you will receive a pointer of the proper type for the shared region and re-establish a shared memory connection with the proper permissions. You must, however, make sure you keep track of what shared objects are in use so you don't detach from a shared memory region holding information you are using. Please note that nothing is keeping you from storing objects of many types in a single region. Doing so just requires a scheme for hiding the shared information type when you want to use it. For simplicity's sake I either inherit shared objects from a common type or, more typically, share only a single type in a region.

If you examine the shared memory implementation, you may notice that the default constructor doesn't seem to do very much. I could have not had a default constructor, forced the user to specify the shared memory parameters and created the shared memory region with the constructor. I avoided that approach because it did not fit within the client server paradigm I had found very useful when attempting to prevent run-time errors. Including a default constructor also allows a programmer to allocate arrays of an object in a known state. I debated the use of fstream type permissions in this class but settled on a unique specified type because some of those operators don't make sense in a shared memory context. I could also have used the string parameters of the C file interface but a class specific enumeration will allow the compiler to catch the passing of some bad permissions. Clients who attempt to attach with unallowed permissions can only be detected at runtime.

I could also have constructed a "malloc" and "free" method in the class such as used in the C++ Wrappers distribution (see end of article) However, beginning with C++ 2.1, a placement operator was made available. This feature allows you to specify the address at which an object is to be "allocated". I've included such a new in this shared memory class. An equivalent delete is not required, placed objects can be treated like an other as long as you remember to address the deletion of the underlying storage in the class destructor. I chose this strategy because I wanted the interface to make the allocation of shared memory and the ability of the region to hold disparate objects obvious and explicit. Now I'll cover the lessons I've learned at the IPC School of Hard Knocks.

Subtle Issues

The details contained so far in the presentation of the class should be enough to allow you to use the class without further ado. However, as I stated there, the common_memory_region interface allows access only to the most often used shared memory facilities. The operating system provides significantly more features. This section also contains some of the subtle limitations imposed on the use of shared memory segments by the operating system and the class interface.

First, some limitations of shared memory. There are two operating system parameters, most commonly called SHMSEG_MAX and SHMAT_ MAX which may affect your use of shared segments. (The name and value of these parameters will vary from operating system to operating system. You can also count on these kernel parameters being defined in a different file location for each system. Maybe the common UNIX API will create a common system configuration system- let's hope so.) SHMSEG_MAX governs the maximum number of shared segments that can be present simultaneously. SHMAT_MAX limits the number of shared segments which a single process may attach to. On my machine (SVR4) these parameters are stored in /etc/conf/cf.d/mtune and the parameter names are SHMMNI and SHMSET. On an SGI they are stored in /usr/sysgen/master.d/shm and called SHMMNI and SHMSEG. About the only way to gather this information is to poke around the man pages of your system. I've included a utility (limits) which will examine these limits. This small utility can serve as a portability test and a method of finding the limits of your system without spending hours hacking. While both these limits are beyond the requirements of most applications, if you think you'll be bumping up against these limits on your system it's likely you'll be taxing others' systems as well. It's probably a better idea to redesign your application than require the other users to rebuild their kernel to install your software.

Forking off new processes and exec require some caution. If you fork a thread or new process, the shared segments and any associated semaphores will still be available in the new thread. Unlike forked execution paths, a process which is started using any of the exec family of calls will not be attached to the shared memory segments of its parent. This behaviour is logical but worth noting.

Classes with virtual functions may also have problems with being stored in shared memory. If such a class is to be shared, fork threads using the shared classes from the same parent and be sure to double check the validity of shared values during testing. This problem is generic, the solution may be compiler specific, depending on the scheme used to track virtual method pointers. You can see JOOP 91 for a more complete discussion of the problem. It hasn't been an issue for me so far, but fair warning.

A shared memory structure has a data structure, shmid_ds associated with int. This data structure contains among other information the current owner and user, process and group process which created the region. The system call interface allows you to change the current owner of the region. shmid_ds also contains information on the number of processes attached to the shared region . You can see this call in the class destructor where I use it to determine if the server should destroy the region or just decrement the reference count. shmid_ds also contains information on the time of the last attach and unattach actions and when the id information was last changed. I have never had occasion to use any of these facilities but feel free to add a free form method to common_memory_region for access to these and other facilities if you think you need them. If you do, please let me know as I'm interested in how they'd be useful.

Build Notes

I've built this library on a Sun using 4.1, DECs using ULTRIX 4.3 and 4.3 BSD, an SGI using 4.0.5 and UNIX/386 (SVR4) machines. It should build without modification on any UNIX machine offering network and IPC services. The library will compile with g++ or CC 3.0 using gmake or regular make. I haven't built it with GNU libc or GNU Id although there is no reason why it shouldn't compile with those tools (and thus LINUX and BSD/386) as well. If you notice non-portable aspects of the tools, have enhancement suggestions, or, heaven forbid bugs, please drop me a note and I'll get to work on the problem.

Other Implementations

No class is perfect, if you don't like this one or simply want to see how others have implemented the same functionality, here are a couple of FTP sites which carry other implementations. These may require some porting to compile on your platform. I've included each of these in the code I sent to Francis. If he has room I'm sure he'll put them on the code disk.

C++wrappers-2.4: ics.uci.edu.

SharedCoreManager: svr- ftp.eng.cam.ac.uk. I pulled the limits utility from here