The SHMMAP procedure maps anonymous shared memory, or local disk files, into the memory address space of the currently executing IDL process. Mapped memory segments are associated with an IDL array specified by the user as part of the call to SHMMAP. The type and dimensions of the specified array determine the length of the memory segment.
The array can be of any type except pointer, object reference, or string. (Structure types are allowed as long as they do not contain any pointers, object references, or strings.) By default, the array type is single-precision floating-point; other types can be chosen by specifying the appropriate keyword.
Once such a memory segment exists, it can be tied to an actual IDL variable using the SHMVAR function, or unmapped using SHMUNMAP.
Note: Unlike most IDL functionality, incorrect use of SHMMAP can corrupt or even crash your IDL process. Proper use of these low level operating system features requires systems programming experience, and is not recommended for those without such experience. You should be familiar with the memory and file mapping features of your operating system and the terminology used to describe such features.
SHMMAP uses the facilities of the underlying operating system. Any of several alternatives may be used, as described in Types Of Memory Segments. SHMMAP uses the following rules, in the specified order, to determine which method to use:
SHMMAP [, SegmentName] [, D1, ..., D8] [, /BYTE] [, /COMPLEX] [, /DCOMPLEX] [, /DESTROY_SEGMENT] [, DIMENSION=value] [, /DOUBLE] [, FILENAME=value] [, /FLOAT] [, GET_NAME=value] [, GET_OS_HANDLE=value] [, /INTEGER] [, /L64] [, /LONG] [, OFFSET=value] [, OS_HANDLE=value] [, /PRIVATE] [, SIZE=value] [, /SYSV] [, TEMPLATE=value] [, TYPE=value] [, /UINT] [, /UL64] [, /ULONG]
A scalar string supplying the name by which IDL will refer to the shared memory segment. The SegmentName string must be constructed according to the same rules as an IDL variable name. This name is only used by IDL, and does not necessarily correspond to the name used for the shared memory segment by the underlying operating system. See the discussion of the OS_HANDLE keyword for more information on the underlying operating system name. If SegmentName is not specified, IDL will generate a unique name. The SegmentName can be obtained using the GET_NAME keyword.
The dimensions of the result. The Di arguments can be either a single array containing the dimensions or a sequence of scalar dimensions. Up to eight dimensions can be specified.
Set this keyword to specify that the memory segment should be treated as a byte array.
Set this keyword to specify that the memory segment should be treated as a complex, single-precision floating-point array.
Set this keyword to specify that the memory segment should be treated as a complex, double-precision floating-point array.
The UNIX anonymous shared memory mechanisms (Posix shm_open() and System V shmget()) create shared memory segments that are not removed from the operating system kernel until explicitly destroyed (or the system is rebooted). At any time, a client program can attach to such an existing segment, read or write to it, and then detach. This can be convenient in situations where the need for the shared memory is long lived, and programs that need it come and go. It also can create a problem, however, in that shared memory segments that are not explicitly destroyed can cause memory leaks in the operating system. Hence, it is important to properly destroy such segments when they are no longer required.
For UNIX anonymous shared memory (Posix or System V), the default behavior is for IDL to destroy any shared memory segments it created when the segments are unmapped, and not to destroy segments it did not create. The DESTROY_SEGMENT keyword is used to override this default: set DESTROY_SEGMENT to 1 (one) to indicate that IDL should destroy the segment when it is unmapped, or 0 (zero) to indicate that it should not destroy it. All such destruction occurs when the segment is unmapped (via the SHMUNMAP procedure) and not during the call to SHMMAP.
The DESTROY_SEGMENT keyword is ignored under the Windows operating system. Under UNIX, it is ignored for mapped files.
Set this keyword equal to a vector of 1 to 8 elements specifying the dimensions of the result. Setting this keyword is equivalent to specifying an array via the D argument.
Set this keyword to specify that the memory segment should be treated as a double-precision floating-point array.
By default, SHMMAP maps anonymous shared memory. Set the FILENAME keyword equal to a string containing the path name of a file to be mapped to create a memory-mapped file. A shared mapped file can serve as shared memory between unrelated processes. The primary difference between anonymous shared memory and mapped files is that mapped files require a file of the specified size to exist in the filesystem, whereas anonymous shared memory has no user-visible representation in the filesystem.
By default, files are mapped as shared, meaning that all processes that map the file will see any changes made. All changes are written back to the file by the operating system and become permanent. You must have write access to the file in order to map it as shared.
To change the default behavior, set the PRIVATE keyword. When a file is mapped privately, changes made to the file are not written back to the file by the operating system, and are not visible to any other processes. You do not need write access to a file in order to map it privately — read access is sufficient.
Note: The non-private form of file mapping corresponds to the MAP_SHARED flag to the UNIX mmap() function, or the PAGE_READWRITE to the Windows CreateFileMapping() system call.
Set this keyword to specify that the memory segment should be treated as a single-precision floating-point array.
If SegmentNameis not specified in a call to SHMMAP, IDL automatically generates a name. Set this keyword equal to a named variable that will receive the name assigned by IDL to the memory segment.
Set this keyword equal to a named variable that will receive the operating system name (or handle) for the memory segment. The meaning of the operating system handle depends on both the operating system and the type of memory segment used. See the description of the OS_HANDLE keyword for details.
Set this keyword to specify that the memory segment should be treated as an integer array.
Set this keyword to specify that the memory segment should be treated as a 64-bit integer array.
Set this keyword to specify that the memory segment should be treated as a longword integer array.
If present and non-zero, this keyword specifies an offset (in bytes) from the start of the shared memory segment or memory mapped file that will be used as the base address for the IDL array associated with the memory segment.
Note: Most computer hardware is not able to access arbitrary data types at arbitrary memory addresses. Data must be properly aligned for its type or the program will crash with an alignment error (often called a bus error) when the data is accessed. The specific rules differ between machines, but in many cases the address of a data object must be evenly divisible by the size of that object. IDL will issue an error if you specify an offset that is not valid for the array specified.
Note: The actual memory mapping primitives provided by the underlying operating system require such offsets to be integer multiples of the virtual memory pagesize (sometimes called the allocation granularity) for the system. This value is typically a power of two such as 8K or 64K. In contrast, IDL allows arbitrary offsets as long as they satisfy the alignment constraints of the data type. This is implemented by mapping the page that contains the specified offset, and then adjusting the memory address to point at the specified byte within that page. In rounding your offset request back to the nearest page boundary, IDL may map slightly more memory than your request would seem to require, but never more than a single page.
Set this keyword equal to the name (or handle) used by the underlying operating system for the memory segment. If you do not specify the OS_HANDLE keyword, SHMMAP will under some circumstances provide a default value. The specific meaning and syntax of the OS_HANDLE depends on both the operating system and the form of memory used. See the following sections for operating-system specific behavior, and Types Of Memory Segments for behavior differences based on the form of memory used.
Use the OS_HANDLE keyword to supply a string value containing the system global name of the shared memory segment. Such names are expected to start with a slash (/) character, and not to contain any other slash characters. You can think of this as mimicking the syntax for a file in the root directory of the system, although no such file is created. See your system documentation for the shm_open() system call for specific details. If you do not supply the OS_HANDLE keyword, SHMMAP will create one for you by prepending a slash character to the value given by the SegmentNameargument.
Use the OS_HANDLE keyword to supply an integer value containing the system global identifier of an existing shared memory segment to attach to the process. If you do not supply the OS_HANDLE keyword, then SHMMAP creates a new memory segment. The identifier for this segment is available via the GET_OS_HANDLE keyword.
Use the OS_HANDLE keyword to supply a global system name for the mapping object underlying the anonymous shared memory. If the OS_HANDLE keyword is not specified, SHMMAP uses the value of the SegmentName argument.
The OS_HANDLE keyword has no meaning for UNIX memory mapped files and is quietly ignored.
Use the OS_HANDLE keyword to supply a global system name for the mapping object underlying the mapped file. Use of the OS_HANDLE will ensure that every process accessing the shared file will see a coherent view of its contents, and is thus recommended for Windows memory mapped files. However, if you do not supply the OS_HANDLE handle keyword for a memory mapped file, no global name is passed to the Windows operating system, and a unique mapping object for the file will be created.
Set this keyword to specify that a private file mapping is required. In a private file mapping, any changes written to the mapped memory are visible only to the process that makes them, and such changes are not written back to the file. This keyword is ignored unless the FILENAME keyword is also present.
Mapping a file as shared requires that you have write access to the file, but a private mapping requires only read access. Use PRIVATE to map files for which you do not have write access, or when you want to ensure that the original file will not be altered by your process.
Note: Under UNIX, the private form of file mapping corresponds to the MAP_PRIVATE flag to the mmap() system call. Under Windows, the non-private form corresponds to the PAGE_WRITECOPY option to the Windows CreateFileMapping() system call. When your process alters data within a page of privately mapped memory, the operating system performs a copy on writeoperation in which the contents of that page are copied to a new memory page visible only to your process. This private memory usually comes from anonymous swap space or the system pagefile. Hence, private mapped files require more system resources than shared mappings.
It is possible for some processes to use private mappings to a given file while others use a public mapping to the same file. In such cases, the private mappings will see changes made by the public processes up until the moment the private process itself makes a change to the page. The pagesize granularity and timing issues between such processes can make such scenarios very difficult to control. Avoid combining simultaneous shared and private mappings to the same file.
Set this keyword equal to a size vector specifying the type and dimensions to be associated with the memory segment. The format of a size vector is given in the description of the SIZE function.
Under UNIX, the default form of anonymous memory is Posix shared memory, (shm_open() and shm_unlink()). Specify the SYSV keyword to use System V shared memory (shmget(), shmctl(), and shmdt()) instead. On systems where it is available, Posix shared memory is more flexible and has fewer limitations. System V shared memory is available on all UNIX implementations, and serves as an alternative when Posix memory does not exist, or when interfacing to exiting non-IDL software that uses System V shared memory. See Types Of Memory Segments for a full discussion.
Set this keyword equal to a variable of the type and dimensions to be associated with the memory segment.
Set this keyword to specify the type code for the memory segment. See the description of the SIZE function for a list of IDL type codes.
Set this keyword to specify that the memory segment should be treated as an unsigned integer array.
Set this keyword to specify that the memory segment should be treated as an unsigned longword integer array.
Set this keyword to specify that the memory segment should be treated as an unsigned 64-bit integer array.
SHMMAP is a relatively direct interface to the shared memory and file mapping primitives provided by the underlying operating system. The SHMMAP interface attempts to minimize the differences between these primitives, and for simple shared memory use, it may not be necessary to fully understand the underlying mechanisms. For most purposes, however, it is necessary to understand the operating system primitives in order to understand how to use SHMMAP properly.
In modern UNIX systems, the mmap() system call forms the primary basis for both file mapping and anonymous shared memory. The existence of System V shared memory, which is an older form of anonymous shared memory, adds some complexity to the situation.
To memory map a file under UNIX, you open the file using the open() system call, and then map it using mmap(). Once the file is mapped, you can close the file, and the mapping remains in place until explicitly unmapped, or until the process exits or calls exec() to run a different program.
If more than one process maps a file at the same time using the MAP_SHARED flag to mmap(), then those processes will be able to see each others’ changes. Hence, memory mapped files are one form of shared memory. Although the requirement for a scratch file large enough to satisfy the mapping is inconvenient, limitations in System V shared memory have led many UNIX programmers to use memory mapped files in this way.
Anonymous shared memory has traditionally been implemented via an API commonly referred to as System V IPC. The shmget() function is used to create a shared memory segment. The caller does not name the segment. Instead, the operating system assigns each such segment a unique integer ID when it is created. Once a shared memory segment exists, the shmdt() function can be used to map it into the address space of any process that knows the identifier. This segment persists in the OS kernel until it is explicitly destroyed via the shmctl() function, or until the system is rebooted. This is true even if there are no processes currently mapped to the segment. This can be convenient in situations where the need for the shared memory is long lived, and programs that need it come and go. It also can create a problem, however, since shared memory segments that are not explicitly destroyed can cause memory leaks in the operating system. Hence, it is important to properly destroy such segments when they are no longer required.
System V shared memory has been part of UNIX for a long time. It is available on all UNIX platforms, and there is a large amount of existing code that uses it. There are, however, some limitations on its utility:
Preferably, use Posix shared memory instead of System V shared memory for those platforms that support it and with applications that can use it. Under UNIX, SHMMAP defaults to Posix shared memory to implement anonymous shared memory. To use System V shared memory, you must specify the SYSV keyword. See the Examples section below for an example of using System V shared memory.
Posix shared memory is a newer alternative for anonymous shared memory. It is part of the UNIX98 standard, and although not all current UNIX systems support it, it will in time be available on all UNIX systems. Posix shared memory uses the shm_open() and ftruncate() system calls to create a memory segment that can be accessed via a file descriptor. This descriptor is then used with the mmap() system call to map the memory segment in the usual manner. The primary difference between this, and simply using mmap() on a scratch file to implement shared memory is that no scratch file is required (the disk space comes from the system swapspace). As with System V shared memory, Posix shared memory segments exist in the operating system until explicitly destroyed (using the shm_unlink() system call). Unlike System V shared memory, but like all the other forms, Posix shared memory allows the caller to supply the name of the segment. This simplifies the situation in which multiple processes want to map the same segment. One of them creates it, and the others simply map it, all of them using the same name to reference it.
Posix shared memory is the default for SHMMAP on all UNIX platforms — even those that do not yet support it. (To use System V shared memory instead, you must specify the SYSV keyword.) There are several reasons for making Posix shared memory the default for all UNIX platforms:
Under Microsoft Windows, the CreateFileMapping() system call forms the basis for shared memory as well as memory mapped files. To map a file, you open the file and then pass the handle for that file to CreateFileMapping(). To create a region of anonymous mapped memory instead of a mapped file, you pass a special file handle (0xffffffff) to CreateFileMapping(). In this case, the disk space used to back the shared memory is taken from the system pagefile. CreateFileMapping() accepts an optional parameter (lpname), which if present, is used to give the resulting memory mapping object a system global name. If you specify such a name, and a mapping object with that name already exists, you will receive a handle to the existing mapping object. Otherwise, CreateFileMapping() creates a new mapping object for the file. Hence, to create anonymous (no file) shared memory between unrelated processes, IDL calls CreateFileMapping() with the special 0xffffffff file handle, and specifies a global name for it.
A global name (supplied via the OS_HANDLE keyword) is the only name by which an anonymous shared memory segment can be referenced within the system. Global names are not required for memory mapped files, because each process can create a separate mapping object and use it to refer to the same file. Although this does allow the unrelated processes to see each others’ changes, their views of the file will not be coherent(that is, identical). With coherent access, all processes see exactly the same memory at exactly the same time because they are all mapping the same physical page of memory. To get coherent access to a memory mapped file, every process should specify the OS_HANDLE keyword to ensure that they use the same mapping object. Coherence is only an issue when the contents of the file are altered; when using read-only access to a mapped file, you need not be concerned with this issue.
The Windows operating system automatically destroys a mapping object when the last process with an open handle to it closes that handle. Destruction of the mapping object may be the result of an explicit call to CloseHandle(), or may involve an implicit close that happens when the process exits. This differs from the UNIX behavior for anonymous shared memory, and consequently the benefits and disadvantages are reversed. The advantage is that it is not possible to forget to destroy a mapping object, and end up with the operating system holding memory that is no longer useful, but which cannot be freed. On the other hand, you must ensure that at least one open handle to the object is open at all times, or the system might free an object that you intended to use again.
Note: Under Windows, when attaching to an existing memory object by providing the global segment name, IDL is not able to verify that the memory segment returned by the operating system is large enough to satisfy the IDL array specified to SHMMAP for its type and size. If the segment is not large enough, the IDL program will crash with an illegal memory access exception when it attempts to access memory addresses beyond the end of the segment. Hence, the IDL user must ensure that such pre-existing memory segments are long enough for the specified IDL array.
You can see a list of all current memory segments created with SHMMAP by issuing the statement
HELP,/SHARED_MEMORY
To access a current segment, it must be tied to an IDL variable using the SHMVAR function. IDL maintains a reference count of the number of variables currently accessing each memory segment, and does not allow a memory segment to be removed from the IDL process as long as variables that reference it still exist.
SHMMAP will not allow you to create a new memory segment with the same SegmentNameas an existing segment. You should therefore be careful to pick unique segment names. One way to ensure that segment names are unique is to not provide the SegmentName argument when calling SHMMAP. In this case, SHMMAP will automatically choose a unique name, which can be obtained using the GET_NAME keyword.
The SHMUNMAP procedure is used to remove a memory segment from the IDL session. In addition, it may remove the memory segment from the system. (Whether the memory segment is removed from the system depends on the type of segment, and on the arguments used with SHMMAP when the segment was initially attached.) If no variables from the current IDL session are accessing the segment (that is, if the IDL-maintained reference count is 0), the segment is removed immediately. If variables in the current IDL session are still referencing the segment, the segment is marked for removal when the last such variable drops its reference. Once SHMUNMAP is called on a memory segment, no additional calls to SHMVAR are allowed for it within the current IDL session; this means that a segment marked by SHMUNMAP as UnmapPending cannot be used for new variables within the current IDL session.
Note: IDL has no way to determine whether a process other than itself is accessing a shared memory segment. As a result, it is possible for IDL to destroy a memory segment that is in use by another process. The specific details depend on the type of memory segment, and the options used with SHMMAP when the segment was loaded. When creating applications that use shared memory, you should ensure that all applications that use the segment (be they instances of IDL or any other application) communicate regarding their use of the shared memory and act in a manner that avoids this pitfall.
Create a shared memory segment of 1000000 double-precision data elements, and then fill it with a DINDGEN ramp:
SHMMAP, 'MYSEG', /DOUBLE, 1000000
z = SHMVAR('MYSEG')
z[0] = DINDGEN(1000000)
Note: When using shared memory, using the explicit subscript of the variable (z, in this case) maintains the variable’s connection with the shared memory segment. When not using shared memory, assignment without subscripting is more efficient and is recommended.
Create the same shared memory segment as the previous example, but let IDL choose the segment name:
SHMMAP, /DOUBLE, DIMENSION=[1000000], GET_NAME=segname
z = SHMVAR(segname)
z[0] = DINDGEN(1000000)
Create the same shared memory segment as the previous example, but use a temporary file, mapped into IDL’s address space, instead of anonymous shared memory. The file needs to be the correct length for the data we will be mapping onto it. We satisfy this need while simultaneously initializing it with the DINDGEN vector by writing the vector to the file. The use of the OS_HANDLE keyword improves performance and correctness under Windows while being quietly ignored under UNIX:
filename = FILEPATH('idl_scratch', /TMP)
OPENW, unit, filename, /GET_LUN
WRITEU, unit, DINDGEN(1000000)
FREE_LUN, unit
SHMMAP, /DOUBLE, DIMENSION=[1000000], GET_NAME=segname, $
FILENAME=filename, OS_HANDLE='idl_scratch'
z = SHMVAR(segname)
Create an anonymous shared memory segment using UNIX System V shared memory. Use of System V shared memory differs from the other methods in two ways:
In this example, we will use the type and size of the existing myvar variable to determine the size of the memory:
SHMMAP, TEMPLATE=myvar, GET_NAME=segname, /SYSV, $
GET_OS_HANDLE=oshandle
In this case, the SYSV keyword forces the use of System V shared memory. The absence of the OS_HANDLE keyword tells SHMMAP to create the segment, instead of simply mapping an existing one. In a different IDL session running on the same machine, if you knew the proper OS_HANDLE value for this segment, you could attach to the segment created above as follows:
SHMMAP, TEMPLATE=myvar, GET_NAME=segname, /SYSV, $
OS_HANDLE=oshandle
In this case, the OS_HANDLE keyword tells SHMMAP the identifier of the memory segment, causing it to attach to the existing segment instead of creating a new one.
|
5.6 |
Introduced |
|
6.0 |
Added FILENAME and PRIVATE keywords |