Data Distribution Sub-System


The Data Distribution sub-system receives real-time interferometer data from the CDS Network Data Server (NDS) and distributes it by way of shared memory buffers to any Data Monitor Applications that may be running.

The Data Distributor is comprised of the following components:

Each component is described in detail in the following sections.

Shared Memory Buffer Manager


The Shared Memory Buffer Manager is distributed among the data producer and consumer processes. The buffer queues, semaphores, status information, etc. needed to manage the buffers are stored in the same shared memory segment as the data (in future versions, the control data may be moved to a separate partition to allow write access control of the buffers).

Buffer Allocation

The control area contains two buffer lists. A free-list contains all buffers that have no valid data and are therefore available for distribution to a producer for filling. The full-list contains all buffers with valid data that are available for (shared) distribution to the consumers. By default a buffer is allocated to a producer from the free list and added to the full list when it is distributed by the producer only after it has been read by at least one consumer. Note that if there are no consumers the buffers will fill up immediately and the producers will hang attempting to allocate a buffer. For this reason the first buffer(s) read by a consumer may be quite old. The default buffer allocation technique may be adapted to the specific needs by setting the partition allocation mode with any combination of the following modifiers.
  • RELBUF Modifier: A full buffer is released immediately to the free list if there are no pending requests from a consumer.
  • SCAVAGE Modifier: A buffer in the full list that is not in use and is not reserved for any consumer may be allocated to a producer if no buffers are available on the free list. In this case, the oldest available full buffer is used.
  • RQSYNC Modifier: A buffer is allocated to a producer only if there is a pending request for a buffer by a consumer.
  • Consumer Control

    Each full buffer has an associated trigger mask set by the producer when the buffer is filled. The consumer process may specify a mask of interesting triggers at the time the consumer registers. Only those buffers which have one or more trigger bits in common with the consumer's trigger mask will be considered for distribution to the consumer.

    The consumer interface allows the consumer to reserve any number of buffers. By default no buffers are reserved for the consumer. When this is the case, each time a consumer requests a buffer the consumer interface takes the oldest buffer in the partition's full-list. If there are no buffers in the full-list, the consumer request fails or the consumer waits for the next available buffer. When the buffer is released it is marked as having been seen by the consumer to prevent it from being reread with the next request.

    If the consumer requests that one or more buffers be reserved for it, the (shared) reservation of the buffer is performed at the time the buffer is distributed by the producer. The buffer will be reserved for all consumers that have selected the buffer by the trigger mechanism and have not filled their specified quota of reserved buffers. The reservation is canceled when the consumer releases the buffer.

    Programming Interface

    The programming interface consists of three components: The public interface for these classes and others used in the data distribution subsystem are described in more detail in the Data Distribution API Document.


    I ran some timing tests of a prototype Buffer Manager on my Sun WorkStation (a 167 MHz Ultra-1 with 83 MHz memory). The programs used were compiled with "CC -O2" . The results of these tests are summarized in the table below.
    Producer Overhead
    44 us
    Consumer Overhead
    58 us
    Process Swap
    56 us
    Write data rate
    150 MB/s
    Read data rate
    100 MB/s
    The fine print:
    1. The times tabulated here are real-time intervals not CPU-time. I assumed that this would give a more reliable result since overhead incurred in context switching may not be reflected in the CPU time. The side effect of this decision is that the time measurements are susceptible to other background processes
    2. No process swapping is required when the producer is run alone. When one or more consumers are running, the rate of process swaps/record is assumed to be (Ncons+1)/Nbuffer where Ncons is the number of consumers and Nbuffer is the number of buffers (in this case 8).
    3. Read and write data rates are probably dominated by the programmed data transfers. The point here is to demonstrate that there are no major bottlenecks in writing-to/reading-from the shared memory.

    Interface to Frame Packages

    The shared memory API has been interfaces to the two popular frame libraries: FrameCPP, the C++ library written and supported by LDAS, and FrameL, the C library from Virgo. The FrameCPP interface is by way of two classes iSMbuf and oSMbuf based on the C++ stream classes istream and ostream, respectively. These classes are also documented in the Data Distribution API Document.

    The interface to FrameL is by way of a redefined FrIO structure and the associated interface functions FrIOClose(), FrIOOpen(), FrIORead() and FrIOWrite(). These are defined in ONLFrIO.h and Note that ONLFrIO is implemented as a C++ package with a C interface in order to communicate with both FrameL and LSMP.

    The Data Pushers (DpushF, DpushM, DpushRT)

    The data pusher programs receive data from various data sources and copy them to a shared memory buffer. One or the other program must be running at all times in order to provide data to the monitor programs. Several data pushers may be run in parallel on different partitions to provide multiple data sets to different monitor groups. Care should be taken not to start more than one Dpush process on a single partition. as this might cause a monitor to receive duplicate copied of a single data frame. The three Dpush programs read data from different sources, but otherwise function in much the same manner. DpushF reads frames from the CDS Network Data Server (NDS), DpushM receive frames broadcast by the Frame Broadcaster, and DpushRT replays frames stored on spinnig media in real time. On startup, Dpush will create a shared memory partition with the specified name, length and number of buffers if the named partition does not already exist. It then copies data from the specified source to the shared memory partition. It also keeps statistics on the number of missing Frames and the average rate of arrival of the frames and prints them to a log file every 20 minutes. The syntax for the Dpush command is the following:

        Dpush [NDS <node>] [infile <file>] [partition <pname>] [lbuf <nby>] [nbuf <n>] \
            [log <logfile>] [time <ticks>] [debug]

    The arguments to Dpush are:
        <node>     Node name or IP address of the network data server (DpushF, DpushM).
        <file>     Input file name or wild-card expresion. (DpushRT only)
        <pname>    Name of shared memory partition to receive the data.
        <nby>      Minimum buffer length for the partition.
        <n>        Minimum number of buffers in the partition.
        <logfile>  Name of file to receive error and statistics messages.
        <ticks>    Number of seconds between statistics summaries.

    Entering an invalid argument causes a help message to be printed out. The Dpush executables reside in the $DMTHOME/bin directory.

    Using DMTrestart to run DpushF

    DMTrestart is a script used to run DpushF and to insure that it will be restarted if it fails. It is most useful in testbed situations since the combination of the broadcast version (DpushM) and procmgt are now the preferred running mode. It performs the following functions:
  • Make sure that a copy of DpushF is not already running.
  • Check that the NDS server is alive and responds to a version number request. If not, wait for it to start, testing in 20s intervals.
  • Generate a log file name of the form $HOME/logs/
  • Start DpushF with all the appropriate arguments.
  • Wait for the DpushF to fail/terminate.
  • When running under DMTrestart, DpushF can usually be upgraded without affecting any running monitors by replacing the DpushF executable and killing the process. It will be restarted with the loss of at most 1-2 frames, without otherwise affecting the client processes.

    Data Distribution Utility Programs

    A set of utility programs is has been written for maintaining the Shared memory partitions. These are listed below:

    smcreate - Create a shared memory partition.

    smcreate is used to create a shared memory partition. The syntax for the smcreate command is

        smcreate <partition> [-nbuf <n>] [-lbuf <length>]

    The parameters to smcreate are:

        <partition>  Name of the partition to be created.
        <n>       Number of buffers in partition.
        <length>    Length of buffers.

    smdump - Print status of a shared memory partition.

    smdump prints the current status of a shared memory partition. The syntax for the smdump command is

        smdump [<partition>]

    The parameter to smdump is:

        <partition>  Name of the partition to be dumped. If <partition> is not specified, the
                     partition name will be taken from the LIGOSMPART environment variable.

    smraw - Formatted dump of the data from a shared memory partition.

    smraw is used to produce a formatted dump of the data stream from a shared memory partition. The syntax for the smraw command is

        smraw <partition> [-nwmax <nw>]

    The parameters to smraw are:

        <partition>  Name of the partition to be dumped. If <partition> is not specified, the
                 partition name will be taken from the LIGOSMPART environment variable.
        <nw>          Number of (32-bit) words to be dumped per record, by default <nw> is 64.

    smrepair - Remove  inactive clients.

    smrepair removes inactive clients (i.e. clients for which the registered process no longer exists) from the client list of a shared memory partition. All buffer reservations for the deceased consumers are canceled and if no active consumers remain, all the buffer use counts are set to zero. In most cases this will free a hung-up partition. The syntax for the smrepair command is

        smrepair <partition>

    The parameter to smrepair is:

        <partition>  Name of the partition to be repaired. If <partition> is not specified, the
                 partition name will be taken from the LIGOSMPART environment variable.

    smstat - Print shared memory partition status parameter

    smstat prints requested parameters of the shared memory partiton. The parameters which may be interrogated are:
    buffer_tot     Total number of buffers read into partition
    freebuf          Number of buffers in the free list
    fullbuf          Number of buffers in the full list.
    last_id          Data ID of most recent buffer.
    lbuf               Maximum buffer length.
    maxcons           Maximum number of consumers.
    name               Partition name
    nbuf               Number of buffers allocates in the partition.
    ncons              Number of active consumers.
    pflags            Partition flag bits.
    semglobal       Global semaphore ID.
    shmid              Shared memory partition ID.
    use_count       Number of processes using the shared memory partition.
    usedbuf          Number of in-use buffers.
    version           LSMP version that created the partition.
    The command syntax is:
    smstat   <param1> [<param2> ...] <partition>
    Where <paramx> is a parameter name from the list, and <partition> is the name of the shared memory  partition to be interrogated.

    Last Update: July 25, 2000
    Please send comment and suggestions to: John Zweizig