LIGO Data Acquisition Handbook

    1. Table Of Contents

      1. Subsystems

      2. Reflective Memory

      3. Data Types

      4. Data Acquisition Rates

      5. Timing of A/D Inputs

      6. GDS Test Points And Injection Points

      7. Configuration Files

      8. Making Changes To DAQ System

      9. Adding DAQ Channel

      10. Removing DAQ Channel

      11. Test Point Numbers

      12. DAQ System Problems

  1. Subsystems

    DAQ system consists of several data gathering computers, connected with the reflective memory network in the token ring fashion. Each system sending a data packet into the ring is responsible for elimination of that data packet from it upon the first encounter with it after the said packet makes a full trip around the token ring network. This elimination function is implemented in hardware. Every computer system present on the reflective memory network needs to use the same model communication board. There are two types of the reflective memory hardware currently in use at LIGO. Both are extremely fast fiber communication networks, which provide a reflective memory feature. The reflective memory functionality of these communication adapter boards is unusual but essential for LIGO, as it allows us to avoid all software overhead associated with the traditional network communication cards, like Ethernet. These reflective memory ( RFM) boards present a system processor memory mapped area of several megabytes, which gets magically replicated among all other systems on the same RFM network ring. The communication board takes care of sending updates to all other systems on the ring. It provides for completely automated receiving of the data updates from all other systems and synchronizing its own RFM image with the images of all other systems on the same RFM ring.

    The list of control systems is not long. There are several digital suspension controllers (SUS), which are responsible for damping the optics using the shadow sensor inputs and in some cases optical levers. Digitized inputs are sent to the DAQ RFM and archived. There is interferometer Length Sensing and Control system (LSC), which reads demodulated photo diode data, carrying the information used by the LSC software to calculate forces applied to the optics to keep the interferometer on resonance. Many of the LSC sensors and control signals are sent to the DAQ RFM and archived. The LSC system is using the same RFM network to communicate mirror position correction signals to the SUS controllers. Another vital interferometer system is the Angle Sensing and Control (ASC) system. It reads a set of photo diode data, which carries interferometer angular misalignment information. Similarly to the LSC system, the ASC is calculating several forces applied to the mirrors required to achieve higher power buildups. The ASC is sending its own data block to the DAQ network. The Hydraulic External Preliminary Isolator (HEPI) was recently installed at LLO and it consists of several computers, all sending DAQ data to the RFM network. There are few simple data collection systems, which do not perform any control function. These systems are called ADCU or Analog Data Collection Units. There are five ADCU systems presently installed.

    Control and ADCU systems collect data at rates higher that 16 Hz. We call this data fast. There exists another set of data, which is saved at 16 Hz sampling rate, called Epics data. Epics is the data channel access system used to send control signals and read responses from the systems at slow rate. Real data exchange rate depends on the system. Sometimes it could be as low as 2 Hz. Regardless of real data rate supported by a particular system, the DAQ Epics data is collected at 16 Hz sampling rate. More often than not it is oversampled by a factor of two or three. Epics data is flowing on the regular Ethernet network and it is not synchronous, unlike the fast DAQ data. Epics data is collected by the frame builder.

    Frame builder is the UNIX system responsible for fast data collection from the RFM as well as Epics data acquisition from the Ethernet port. Frame builder is running on Sun Sparc computer and benefits from using Solaris real-time scheduling extension to guarantee on time data acquisition of synchronous fast data from the RFM network. Frame builder provides network data access to users, which run data display and analysis client programs. DTT and the DataViewer are the two major client programs. Frame builder also gives access to DMT data archives. It acquires some of the DMT Epics channels as well.

  1. Reflective Memory

    5579 and 5565 VMIC reflective memory communication networks are used at present. One could distinguish these two types of network adapters by the size of the fiber connectors. 5579 is using larger square fiber connectors the size about 1 cm. 5565 is newer product and it comes with smaller connectors, about one third of the size of 5579 connector.

    There are two types of fiber used with 5579 communication boards. Single mode and multi mode. Single mode fiber is used for communication to remote End station. Short runs of single mode fiber usually has yellow jacket. Multi mode fiber is used everywhere else and usually orange in color.

    To avoid breaking the loop when powering down on of the systems on the RFM network we use the bypass switch. These switches are located in the mass storage room (MSR). There exists Epics interface to the 5579 bypass switches and Web interface to 5565 switch. It is highly recommended to use these switches to manually bypass any system or a group of systems which require powering down or system reset.

    AWG system requires a reset to change calibration line information. It is highly recommended to manually bypass AWG on 5565 network first and then reboot by issuing VME bus software or hardware reset. If the manual bypass is not done prior to the reset, there will be data glitch introduced into the reflective memory loop which will likely cause subsequent loss of LSC control and loss of interferometer lock. Additionally the frame builders could be affected as their timing source would glitch. The frame builder might be forced to restart by resetting AWG without prior manual bypass of 5565 RFM network.

    There exists mostly unexplained phenomena, observed on both 5579 and 5565 networks, which could be described as RFM data corruptions. It is essentially an overload of an RFM network card by the software program trying to transmit too much data without delay. Internally each RFM adapter is equipped with the local data FIFO which provides for data buffering prior to transmission into the fiber loop. However, to support the token ring network, the RFM card also needs to receive the data from the loop and transmit it back into the loop. Hence the requirement to multiplex local data with the remote input data. It seems that when the FIFO is overrun, the boards starts corrupting remote data, which affect all the systems on the same RFM loop. This appears to be a hardware feature: normal behavior should be an issue of PCI bus hold to the CPU until the bottleneck has cleared. Software is avoiding sending long blocks of data by interleaving RFM writes with delays or local CPU processing, which seems to allow us to control and solve this problem.

    Aforementioned data corruptions are detected by the frame builders. They show up as CRC errors of the DAQ data. At present we do not check control signals for mismatch. Since the same RFM network is used for passing both DAQ data and the control signals, we are capable of detecting RFM error condition by just looking at the DAQ CRC error accumulators.

    There exists yet another mode of RFM failure. It is suspected that sometimes some of the RFM cards enter into a bad mode, which could be cleared only by power cycling the affected system. Historically we have seen things like that few times with different types of the RFM. If there is a suspicion of loss of RFM network connectivity or CRC data corruptions appear suddenly when no software change was performed, recommended action should be to power cycle the frame builder machines first. We have seen most of the RFM board problems on frame builder machines, Sun computers.

    In June 2005 frame builder machines were taken out of the reflective memory loops. This was possible by the feature that is provided to us by the reflective memory bypass switch. We have simply disconnected the transmitting fibers on all reflective memory cards in both frame builder machines. The bypass unit has detected the loss of the signal and bypassed the frame builder connection. However, the bypassing action is performed in such a way that the frame builders are still receiving all data from the reflective memory. All DAQ data is still arriving to the frame builder intact. The difference now is that the frame builders are not resending the data into the reflective memory any more. This provides us with more reliable system. Additionally the communication around the reflective memory network should be faster, since we have eliminated two nodes from each of the reflective memory loops. Also now we have the ability to power down the frame builder machines without worrying about causing any disturbance to the control systems.

  1. Data Types

    There are three data types used in LIGO. Short data (two byte long signed integer), floating point data (4 byte long single precision floating point), complex data (two floats). Data types are encoded with numbers in the configuration files. Short is type 1, float is type 4 and complex is type 6.

Data Type

Type Code

Short

1

Float

4

Complex

6

  1. Data Acquisition Rates

    In LIGO all data is acquired only at power of 2 rates. Minimum data rate is 16 Hz. Maximum rate is 262144 Hz (or 256K Hz). Suspension controllers, ASC and HEPI have maximum DAQ rate 2048 Hz. Fast ADCU is the only system supporting 256K sampling rate.

  1. Timing Of A/D Inputs

    Data acquisition is triggered by the external 2^22 Hz (4194304 Hz) square wave clock, which is aligned to a second boundary. To maintain high absolute timing accuracy the GPS receiver is used to discipline local oscillator. Receiver also has a one pulse per second signal (1PPS) which indicates clock signal alignment points and provides for precise data acquisition alignment to GPS time. The signals needed by the ADC input boards are derived from the GPS signals. To do this the custom GPS Clock Driver Module (CDM) board is used.

  1. GDS Test Points And Injection Points

    GDS system in LIGO provides us with a set of channel names, which are test points and injection points. Test points are the data channels coming from the control systems to the software client programs running in the control room. Injections are originated by one computer system only. It is a Arbitrary Waveform Generator (AWG) computer system. It is capable of generating various digital waveforms and sending them to the control systems to perform tests or calibrations.

    Test points are the temporary data channels, which are established on demand. They are only present for the time they are used by the client programs. Once a program is finished or the channel name is changed, the test point is no longer needed and it could be turned off by the test point manager process. The control system stops sending the data then and the test point slot in the RFM is freed and becomes available for the use by another test point. Test point manager task is running on the AWG computer. It is responsible for counting test point requests. This allows multiple clients to use the same test point channel at the same time.

    There is limited number of test point slots. Control systems running at 16384 Hz rate are using one GDS test point RFM area and all the systems running at 2048 Hz rate are using another area. 16384 Hz systems are also called collectively LSC systems. Slower 2048 Hz systems are called the ASC systems. Test point data areas are also frequently called LSC and ASC because of that. There are 15 LSC slots and 54 ASC slots available. These are the limits of how many test point it is possible to have running at the same time.

    Excitation slots are limited too. There are 7 LSC and 24 ASC excitation slots available. These are just the overall limits on injections supported by the DAQ system. There are also different limits, which are less clearly defined. One is defined by the processing power of the AWG computer. Depending on the complexity and the number of components of the generated digital waveforms, the maximum number of simultaneous injections could be lower than the combined LSC and ASC limit of 31.

    Another vaguely defined limit is the number of injection inputs that a control system could support at the same time. This is generally applicable only to the control systems running at 16384 Hz rate. Each selected test point or injection signal takes a certain fraction of the control system's processing cycle. If the cycle time is increased above 61 microseconds, the system might start loosing data and could crash. The control loop will be negatively affected too. This might show up as increased noise in the control signals and could lead to the loss of control. There is protection against this in the control system's code. It is basically the limiting of the total number of test points one could set at the same time on the control system.

  1. Configuration Files

    DAQ system is configured with INI files. They reside in chans/daq directory. The list of all input configuration files is kept in chans/daq/master file. Master file is merely a list of all INI files to read. Each line is the INI file name for frame builder to open, parse and configure. If one desires to create new configuration file, all that's required would be to create new INI file and include its full file system path file name into the master file. Positioning of INI file names in the master file is unimportant as the frame builder sorts all configured channels by name at the end of the input.

    There are two major parts in INI configuration file. Default section is the first section of the file. It is not a required section. The default section is indicated with [default] header. Following optional default section there is a number of channel configuration sections in the INI file. These sections configure on DAQ channel per single section. DAQ channel name is the section name, it appears in square brackets.

    Each INI file section could have a number of configuration parameters. Here is the list of supported parameters.

    Configuration Parameters

Parameter

Type

Values

Description

dcuid

Integer

4 through 30

Data Collection Unit number.

datarate

Integer

16 or greater, less than 262144. Must be power of 2.

Sampling rate.

gain

Float

Any floating point number.

Channel gain, can be applied in Dataviewer with Units selection.

acquire

Binary

0 or 1

0 if don't want to save data channel in frames. 1 if need to save data in frames.

ifoid

Binary

1 or 2

L1, H1, C1 are 1; H2 is 2

datatype

Integer

1, 4 or 6

Short is type 1, float is type 4 and complex is type 6.

units

String

Up to 32 characters, no quotes

Engineering units specification can be displayed in the Dataviewer with Units selection.

slope

Float

Any floating point number.

Channel slope, can be applied in Dataviewer with Units selection.

offset

Float

Any floating point number.

Channel offset, can be applied in Dataviewer with Units selection.

chnnum

Integer

Greater than 10000

Test Point Number defining channel data source

Data collection Units

DCU ID

Mnemonic

Name

4

EDCU

Epics Data Collection

5

ADCU_PEM

Physics and Environment Monitors Data Collection

6

ADCU_SUS

Suspension Data Collection

7

ADCU_EX

X-End Data Collection

8

ADCU_EY

Y-End Data Collection

9

SUS1

LVEA ITMX ITMY Suspension Controller

10

SUS2

LVEA RM BS Suspension Controller

11

SUS3

LVEA MC2 Suspension Controller

12

SUS4

LVEA FMX and FMY Suspension Controller on H2 IFO

13

LSC_EX

16K AWG Excitations

14

ASC_EX

2K AWG Excitations

15

LSC_TP

16K GDS Test Points

16

ASC_TP

2K GDS Test Points

17

LSC

Length Sensor and Controller

18

ASC

Alignment Sensor and Controller

19

SOS

Small Optics Suspension Controller

20

SUS_EX

Interferometer end-station X Suspension Controller

21

SUS_EY

Interferometer end-station Y Suspension Controller

22

SEI1

MC1, MC2, RM HEPI Controller

23

SEI2

BS, ITMX, ITMY HEPI Controller

24

SEI_EX

ETMX HEPI Controller

25

SEI_EY

ETMY HEPI Controller

26

IOO

Mode Cleaner WFS Controller

27

ADCU_FAST1

256K Data Collection, first DCU area

28

ADCU_FAST2

256K Data Collection, second DCU area

29

ADCU_FAST3

256K Data Collection, third DCU area

30

ADCU_FAST4

256K Data Collection, fourth DCU area

  1. Making Changes To DAQ System

    First step involved in making changes to the DAQ system configuration is modifying INI file or the master file. If you change any of the INI files, you will have to reload front-end DAQ configuration first. This is accomplished using Epics channels named H1:DAQ-SOS_LOAD_CONFIG, where SOS is the subsystem name. These channels are available on H0DAQ_DETAIL.adl MEDM screen, they are push buttons in the RECONFIG column on that screen. The Epics system, which loads new configuration, will recalculate CRC checksum for the file you are reloading. This CRC checksum is displayed in H1:DAQ-SOS_MSG Epics channel. These channels for all subsystems are available on H0DAQ_DETAIL.adl MEDM screen. You should see the CRC number change when you load new configuration. At this time you should also notice the change in the frame builder status for that DCU. Epics channel is H1:DAQ-FB0_SOS_STATUS and it indicates normal condition when the status is 0. After reloading new configuration file you should see this status change to 0x2000. 0x stands for hexadecimal display notation. 0x2000 status indicates the mismatch of the INI configuration file between the front-end system and the frame builder. To fix this condition you will have to restart frame builder. After frame builder is back you should see 0x0 status again. At this point your new DAQ configuration should take effect fully.

  1. Adding DAQ Channel

    DAQ system channels are configured using test point numbers. It is possible to add any test point as a new DAQ channel and record data in the archive. All channel names must be unique. Before adding new data channel, one should grep for its name in all the INI files to make sure this name is not taken already. If you add many new channels, there is strong possibility that the data archive file system would eventually run out of room. In this case the number of data directories will have to be reduced in the frame builder configuration file or more disk space added to the file system.

  1. Removing DAQ Channel

    You can save the current copy of the INI file with the date tacked to the end of the filename and then delete the channel name section for the channel you want to remove. Alternatively you could comment out that section using # in the beginning of the lines to be removed.

  1. Test Point Numbers

    Test point numbers are encoded into the front-end software programs. User cannot change test point number assignments. User can only control which test point goes into which DAQ channel. There are two types of test points. One is the test points related to the filter modules. The other test point type is the generic test point unrelated to a filter module. For each front-end system there exist ranges of test point numbers.

      Test Point Ranges

Front End Name

Filter Module Test Points

Generic Test Points

Filter Module Excitations

Generic Excitations

LSC

10000 – 10199

10200 – 10268

1 – 35

100 – 111

SUS1 (ITM)

11000 – 11200

11200 – 11220

1000 – 1400


SUS2 (RMBS)

11400 – 11600

11600 – 11640

1400 – 1799


SUS3 (MC2)

12200 – 12399

12400 – 12410

2200 – 2399


SUS4 (FM)

13400 – 13599

13600 – 13610

3400 – 3599


ETMX

13800 – 14000

14200 – 14400

3800 – 4000


ETMY

14000 – 14200

14400 – 14600

4000 – 4200







IOO

30000 – 30199

30200 – 30399

25000 – 25099


ASC

30400 – 30699

30700 – 30870

20400 – 20999


SOS

31000 – 33700

33800 – 34000

21000 – 21400







SEI1

34400 – 36000

36000 – 36399

24000 – 24400


SEI2

37200 – 38400

38400 – 38800

24500 – 24800


SEI_EX

36400 – 36800

36800 – 37200

24400 – 24500


SEI_EY

38800 – 39200

39200 – 39600

24800 – 24900







ADCU_PEM


15000 – 15200



ADCU_SUS


15200 – 15400



ADCU_EX


15400 – 15527 are four ICS110B boards, 32 channels each.

15528 – 15531 are four ICS115 excitation output channels, these correspond to excitation numbers 41000 – 41003.


41000 – 41999

ADCU_EY


15600 – 15727

15728 - 15731


42000 – 42999






  1. DAQ System Problems

    1. Limit of 128 channels per DCU currently exists in the CDS software, which implements DAQ function of a control system front end. Defining more than 128 channels has an undefined effect currently.

    2. Channel names must be unique. They are case sensitive. Frame builder will not start, if it detects any name appearing more than once.

    3. Fast data CRC mismatch between two frame builders. This could indicate a different configuration used by the two frame builders. If a change was made to the DAQ system configuration and only one frame builder was restarted to implement this change you would most likely have a different data acquired by the two frame builders. All non-epics data is checksummed and the two numbers are displayed on H0DAQ_DETAIL.adl screen.

    4. DAQ data corruptions. These are counted by the frame builders and displayed on H0DAQ_DETAIL.adl screen. Some system reboots generate data corruptions. This is considered normal in the present DAQ system. We need to make sure that the numbers are not increasing all the time, which is an indication of the problem on the reflective memory or one of the DCUs falling out of sync.