LIGO Weekly Report for the week of 3 October - 7 October 2016
Past Weekly Reports
The LIGO Executive Committee Agenda for Monday 10 October 2016 will be:
(Meeting time: 10:30 Pacific Time/12:30 Central Time/13:30 Eastern Time)
- LSC - Gonzalez
- Focus Topic
- Commissioning - Fritschel
- Operations Management Team (OMT) - Coyne
- Hanford - Landry
- Livingston - Giaime
- India - Raab
Review, assignment of actions, if any (Action item list -- restricted access )
- Administration - Hansen
- Engineering Optical and Mechanical - Coyne
- Control and Data Systems - Bork
- Lab Computing - Anderson
- Instrument Science and 40M - Gustafson
- Data Analysis and Astrophysics Lab-wide Activities
- Weinstein, Katsavounidis
- MIT - Fritschel
LIGO gives NSF Director and MPS Assistant Director for MPS a plaque acknowledging the Foundation's long-term commitment and vision in support of LIGO
Click on image for hi-res photo
L to R: Dr. Mark Coles (NSF LIGO Program Officer), Albert Lazzarini, Dr. France Córdova (NSF Director),
Dr. Fleming Crim, (NSF Assistant Director for MPS) and Dave reitze
LIGO-India milestone: monument markers erected at L-I site
Click on image for hi-res photo
Coverage in Indian media: New Delhi TV, Indian Express and The Hindu
L1 Interferometer reaches Binary Neutron Star range of 90Mpc
Click on image for higher resolution version
See LIGO Livingston Commisioning below for details
LIGO Laboratory Administration
Reports & Editing
- Editing FY16 NSF Annual Report
- Assisting Amber Henry with update/redesign of LIGO trifold brochures
LIGO Laboratory Safety
I held a General Safety Awareness Orientation for the new LLO Administrator Assistant.
In discussion with LHO and LLO Facility Managers on equipment operator training and equipment inspections. Both facilities are in the process of updating refresher training.
In the process of collecting safety related info for the Q4-FY2016 Safety Performance Report.
Reviewed and signed 3 LIGO Caltech Laser SOP’s this week.
The LIGO Lab September 2016 Safety Message “Housekeeping for a Safe Workplace” is available for your reading at the following URL:
Now is a good time to apply the following housekeeping task before LIGO Lab enters into the Observation–2 Science Run.
Report “Good Catches” to your supervisor or at the following URL: https://safety.ligo.caltech.edu/index.html. When we report a “Good Catch”, we give ourselves the greatest opportunity to correct unsafe conditions, equipment or even risky work behavior before an incident occurs
- Reviewing security scanner findings and following up with individual groups.
- Investigating use of ELK-stack for log/event analysis and related incident management tools.
- Loading up new Puppet management server for security infrastructure.
- Follow-up with myLIGO group regarding passwords.
- Mailing out a couple X.509 certificate expiration notices.
- Re-open discussion on daily invasive scanning at sites (tentatively enabled next week).
- Weekly patches checked/applied to servers.
LIGO Hanford Observatory (LHO) and Interferometer Operations
- G. Moreno, C. Romel, K. Ryan, J. Worden - Vacuum
- finished preparing EX RGA for a 5-day bake when IFO allows
- procurement of new materials for GV shroud prototype
- bake oven support
- TCSy chiller support
- rigging/crane operation training (Chandra)
- parts processing for roughing pumps EH2600/EDP200 beam tube connections
- B. Gateley, Facilities
- This week I adjusted the temperature in several offices at the LSB. I also turned the setpoint down on the LSB chiller.
- I adjusted the temperature in the LVEA and in the VEA @ EY by increasing the heat.
- Repaired a small leak on the D I water system located at the LSB Bonding Lab.
- Craned scissor lift out of the corner it was trapped in by the 3 IFO LTS Containers. This lift was also part of a training/certification for the LHO vacuum engineer. Chandra Romel did most of the rigging and operated the crane and did an excellent job on both. She is now certified to operate the cranes.
- After the crane lift, I used the scissor lift to remove one of the cleanroom socks that need laundered, I will need to move the articulating boom lift to access the remaining socks.
- I did a temporary fix on the sink faucet in the unisex bathroom at the OSB, the permanent fix parts are on order and should be here very soon.
- Replaced the drain plumbing on the sink in the mechanical repair shop ( Kyles Lair).
- Replaced outside light on staging building.
- 3 IFO weekly walk through.
- Daily system checks.
- HEPI Pressure Sensor Studies
- Fix excess noise on BSC CPSs–LHO
- Fix EndY STS remote centering w/FClara-LHO
- HEPI Pringle AC Loop study-SEI
- model reviews for cleanup and enhancement updates
- ISI configuration & loop shaping studies-JimW
- Normal SEI weekly maintenance.
- Took weekly H1 ETM charge measurements.
- Spend lots of time on TCSY chiller issue which developed a
mystery/virtual water leak after the flushing exercise earlier
last week. We still have found no actual water leak but have
continued to add small amounts of water daily to keep the
indicator level up.
- Rushed through fabrication of an SRM black glass baffle in the
event LLO needed it for last weeks 1-day vent to fix a scatter
beam problem. Baffle was cut at a local shop from iLIGO baffle
- Started kitting SR Baffle and ROC Actuator hardware that came
out of clean and bake.
- Worked through more 3IFO inventory issues starting to look at
new request for storage size and weight parameters. Had LHO
OPS/Det Eng meeting with management regarding launching
subsystem team members to assist with data mining.
- Worked with Team PSL to understand spares and new “T&T
lab” and need to reorder.
- Working through how to order/stock and pay for FTIR kits for
both LLO and LHO.
- Data Storage reassessment, T1600432
- Cleanup of MEDM Screens for SUS Monitor signals LHO alog 30238
- Updating QUAD control design diagram for L3 stage given new
ESD drivers, T1100378
- Charge Measurements LHO alog 30187
- Guardian DebuggingLHO alog 30222
- Power-glitch recoveryLHO alogs 30099, 30093
- Multiple laser trips over the weekend.
- All were associated with the power meter water circuit. Due
to frequency of trips it was decided on Sunday morning to
leave the laser off until the full PSL team could look at it
- Laser was restarted on Tuesday (LHO alog30217)
after diode chiller work (see next major bullet point). Has
currently operated for more than 48 hours without a trip.
- Initial investigations into the signals directly from the
flow sensors show the sensors functioning as expected, no
signs of glitching or other obvious bad behavior that would
lead to an interlock trip. See LHO alog 30198
for full details.
- We have a plan to isolate the power meter and front end
water circuit flow sensors so we can get an idea if the trips
are due to the flow sensors or some other problem with the
signal chain. We will implement this the next time the laser
trips (since it is up and working, we are leaving it alone for
ongoing commissioning work).
- Front panel controller for the PSL diode chiller quit working
properly. See LHO alogs 30193
- A new front panel is on the way.
- After receiving confirmation from the manufacturer, as a
temporary measure to get the diode chiller up and running we
took the controller from the backup chiller and installed it
in the in-service chiller.
- When the new controller arrives we will install this in the
in-service chiller, put the temporary controller back into the
backup chiller, and ship the broken controller back to the
manufacturer (at their request) for postmortem diagnostics.
- With P. King: Hunting down a spare ISS photodiode. It has
been determined we do not have one, so we are looking into
having new ones fabricated.
- Continued work stabilizing laser SN 189–1.
Electronics, Control and Data Systems Operations
J. Batch, F. Clara, C. Perez, M. Pirello, Ed Merilh
- Looking at software associated with the Diagnostic breadboard shutter switch. It seems to be working so will work on hardware Tuesday.
- Changed the binary I/O card on SEI EY. One channel was not working.
- Reseated the AA interface board and replaced two cables on h1susauxex to fix a problems with the differential signal.
- After testing the new High Voltage supplies for the ESD tried to install them on the ITM PI system. One of the supplies failed on installation. We will be shipping it back to the vender.
- Pulled some TEC boards from Mid Y to ship to MIT for the squeezing group..
Control Room Operators
OPERATOR WEEKLY OVERVIEW
Last week we started 24/7 coverage for a pre-ER10 epoch. Due to PSL issues, the OWL/graveyard shifts were pulled back until the PSL was fixed. Once the PSL reliability is assessed to be better, we will return to having OWL shifts.
Operator Guardian training session given by TJ & Jaimie on 10/5.
OPERATOR OFF-SHIFT WORK
- Quarterly Zero Count and Flow measurements on dust monitors
- Bubba started removing the BSC cleanroom socks and filters for cleaning
- Replace torn curtains on PSL garb room
- Install last section of the cable walkover
- Start work on replacement PSL chiller manifold
- Reviewed LVEA Transition Checklist for Peter K.
- Initial work on O2 Call List.
- Outreach work: Preparing for LIGO at SACNAS conference next week. Humboldt, The Magazine of Humboldt State University, (my alma mater), posted a short article about me.
- Sled swapped and replacement, and HWS code resumed (alog30257)
- Worked with Dave and Aidan trying to troubleshoot the new HWS python code
- Continued Finesse simulation work.
- Troubleshoot TCS guardian
- Continue working with Kai on the 3rd movie.
- Operator Wiki documentation work
- Ops troubleshooting assistance
- Modified ITM LV Low Noise Drivers
- Noise characterizations of new Mid-Eastern Industries High Voltage Power Supplies for the ETM ESDs.
- Re-terminate bad pin on DB37 binary connector for the AS-C QPD whitening Chassis. (WP6198)
- Wiring for Safety Interlock Beckhoff controllers
- Added DTT launch button for Bounce/Roll mode damping to existing Bounce/Roll medm screen
- Converted the PortProtection library and h1ecatc1 from TwinCAT2 to TwinCAT3.
- Testing and updating Conlog installation instructions.
- Started background reading on TwinSAFE.
- ISS2 qpd not well centered after IM2 and IM3 alignment change (alog30021).
- Beam position changes at PRM during O1: pitch due to IMC, yaw due to IM1–3 (alog30055).
- ISS Second Loop QPD orientation of pitch and yaw signals now corrected (alog30107, follow-up to #29542).
- PMC waist to IM1 - distances in meters - focal lengths and ROCs in the attached file (alog30116).
- Distance from PSL upper periscope mirror to Input Viewport (alog30196, comment to #30126).
- Both TCS tables checked for leaks and none found (alog30173, comment to #30170).
- MC2 misalign value too small to prevent IMC flashing (alog30236).
- IM alignment slider values saved in SDF - alignment check on IM1–3 back in DIAG_MAIN (alog30246).
Daniel Sigg (Keita Kawabe)
We identified the broad structure between 500 Hz and 1 kHz as caused by jitter from the HPO. The coupling is not understood and is not looking like traditional jitter coupling through misaligend interferometer optics.
Below 30 Hz we have excess noise in the auxiliary loops which was introduced this spring. Cause unknown. However, it limits the performance of the darm subtraction paths.
Education and Public Outreach
Hosted joint virtual visit with ATLAS experiment for Peachtree Charter MS in Atlanta GA
LIGO Livingston Observatory (LLO) and Interferometer Operations
Short turn around vent and incursion into BSC1 for adjustment of CPy pitch
Following finding by commissioners that the recently installed and new CPy was installed significantly out of alignment in pitch it was decided to conducted a short turn around vent and incursion into BSC1 to adjust the CPy pitch back into range. The work was completed in one day and chamber pump down proceeded over the weekend and Monday.
The gate valves were opened up to the main volume on Tuesday evening and commissioning of IFO recommenced on Wednesday. This hard work and excellent team work was rewarded by an increase in sensitivity in BNS range approach ~90 Mpc. See Valera’s report for details.
Many thanks for to all of the staff and visitors that have worked so hard to achieve this milestone.
Operations is preparing for ER10 which was scheduled to commence at the end of September but this week was pushed back until end of October to allow for more commissioning time.
From Monday 12th Sept LLO commenced on-duty operators both day shift and evening shift. Gary is planning to commence full day, evening and night operator shift schedule starting Oct 17th 2016 to be ready for ER10 and O2.
Second Round of Commissioning Post O1
Now that the Planned Engineering work has been completed the second round of Post O1 Commissioning and locking effort has begun(08/01/2016). Access to the VEAs is now restricted and requires pre-approval.
For commissioning progress see Valera’s report below
Observatory Computerized Maintenance Management.
FRS - Corrective Maintenance
Link to alog summary of Faults in FRS for this week: Closed
Link to alog summary of Service Requests in FRS for this week: Closed
Key Performance Indicator dashboard of issues tracked in the FRS.
Work Permits opened and or executed this week.
Link to alog of Work Permit Summary is here:
FAMIS- Preventive Maintenance
Operations staff executed a smooth Tuesday maintenance period. Note LLO has time shifted the maintenance period the back by one hour. From now on it is scheduled from 9:00 am until 1:00 pm (local time) instead of 8:00 am until 12:00 pm.
Preventive Maintenance activities that were scheduled for execution during the month.
The interferometer locking restarted on Tuesday after the vertex vent to correct the ITMY compensation plate pitch. The first order ghost beams are no longer present on the SRM suspension cage. There is a higher order ghost beam which disappears from the cage when the CPY is yawed into the nominal position. As result of the past few weeks scattered light mitigation (the CPY ghost beams realignment but also removal from the vacuum chamber and shut down of the unused equipment that produced vibration) the DARM noise has improved but we are still witnessing a few non-stationary noise peaks at 65, 85, and 135 Hz that vary on the time scale of several hours and modulate the BNS inspiral between 80 and 90 Mpc.
The DARM actuation above 20 Hz was transferred from the nominal electrostatic actuator to the photon calibrator while ITMX/Y and ETMX ESDs were shorted and grounded at the vacuum chamber feedthrough to look for the electrical pick up noise from the ESD electronics and cabling - we will also attempt to short the ETMY ESD but it will have to be in low noise lock on Pcal. During this test we rediscovered the 18 bit DAC zero crossing noise on the ETM ESD actuator. The DAC was last calibrated two months ago (recalibration requires model restart) and we still have not implemented the periodic checking of the DAC zero crossing glitches which we plan to do weekly and restart the models when calibration drifts outside of the limit. As a stop gap we are offsetting the control signals from zero after the interferometer is locked (this already done on PRC and SRC optics). We will have to switch back and forth between the ESD/Pcal to evaluate if there are other noise associated with the ESD electronics/cabling.
When the coupling from the non-stationary noise peaks is at its minimum the DARM is mostly limited by the quantum (shot) noise down to 80 Hz. We performed the cross correlation measurement between the two antisymmetric port photodetectors and obtained the new limit on coating thermal noise of 1.21m/rtHz at 100 Hz. This is only 20% above the design value.
L1 Detection Coordination
This week Jess McIver visited to provide training for LSC Fellows in
advance of O2. It was a very successful visit.
This week we welcomed two new LSC Fellows: Chris Billman from UF and
Karan Jani from Georgia Tech.
Week of 10/3
Worked on modelling the scattering upconversions seen in DARM. Made
models that could explain the bumps seen around 59 Hz and its harmonics.
Compared the results with other existing models.
Worked on identifying the presence on low magnitude earthquakes in
seismometer channels. Compared Earthquake GPS (USGS list) against
seismometer BLRMS channels. Planning to extend the code to study effects
from wind and its impact on locking the interferometer.
Improved Sesimon lockloss prediction model by optimizing the
corresponding machine learning algorithm.
Week of 10/3
This week I did my laser and containment training with Danny. I filled
out my form with Melanie to get the Caltech account setup, but we’re
still waiting on it to come through so I haven’t completed the online
safety training yet. i
We went through LSC Fellow training with Jess, learning out to use MEDM,
the summary pages, LigoDV and LigoDVWeb. We
joined the Detchar call Thursday, and went over the responsibilities of
an LSC Fellow during observational runs. Using LigoDV and the summary
pages, we investigated the loss of lock on Thursday at 1430. We believe
that we found the opening of the roll-up door on a microphone and
seismometer, but it occurred after the lock was lost. We found a 120 Hz
signal in a microphone at the vertex that coincided with the loss of
lock, but could not find a signal at the same frequency in other
On Monday, I met briefly with Anamaria about
working on PEM and continuing Filipe Da Silva’s regression project.
Scott Mccormick (Richard Oram)
Reporting for the Vacuum Group,
Harry Overmier and Scott McCormick
For Post O1 Vent Progress see https://dcc.ligo.org/E1600023
- Main efforts beginning this week were directed towards the successful Vertex vent for CP alignment.
- Continued supporting commissioning activities with noise hunting related to vacuum equipment.
- Replaced batteries in the Seiko turbo pump controllers for the corner station.
- Replaced LGV3 annulus ion pump due to an intermittent internal short.
- Performed maintenance on the corner station dry air system.
- Re-located the instrument air regulators for LGV5&6 for improved access.
- Continuing with housekeeping and organization activities site wide.
- Procurement activities carried out for vacuum equipment and services.
- Performed various maintenance activities related to site wide vacuum equipment.
Janeen Romie (Richard Oram)
Reporting for the LLO Detector Engineering Group: Stuart Aston, Matt Heintze, Adam Mullavey, Arnaud Pele
Performed successful one-day turnaround BSC1 chamber entry/exit to adjust pitch of CPy. See alog 28377 for a log of the day's events and alog 28417 for more details. SYS provided detailed Zemax-based calculations and a template to use at the output of the ITMy optical lever showing all beam positions, D1600385, which was extremely helpful.
Many volunteered to oversee the vacuum pumping over the weekend, such that the gate valves could be opened on Tuesday evening. The gate valves were opened on Tuesday at 6:40pm; see alog 28454. Low noise was recovered very soon afterward and the CPy pitch moves were considered a success, see alog 28456, reported on Wednesday at 1:41am. The spectrum was reported to be much improved, especially in terms of variance and the beams on SR3 camera we were able to move before with the ITMY CP are gone.
In parallel to the vent effort, folks here and at LHO & CIT are working to identify black glass baffle material and configurations for baffling for the SRM in a future vent.
Planning work needed to be completed before ER10, nominally Oct 31st. Tasks include incremental commissioning of the SUS hardware watchdogs FRS 4504 & the upgrade on the CDS digital timing system firmware to remove the Hz LED flash impact FRS 6062. We are working now to understand the scope of the SEI hardware watchdog install and commissioning.
Identified ETMY 1.25 Hz platform ring up, when stage 1 is controlled. Open loop where measured without indication of instability. The new stage 2 rotation loops help reduce it, but we need to keep investigating, as this frequency is close to the quad mode frequencies.
HEPI pressure readout was mapped out to prepare for channel addition to the epics database. Two new channels need to be added per HEPI pump so we can read the VEA fluid pressure. Michael L is working on the modifications to the 6 cables that need modification. Once all channels are wired, Doug L will create the readout channels.
French drains are being dug around the vault such that condensation will not accumulate inside the box. As soon as this work is completed, the STS will be installed.
PSL summary screens are being updated with Duncan's help, prior to the run start.
Working with Alastair and EE group on testing the TCS chiller summing chassis under different working configurations.
Working with contracts/procurements and our vendor for the front end laser diode boxes to set up a refurbishment contract such that they won't get stuck in Customs hell, as one is now.
Working with AEI on trip to their facility for 70W amplifier integration and testing.
Working with the LIGO business office on PSL spares procurements. Hosting PSL meetings on spares. Keeping google doc on PSL spares updated and dividing tasks between Melanie, Betsy and others.
Ongoing monitoring of charge accumulation on ETMX and ETMY.
Continuing with OpLev upgrades in accordance with ECR E1500224. This week deployed modified ETMY OpLev, and now monitoring for stability and glitches. Also noticed IFO beam coupling into the OpLev QPDs during lock.
Jess McIver is visiting this week, training the new Fellows. She provided a useful DetChar refresher to detector engineers and operators today. It was a lively discussion with operators interested in the detective work of making the connection between the glitches and the source phenomena.
Control Room Operators
Gary Traylor (Richard Oram)
Gary Traylor, Mike Fyffe, Doug Lormand, Joe Hanson, Danny Sellers, Jeremy Birch, Tom Evans, William Parker and Bryan Smith.
Jeremy Birch, Bryan Smith
- Site safety, security and commissioning support with daily shift schedule
Assigned FAMIS work orders
Working on an updated cross training spreadsheet for all operators
Executed a plan to align the CP-Y during a one day vent
Completed assigned Famis tasks
PSL weekly checks
T&T Lab space set-up
- Clean room modifications in high bay.
Clean room moving and assembly
Completed assigned Famis tasks
Provided vacuum assistance to replace GV3 annulus ION pump
Provided Stuart with optical lever assistance at the Y-end station
Supported the CP-Y alignment vent
Supported the CP-Y alignment vent
Provided Fellows with level one contamination control training
Completed assigned Famis tasks
Clean room modification
Cleaned Black Glass for possible baffle work
Performed LSO duties
Replaced the boot on ITMy optical lever after the vent work
LVEA clean up after the vent
Planning to add alarm fields for ion pump status
Checked that the beam tube enclosure sensor read outs are online and recording good data per my assigned FAMIS task
Started another long duration run of the EPICS channel client on the DAQ test stand. Planning to add some new features.
- Continued cleaning up control room work station wiring
Computing and Electronics (CDS, etc.)
- Reviewed existing EPICS databases for ion pumps. Will work with Scott McCormick to determine if calculations needs change. Planning to add alarm fields for ion pump status.
- Checked that the beam tube enclosure sensor read outs are online and recording good data.
- Started another long duration run of the EPICS channel client on the DAQ test stand. Planning to add some new features.
- Working on updated cdsutils, gpstime, guardian RPMs for SL7
- Updated OS, QFS on spare QFS server (before doing same on production servers)
- Documented scikit-image installation on SL7 for HWS servers
- Restarted matlab license service to properly support up to r2016a. Installed r2016a for use on all workstations.
- Started working on a systemd unit file for the epics gateway daemon on SL7
- Found a solution to the ‘invisible cursor’ problem that appears on some users’ SL7 logins
- Reduce frequency of mandatory disk checks on EE Lab test stand restarts
- Reinstalled PSL computer with SL7. With Keith Thorne, installed SL7 on another wall display workstation.
- Rebuilt epics-base package with a usable pkgconfig file. Installed on DAQ test stand for testing.
- Prepared spare boot server chassis for swap of disks upon next lockup. Placed order for new boot server hardware.
- Maintenance updates on several CDS servers. Assisted with DAQ restarts.
- Started build-up of new Guardian server with SL7 OS for testing. Worked with Jamie Rollins on planning out Guardian upgrade at LLO. Testing of new Guardian code on SL7.
- Worked with Paul Corban to clarify best approach to investigate AC power jumps at X-end affecting IFO signals, also seen in vacuum monitors.
- Discussions with Systems, Stuart Anderson on change to single frame type on DAQ for O2.
- Tested back-ported DAQ software changes to support newer framecpp releases on existing production branch. Made corrections as needed to complete back-port.
- Identified primary candidate for sys-admin position. Write-up for staffing committee for review.
- Coordinate Electrical Contractor's daily activities on site:
- Oversee Electrical outfitting of new Laser T&T Lab (ongoing)
- Re-lamp Multi-Purpose room to all new LED Lights
- Investigate 60HZ Noise at X-End relating to an Ion Pump monitoring channel. Discuss additional options to mitigate (In Progress)
- Receive Electrical Materials for projects
- Finalize SKM One Line to document Electrical system additions for the new Laser T&T Lab
- Research Battery Charger for Vacuum Pumps portable battery packs advise Scott McCormick on selection for purchase purposes
Michael Laxen, Carl Adams
- SR3 heater power supply relocation
- ESD Flange connector shield isolation
- 18 bit DAC testing, ADC testing
- DC power monitoring prototype buildout and testing
- Timing firmware update and qualification. Found issue with remaining flashing of internal LEDs on comparator.
- Noise coupling testing at Yend, shutdown of all electronics
- Electronics inventory (ongoing)
- Measure VCO in ISC rack
- TCS Chillers testing
- Support of ITMY chamber vent (HV supplies, ground loop checks)
Timothy Nelson (Richard Oram)
Fault report for this week can be reviewed through aLog link below:
Each item listed has been identified by Facilities with a resolution. The requests have been completed or the work has been scheduled and is pending completion.
Projects and Enhancements
- Facilities’ Dave Barker and EE Paul Corban have been working to resolve electrical spike issues at the X End associated with the building electric heat cycling on and off. SIEMENS Technicians adjusted the programming for the heaters at the X-End to match what we are doing at the Y End…we are using warm return air to supplement the HVAC heating system thus reducing the demand for electric heat. The 70A–80A Jumps due to intermittent heaters turning on and off have been resolved. See aLog entry https://alog.ligo-la.caltech.edu/aLOG/index.php?callRep=28486
- Completed direct burial of the Induction coil Magnetometers and cables this week. Sensors were buried two feet deep at a distance of sixty feet from the seismic vault. The data cables were also run underground from the sensors to the seismic vault. A waterproof penetration was made in the side of the vault and the data cables were ran into the vault for final termination.
Major Infrastructure efforts
Preparing Facility Condition Assessment Report
"Subject to the availability of funding, the Awardee will provide by October 1, 2016 an Asset Condition Report evaluating the remaining life of civil infrastructure at the LHO and LLO sites, and estimating the cost of significant replacement or refurbishment to be scheduled during Oct 1, 2018 - Sept 30, 2023."
Operations Management Team: Bug 112 - Provide Civil Infrastructure Condition Report to NSF by Oct 1, 2016
In response LIGO selected a consulting civil engineering firm (Accruent LLC.) and is conducting condition assessment of the sites and staff training during June and July 2016.
The final report was sent to the LIGO directorate for review:
1) Overview: LLO & LHO Facility Condition Assessment report
2) VFA Facility Condition Assessment of LLO/LHO Final Report Sept 2016. Note there are two reports one for each site.
The data from this assessment are now entered into web based VFA.auditor and will be used for future maintenance management, capital planning & budgeting and report generation.
An appendix to this report is being developed by the LIGO Vacuum/ System Engineering teams and will include the vacuum Infra-structure.
An appendix to this report is being developed to include assets beyond the scope of this third party assessment, i.e. assets not part of the building systems, e.g. the Beam Tube Enclosure and roads, parking lots and grounds at both sites.
Repurpose High Power Laser Facility to HPO Test and Training Facility.
Status: In progress. The project requires improving the particle control and HVAC capability. Ceiling tiles have been replaced with appropriate cleanroom compatible tiles. The wall between the main lab and the adjacent lab (which will house the laser diode power supply) was knocked through and a new metal door added. The old laser table has been removed along will associated cabinets both labs are now completely empty. A new vinyl has been laid over the existing concrete floor to improve particle control and a new wall mounted HVAC unit has been installed. Delivery of new 5’X 12" optical table which has now been relocated into the lab. Installation and set up of mobile HEPA filter clean room over laser table. Upgrades the electrical panel for the room.
Progress this week: None reported
Decommissioning of the initial LIGO LDAS Facility.
Status: In progress. The LDAS facility was shutdown on (07/14/2016). Work has began to decommission this space which will re-purposed to provide additional storage.
No progress this week. Pending: Property disposal
Education and Outreach
- Conducted four field trips. Facilitated by Kathy Holt, William Katzman & Tien Huynh-Dinh.
- Conducted one public tour. Facilitated by Amber Stuver.
- Met with staff from the WWII Museum regarding their expansion & possible future collaboration.
- Scheduled fall groups.
LIGO India Update
LIGO-India Site Acquisition Progress:
The joint measurement survey has begun at the preferred site! This survey provides the detailed legal description of the properties to be acquired. Concrete pillars have been embedded on site to mark the locations of the LIGO-India corner station and end stations.
Separate “tentative” proposals for land acquisition under the forest, government and private land categories were submitted to the District Collector (analogous to a county commissioner in US), Hingoli on Sep 27. (The word “tentative” is part of the actual name of the forms; “tentative” indicates that details will be revised/refined after joint measurement survey).
Another form for requesting land from Latur district to compensate for the forest land to be given to LIGO-India in Hingoli district was submitted.
Maharashtra State Cabinet considered LIGO-India land request and quickly granted approval for the government land request on Oct 3. According to the Hindu newspaper, http://www.thehindu.com/todays-paper/maharashtra-to-be-on-the-crest-of-science/article9186578.ece this decision covers approximately 20% of the required lands.
Data Center Progress:
- The IUCAA LDG cluster is ready for the LIGO observation run O2 which will start soon.
The main computing as well as supporting services (some of which are LIGO authenticated) are working fine. LIGO-India needs guidance on how to open it up to the broader LSC.
The LIGO-India Project Management Board held its first meeting this past week.
LHO Operators and Detector Engineers met this week to discuss the newly distributed assignments for 3-IFO storage inventory and ICS data entry. This work will dovetail with LIGO-India efforts to begin development of logistics planning. Early definition of the sizes and weights of the largest stored items will be used to assess modes of shipment (e.g., air freight or sea shipment) which have different lead times.
DCSEM had requested any available information about potential impacts of future detector upgrades on the civil infrastructure design. We shared LIGO-D1600275-v2 to indicate possible and preferred options for accommodating a long vacuum-squeezer filter cavity.
Mechanics, Optics & Vacuum
Output mode cleaner - The board, mounting prism and PZT were cleaned and are now with K. Arai for re-assembly.
Pre-mode cleaner #11 mirrors have been examined. The mirror with the PZT is the only one that shows unusual contamination (the other mirrors have bits of dust and shattered glue.) For PMC #10 we only had the opportunity to look at the mirror with the PZT. Contamination is similar in character for the PZT mirrors #10 and #11, but not identical.
A new method of high spatial frequency surface figure analysis is being explored with H. Yamamoto.
CP04 is up on the bench, being prepared for trimming of the ESD gold mask overspray.
- Second (final) batch of 4 BRDs for LHO secured in tooling and packed up for shipment (Calum and Norna).
- Remeasured non-magnetic blade dampers for LHO after clean and bake (serial numbers 010 to 017 plus a spare), to check damping was still OK (Calum and Norna).
- Started thinking about design of dampers for OMC bench (~1kHz resonance) and gathered parts for tests (Calum and Norna).
- Ongoing work on BRD paper.
Suspension testing: we installed damping loops. preliminary results tend to show that the A-OSEM will be strong enough to damp the suspension, but not to provide DC alignment (which would have been a bonus, but is not a requirement).
Fiber feedthru: test results show that the acrylate coated fiber are less sensitive to temperature gradients that polyimide coated fiber in terms of polarization stability. We are performing additional tests to confirm these results before using this info to proceed with final design.
Oven: we are characterizing the oven response in order to define the control parameters.
Impact of squeezing: we continue the study showing the impact of squeezing on the probability of detection of BNS.
- PMC Upgrades: Designed fit test base plate. Ordered mirror mount, and test base plate, 1 ea. for testing purposes.
- Corrected minor dimensional error on Figure Metrology, Adapter.
- Ordered additional 75mm disk centering/retain systems.
- Followed up on outstanding orders.
- Detailed SRM/SR3 black glass baffles.(WIP)
- Chevron baffle: Update SW assembly as per T1600426, and detail components (WIP)
- Prepared REFL path sketch between WHAM1 and WHAM2 per J.Kissel’ request
- RFQs and order of Gold Plating process for Thermal Noise R&D, D1500403 and D1500213
- RFQs for fabrication of AERM parts per D1500241-v6
- RFQs for rework of PMC #10 per D1600227-v5
- Test COTS optic mount for information used in designing PZT Mount and testing with D1600398-v1
- Troubleshooting SR3 ROC Actuator RGA experiment at CIT.
- Reviewing and summarizing PSL and TCS closed loop cooling system issues from E1600282
- Install viewport covers on Thomas, CIT 48" vacuum chamber testbed
- PZT jitter test: Setup upgrade to measure transfer function of the PZT actuator. Made a test measurement with help of Rich Abbott
- Chevron baffles for ion pumps: Gathering more coating samples for RGA tests and bead blasted substrates for new chromium oxide samples
- SR3 actuator: made position dependent temperature measurements at different power and ordered new materials for making thermal gaskets
- CRIME: Purchasing supplies and equipment for handling samples of the coated optics. Testing the annealing furnace with other folks.
Control and Data Systems
- Luis has finished schematics and simulations for the electronic fuse part of the SR3 heater driver function. The remainder of the design is straight forward, and will be drawn next.
- Calum and I have been working on setting up the fast shutter for high speed filming this Friday. A film crew is showing up early Friday to take video of various operational shutter modes in hopes of finding a better solution to the reliability.
- Helping with electronic setup of Alena’s PZT study as relates to a newly purchased QPD
- Working on Squeezer electronics manufacturing process by refining assumed initial conditions relating to potential spares
I revved the ISI Outer Loop Transimpedance board per Keita and Daniel’s suggestions, and sent it to them for comment. I am making some test assemblies to vet the SUS Satellite boxes in the 40m lab. I also modified the design of the Eurocrate extender boards, and sent them out for production for Koji.
Data Analysis and Astrophysics LabWide Activities
- In response to some crazy talk, work out location of solar system bodies with respect to detectors a the time of our three detections. No correlation…
- Lots of meetings with group members, reviewing their work plans.
- Prepare michelson demo for shipment to a conference in Pittsburg.
- LOSC O1 release review prep work.
- Continue working with Jonah, Mykyta, and HW injection team on validating and documenting HW injections for O1, and prep for O2.
- LSC ORC related duties dominated this weeks duties.
- Worked with Rory to understand the difference in performance between his ROQ pipelines
and my own runs. This seems to come down to an issue with the condor scheduler placing
a significant percentage of the jobs in a hold state, sometime multiple times for the
same job. Checks of the job requirements indicate that there is not a resource issue.
In one case a job simply disappeared.
- Wrote a simple script to help trace what was happening in all the lalinference_nest jobs
by parsing out bits of info from the error log files. This was shared with Rory.
- Performance of the lalinference_nest jobs are quite impressive when condor allows them
to run to completion.
- Worked with Rory to test waveform generation between GCC and ICC builds. ICC builds are
giving the expected results.
- Attended usual LSC working group weekly meetings.
- Met with Caltech undergrad (and past SURF student) to discuss his plans for graduate
school and application for an NSF Fellowship.
- Identified a failed node in the cluster that had run out of /tmp disk space from another
user’s job that was causing bizarre failures for several other users.
- Built a mini workflow that reads in h(t), injects a CBC waveform into the data, writes that data to a new frame and runs a series of tests: recover the signal with pycbc single template, plot an omega scan of the signal, plot a whitened time series before and after the injection. This will be used to characterize CBC detections in the presence of transient noise.
- Wrote code to subtract auxiliary channels from the h(t) PSD based on coherence to see how alignment control noise was limiting inspiral range. (LLO alog 28492)
- Helped develop a plan for the DetChar rapid response team (validation of low latency candidate signals).
- Reviewed the LigoDV-web technical paper for P&P.
- Characterized L1 post-vent noise.
- Visited LLO this week
- Spent most of the week training the LSC fellows on detector characterization, tools, search backgrounds, and instrumentation ahead of O2: [wiki]
- Led call to prepare detchar representatives to the rapid response team for O2
- Trained with Duncan Macleod on the online Omicron processor to prepare for O2
- Circulated JSI conference abstract [DCC]
- Gave an update on PE detchar project to CBC PE group [wiki]
- Contributed to APS DGRAV nominating committee
- Testing Michael’s tidal ROQ on CIT
- Setting up lalinference_nest jobs to run on Skylake CPUs (CPUs are currently not accessible so will have to wait for the actual submission)
- Working on tensorflow model of nonlinear noise regression.
- Telecon with Eric Thrane and Pablo Rosado on stochastic CBC background search
- Provided Pablo and Eric python code to interface with lalinference waveform generators
- Updated the FIR filter bank at LHO to reflect some naming convention changes in the configuration files, see aLOG 30013
- Documented changes to the filter configuration on the TD calibration wiki page
- Reviewed Laura Nuttall’s training materials for detchar DQ shifts during O2
- Worked on the PTF fast transient rates paper
Advanced Distributed Data Analysis R&D
- Tracked down an issue with factor’s of two difference in performance for lalinference
parameter estimation jobs between Rory and myself to a cause of jobs being placed on
hold and then resuming using the internal checkpointing capability of the lalinference
jobs. Why the jobs were being placed on hold is still unclear, but the resuming of the
jobs was found to have more overhead and when this happens several times to the same
job the performance is impacted.
- Identified an issue with new worker nodes running out of /tmp space and causing other
users’ jobs to behave oddly. Juan was able to fix this once the impacted nodes were
identified in the logs.
aLIGO Analysis and Open Data
- Upgrading LOSC machines to modernize
- Preparing presentation on Skymap Viewer
- Fixing phase error in LOSC open data
- Helping with myLIGO password change
LDAS Software Systems
- Added EXCLUDED_PATTERNS as a general rule of filename patterns to exclude
- Reworked diskcache signal handlers for reloading configuration file and termination.
- Added new command for diskcache to request reloading of configuration file
- Enhanced diskcache.service file to support ‘reload’ and auto restart
- Reworked diskcache daemon mode to handle fast reacquisition of server port on restart.
- Started adding embedded Python documentation of SWIG bindings.
- Help with hardware installation
LDAS System Administration
Continued rearchiving of T10KC -> T10KD
ER8 playback finished.
O1 playback resumed.
Testing Nexsan controller upgrade.
Upgraded samhome2 to SRU–220.127.116.11.0 and HSM–6.1.24a1.
Fixed Nagios monitor for diskcache age (moved script from ldas-cit
Got further instructions from Juan on how to update the qfsnfs2 nagios
threshold (and make it stay updated) and did so.
Working on Oracle tape cases.
Opened a case with Oracle to replace a failed 16Gb fiber channel optic.
- Added auto-creation of static directory to Weekly Report init script.
- Responded to a “help” email about authcore-dev being unresponsive. Discovered that it had nearly 6GB of yum cache. Ran ‘yum clean’ and all was well.
- Diagnosed accidental shutdown of authcore at 2:15 on 10/04.
- Minor involvement in the auth call this week.
- Provided guidance re MyLigo Apache configuration.
- Configured the Dell CMCs with static IPs in the 10.10 subnet
- Assisting Roy with Shibboleth on losc-scratch.ligo.caltech.edu
- Reclaimed the dedicated FFTW benchmarking nodes
- Ran memtest on the first article Dell FX2 nodes
- Stood up ldr-ncsa.ligo.caltech.edu
- Assisted Larry moving dcc-backup.ligo.org to Bridge
- Assisted Michael with sudo privilege on authcore
- Attempted replacing corrupted HDFS file from a copy in /archive
- Enabled DHCP in the 10.10 subnet for management access
- Managed Grid-enabled user accounts
- Requested and installed IGTF, InCommon/Comodo and/or LIGO-CA SSL certs
- Help restart a few LDAS services and followed up on issues after a DCS
power loss at LLO.
- Coordinated the update the calibration code on the DMT to gstlal
calibration–1.0.3–1 at LHO.
- Added new nagois monitoring to the LDAS service group.
- Enable automatic Missing Data Flag insertion for H1_HOFT_C00 and
L1_HOFT_C00 for ER10/O2.
- installed NVIDIA DGX–1 server
- installed 4th Dell FX2 server
- upgraded Condor, CUDA
- set up Squid and CVMFS for OSG
- checked QFS servers in MSR
With Oracle rep, replaced faulty tape drive
Updating labels, binder documentation in DCS room.
Racked, cabled, installed, and brought online 32 new blades on Dell FX–2 cluster nodes
Updated DNS to include more ILOM names
Added ‘check-yum’ nagios probe to track hosts with pending updates
Moved ldrslave web server configuration into puppet
Created shared ldas account
Software updates on all Linux hosts
Ran samfsck on all HSM-QFS filesystems
Recovered from a room overtemp issue that tripped off power to the room
- The group has been working with IMSS in preparation for the e-mail migration to office 365. The migration is to take place on Oct. 12th.
Those that have received their CIT e-mail account in the past year, are already migrated and those that are forwarding their e-mail from CIT to another location don’t have to do anything. The users that do use their CIT e-mail account, will need to make some changes on their client setup on the 12th. The instructions to make the changes should be sent out this week. Just-in-case, a copy has been sent to the local sysadmins.
- Answered questions, made documents public, reactivated accounts, changed group affiliations, deleted versions and documents, added signature capability
- Moderating the dcc-help mailing list due to spams hitting the systems
- Tested dcc-backup replication scripts
- Work on RTs - reactivations, manual updates to the database
- Closed old tickets.
- Test changing the myLIGO passwords in the development system and deploying the changes in the production.
- Remotely monitored all network printers and scanners.
- Bob Taylor- Fixed minor issues with wireless access point.
- Worked on e-waste equipment this week.
- Calum Torrie- Configured new wireless webcams for Calum’s lab.
- Continued testing upgrades from different Mac OS’s to Sierra.
- Some troubleshooting and repair work on Barry’s computer.
- MyLIGO support this week.
- Other misc.: Continued onsite software/phone support.
- Work on a news item: Set up 2 versions of a news page, generated graphics, staged content on ligo.caltech.edu and ligo.org sandboxes. The new content ended up not being published. Rolled back the changes.
- ligo.org updates.
- ligo.caltech.edu updates.
- Working my way through a git tutorial.
- Misc web maintenance tasks/requests.
- Some minor procurement’s to support various computer setups. Order another mac-mini to replace a under-powered unit.
Some minor assistance to others placing orders.
- A lot of logistical work in preparation to upgrade the core switch. The tech. should be in next week to go over the process for the upgrade.
Updated the firmware on the new switch. Still have a few boards to install and update.
Continued working the wireless issue. Received a quote for a special radius installation to handle the new wireless units. Also, continue working on trying to find some other solution to get the system working without too much disruption of how we are presently using the wireless.
- Racked a few more computers.
Reworked more of the security computers. In the next few weeks they should all be in the correct rack locations.
Reworked some of the h/w on the new dcc-test box. It is now ready to be relocated and connected to a 10G switch.
Working with Juan in getting things ready to move the dcc-backup to another location.
- More cable cleanup in two of the computer rooms.
- Quite a bit of spam-appliance work. Made a few modifications that should cut-down on one of the bigger spam sources.
- Patched a number of server and client boxes.
Ran virus-scans on files for a user, they were clean from multiple scans.
- Picked up a number of computer items, left by people no longer working here at CIT.
Worked with Christian on the e-waste items.
- Regular user and meeting support.
- Consultation on setting up Debian8 preseed configuration in CDS.
- Moving forward developing a configuration for Shibboleth federation / Discovery Service replacement for LHO web servers.
- Debian packages for CDS follow-up with Jamie regarding splitting out repos into “production” and “staging/testing”, along with possible avenues for moving forward with the project.
- ZFS/DKMS packages pulling in a kernel headers from CDS staging repo investigated.
- Weekly patches checked/applied to servers.
- Other misc. user support and daily sysadmin tasks.
- Network drops for two offices in the Staging Building
- Provisioned new employee P. Godbey and gave tutorials
- Setup scanner for processing packing slips and receipts
- Configuring replacement DHCP server
- The auditorium sound mixer stopped working, and has a burned smell to it. Working on a temporary replacement while the unit is RMA’d
- Setup a Windows laptop for B. O’Reilly to review information found at the LA Department of Natural Resources
- Fixed a problem with the GC backup service, ended up replacing a LSI SAS card
- Attended weekly GC meeting
- Attended weekly LIAM meeting
- some issues w cds systems, network and timing, cable reconfiguration, resolved
- received other 3 Matt Computers – installed another W10 machine, working on SL
- some seismon issues (conference call)
- rescued macbook pro w usb boot to older os
- migrated more vm’s away from needle to other two esxi hosts
- set up test teamspeak on aws
- set up another qnap iscsi net storage box for vm’s
- archived more older linux home directories from various machines to tsm
- migrated Ken’s ws from local to cloud backup; swapped out mcafee for sophos, set up
daily email notices
- speced out qscan computer w external raid array, getting two other quotes
- took notebook to IS&T for service
- added monitor to mac mini a/v computer in interaction area
- wiped some older disk drives (after archiving)
- did some work on sqlite3 program for backup/sync of teamspeak db to caltech
- looked into zfs raid2z vs brtfs and hw raid6 options for qscan machine; will try ubuntu zfs
- misc purchases
David Shoemaker (Stuart Anderson)
No significant oversight activity.
Data Computing System (DCS)
- Tracking receipt of O2 cluster and data archive orders.
Data Analysis Optimization
- Helped Organize O2-O3 LIGO Data Grid Activities
- Optimization Planning for cWB Pipeline
- Manager Upgrades to O2 Computing Usage Accounting
- Investigation of threading problems appearing in PyCBC
- Suggestions to cWB about compiler debugging
- Studying cWB in preparation for optimization F2F
- Working on quarterly report
Data Analysis Optimization Operations supported
Annenberg Data Center Work
- relocated hp 1920–7 from rack 3.5 to rack 5.4 for an additional net 10.10 switch close to Dell FX2 systems
- re-measured mm fiber needs for row 3 to rack 5.3 (new home of hp 5940s)
- installed (with Ed’s help) Dell FX2 PowerEdge servers (qty 8)
- installed hp 5940 into ToR 5.3, removing hp 2920 to make room
- cable lays and labeling for qty 10 Dell PowerEdge FX2 systems to 10.10 hp 1920s and hp 5940–1
- installed additional hp 1920s to ToR in row 3 and 4
- storage of spare parts/supplies reorganization within empty racks
- O2 equipment spreadsheet and Rack elevation diagrams updated
Additional O2 Compute Supplies ordered
- USB dongles, cat6 ToR management cables, C19/20 shorter power cords
Verified Intel 2017 compilers work ok on gstlal (skylake and broadwell tested)
Verified shoc with cud 8.0 works ok on pcdev11 (Titan X, GTX 10[6,7,8]0 cards)
Found current (new) Caltech Intel Rep for scheduling CPU road map meeting
Pursuing Dell 5940 QoS commands with networking folks @ Dell
Serviced and RMA’s node875 replacement blade - updated repairs spreadsheet