LIGO Weekly Report for the week of 14 August - 18 August 2017
Past Weekly Reports
The monthly LIGO Staffing Committee meeting will preempt the Executive Committee meeting this coming Monday
David Shoemaker (Laura Cadonati)
LIGO Laboratory Administration
- LLO annual property inventory with Caltech Property Services
- Property inventory reconciliations
- Data entry of property records
- Property dispositions in GSA
- Reviewed purchases for proper expenditure type designation and completed equipment screenings
- Managed logistics activities across the laboratory
- International import of fibers from Germany
Procurements and Subcontracts Management
Eric Garcia - CIT, LHO and LLO
- LHO-Access Control RFP (In Progress)
- CIT-Gate Valve Drive System (In Progress)
- LHO-HVAC Mod. (In Progress)
- CIT-XPLR Mod 1 (In Progress)
- CIT-Tape Order (complete)
- LLO- Battelle (Attempting to get additional information regarding their quote ongoing)
- LLO- NeoLASE (Draft SOW submitted to LLO and LHO team for review)
- LLO- LSU (Researched discrepancies with LSU regarding encumbrances)
- LHO- SOW for Landscaping (in process submitted to Bubba for review)
- LHO- U of Oregon Mod. (Final stages to send to Caltech proc. for review - in process)
- CIT- Tape Drive RFP Executed for Sharon Brunett (Complete)
- Executed some small dollar value procurements
Procurement Team Support
- Addressed invoice problems with XPLR
- Field phone calls related to various procurement questions for LHO, CIT, and LLO
- conducted follow up activities for several placed procurements
- Closeouts Resolved issues with closeouts
- executed tax overrides
- worked with Bailey to create a PowerPoint for the Quick Requisition Guide (draft complete)
- Training Bailey Toon (on going)
Gina Salone CIT, LHO and LLO
- Started working on major modification for FY 2018
- All Sites- completed a number of low dollar value procurements and punch-out purchases
- File inventory (in process)
- Provided Document and file development support for Mods. From CIT, LHO, and LLO
Melanie McCandless LLO
- LLO- NeoLASE (70W Builds in process)
- LLO-DLC coatings for SLiC baffles (Complete)
- All Sites- completed a number of low dollar value procurements
- Fields questions and provides procurement support to site end users
- Executed tax overrides
- Processed invoices
- Performed P-Card reconciliation
Technical Writing Sevices
- Working on various reports
General Business Services
- Calendar updates: Travel calendar, LIGOMeetings
- Processing, reconciling Cardquest for Conf & Meetings card, Hiroto, Barish, Adhikari, guest
- Processing arriving staff
- Updating Desk Assignments
- Updating LIGO Roster
- Preparing Staffing Meeting agenda
- Diversity minutes
Randy Trudeau (Ryan Blair)
- Attend NSF CyberSecurity Summit in Arlington, Va.
- Assist CDS team with Puppet workstation configuration issue
- Assist CDS team with Puppet workstation configuration issue
LIGO Hanford Observatory (LHO) and Interferometer Operations
- G. Moreno, C. Romel, K. Ryan, J. Worden - Vacuum
- Collected quotes and placed orders for vacuum repair/upgrade equipment - mostly valves and ion pumps so far
- Final prep of TMDS for dry run operation next week with Rai W. and Mike Z.
- Clean and bake lab support
- Refining post O2 vacuum task schedule
- Met with local GSA rep to discuss site electric vehicle lease options
- Contractor insulated EX outside nitrogen exhaust line to prevent ice buildup and bird noise. LX is half done and will resume Tuesday, along with EY & LY.
- C. Carrasco, B. Gateley, K. Roland, C. Soike - Facilities
- Erect scaffolding at Mid Y to access smoke alarm.
- Attend Benton Co. PUD meeting.
- Parking lot sealing with contractor.
- Attend meeting with GSA Vehicle administrator, looking into the possibility of EVs for the site.
- Investigate AHU - 3 trip at the LSB.
- Investigate CWP - 1 at End Y.
- Hang pictures in OSB Lobby.
- Daily and weekly system checks.
- C&B tasks.
- Investigate faulty fire alarm panel at LSB.
- Coordinated with SYS for some visitors to come help me with a
variety of vent prep tasks (from baffles to optic work) in the
next few weeks.
- Shepherded vendors, placed orders for supplies, received and
reconciled lots of things.
- Consulted with TJ on VOPO sus build, LISA and Fabrice on SQZ
vent plans and action items, Hugh on HAM6 feedthrus, Travis on a
variety of SUS action items, Calum and LLO on various baffle
mods and action items, Hugh on ISI damper addition plan/ECR.
- Worked with the business office on manpower issues for the
- Weekly SYS, SLC, and IAS meetings.
- Aug 12 2017, Second Saturday Public Tour (assisted by Greg
Mendell & TJ Shaffer), G1601653-v2.
- Recovery from (and documentation of) SUS Rack Power Supply
Failure (with Jim Warner & Patrick Thomas), LHO alog 38162
and FRS 8762.
- ECR on Violin Mode Damping Filter Increase, LHO alog 38198, E1700286, and
- PCAL Maintenance (assisting Sudarshan Karki), LHO alogs 38185
- VOPO Assembly (assisting TJ Shaffer), TST alog 11356.
- Development of VOPO SUS Dynamical Model and SQZ SUS digital
Infrastructure and documentation (adapting from Alvro
Fernandez-Galiana, working with Edgard Bonilla, Rich Abbott
& Stuart Aston), E1700287
- End of Run Calibration Measurements, LHO alogs 38232 and
- Regular ETM Charge Assessment, LHO alog 38200.
- Making figures for Ops Proposal and NSF requests (K. Burtnyk
& Dave Reitze)
- Developed timeline of LHO recovery from July 6th 2017 EQ for
Albert Lazzarini Status Talk, LHO alog 38248.
- Sign convention re-research between LIGO and VIRGO Calibration
- Tweaked beam alignment into PMC to correct for high reflected
power. See LHO alog 38197
for full details.
- Created graph of HPO pump diode operating current vs time for
the HPO pump diode boxes. See LHO alog 38257.
- Pump diode operating currents increasing in a linear
fashion, as expected.
- Weekly PSL FAMIS tasks complete. See LHO alog 38195.
- Monitor health and performance of system.
- Monitor health and performance of system:
- SR3 oplev laser is glitching, will replace once laser SN
105–1 is ready.
- The TCSx CO2 laser tripped OFF the morning of 8/17/2017. See
LHO alog 38246
and associated comments.
- Initially looked like a trip due to low cooling water flow.
- Cooling water flow does not appear to drop low enough to
trip the laser.
- Consulting with TCS laser expert at CIT, investigation
- Monitor health and performance of system.
Daniel Sigg (Keita Kawabe)
No commissioning due to O2.
H1 continues to run with an average of 50Mpc BNS range. The duty factor from Aug/11/2017 00:00:00 to Aug/18/2017 00:00:00 UTC was about 80%.
ETMX suspension power supply failed on Saturday, causing a series of troubles (alog 38157 and 38162). The power supply was swapped (alog 38164).
LIGO Livingston Observatory (LLO) and Interferometer Operations
L1 Detection Coordination
L1 duty factor has been above 75% with a BNS range of ~100 Mpc.
The 10m anemometer on the corner weather station was repaired.
l1daqnds0, the main trend-writer went down Saturday evening probably
due to A/C problems. We also had to replace a Beckhoff Terminal, which
we think failed sometime during the maintenance period also probably due
to excess heat in the room.
The A/C was fixed on Tuesday evening. Following the fix temperatures
returned to normal. The PSL enclosure and diode room had also warmed
up due to the extra load on the air handler. This caused some
degradation in performance. (alog 35417 and 35375)
The Hartmann Wavefront Sensors in the corner station were realigned
and the code was restarted. This took place during regular Tuesday
maintenance. (alog 35407 and 35401).
As part of O2 closeout measurements a full set of calibration sweeps
were performed on Wednesday. (alog 35424).
- I have conducted a DQ shift.
- I have classified 43 days of lock losses into categories like
“unknown”, “earthquake”, etc.
- Arnaud and I moved the LVEA STS B to a second new position. I have
been analysing the spectrum comparison of the 3 seismometers during a
windy period to see the tilt noise comparation to other locations.
- I am studying the ETMY saturations
- Assembled and installed table patch panels for ISCT6 and SQZT6
- Assembled double beam periscope for SQZT6
- Measured vacuum fiber feedthrough transmission
- Bought adapters for motorized waveplate mounts
- Tested motorized beam blocks
- Bought cable assemblies for motorized beam blocks
- Installed new custom power cables for PDs on ISCT6
At MIT most of this week.
Scott Mccormick (Richard Oram)
Reporting for the Vacuum Team,
Harry Overmier and Scott McCormick
- Performed scheduled compressor maintenance on dry air systems.
- Clean and Bake activities.
- Planning activities for post O2 vacuum work.
- On going procurement activities in process for vacuum related tasks.
- Conducted FAMIS tasks site wide.
- Attended weekly vacuum group meeting.
Janeen Romie (Richard Oram)
Reporting for the LLO Detector Engineering Group: Stuart Aston, Matt Heintze, Adam Mullavey, Arnaud Pele
Supporting interferometer alignment, debugging, commissioning and locking efforts. On-call 24/7 to help ensure maximum quality up-time of the IFO.
Working on vent plans, preparation and coordination. LLO vent plan is E1700109. Created alternate plan, accounting for delay in baffle delivery, see G1701508. This was discussed at the SYS meeting on Wednesday and there was general agreement.
Worked on improving low frequency HAM 1 to CHARD subtraction. Calculated a new wiener filter which now needs to be adequately packaged before installation.
Worked on the seismic huddle test stand. Gathered documentation to resurrect the procedure for testing the sensors. Met with Joe H. and Carl A. to plan the next steps.
Worked with SURF student Brian S. on simulating the angular cavity motion using the transmon QPDs.
Wrote a script to analyze all O2 ETMy saturation glitches. The data is currently being processed. We will try to use this data for better subtraction of the glitches in offline analysis.
After receiving requests from LHO, tracking down clean parts for the VOPO suspension to enable wire pulling and blade loading, to ship to LHO.
Ordering maintenance and contamination control items for the upcoming vent work.
Control Room Operators
Gary Traylor (Richard Oram)
Gary Traylor, Mike Fyffe, Doug Lormand, Joe Hanson, Danny Sellers, Jeremy Birch, Tom Evans, William Parker
Doug Lormand, Jeremy Birch, William Parker, Mike Fyffe, Joe Hanson, Tom Evans, Danny Sellers
- Site safety, security and data collection with continuous work schedule.
Assigned FAMIS work orders
Completed assigned FAMIS work orders
O2 post vent preparation
Assisted in corner weather station repair
- PEM: Weather Station - Repaired anemometer at corner station weather station by moving the 3 meter sensor to the 10 meter position after a lightning strike and also replaced the lightning rod
- Testing and training laser lab T and T manifold finished and tested, PSL manifold in progress
- Began LLO SRM prism bonding process
Computing and Electronics (CDS, etc.)
- With Alyssa Bramley, streamlined the event followup procedure
- Finished upgrading CDS syslog server to SL7. Transferred old syslog archives to new syslog server
- Removed more readback channels from conlog
- Wrote EPICS calc, stream device module RPMs and installed on test stand for testing
- Tweaked the CDS puppet rules to clean up errors during the initial puppet run on workstations.
- Updated front-end computer SSH config to allow hostbased auth from the sysadmin server.
- With Alyssa Bramley, upgraded cr21 workstation amidst multiple overlapping hardware failures.
- Upgraded cr21, cr45, cr32 to SL7. Installed another Intel NUC workstation (currently cr49) to replace cr21 once the new monitor comes in.
- Set up a workstation for Mike Fyffe in his office
- Fixed the /etc/grace/gracerc file and added it to puppet for SL7 workstations to fix dataviewer issue. Added a new grace module to the puppet production environment
- Made some adjustments to the hibernate and grace modules in puppet, and deleted references to Subversion in the kickstart file so puppet runs more cleanly the first time on newly installed workstations/kiosk machines.
- Cleaned backup vtape7
- With Michael Thomas, streamlined the event followup procedure
- Troubleshooting sleep problem for xfce users on Intel NUC workstations, which requires hard boot to wake up
- Placed orders for wide-screen monitor of vacuum lab use. Received FE LAN network switches, fiber convertors for weather station Ethernet
LLO DAQ test stand
- Replicated problem reported by Gerrit Kuehn with external EPICs data on front-ends with new EPICS Base (3.15) and sequencer (2.2). Still investigating
- Converted test-stand back to current L1 RCG, Dolphin versions.
- Tested 20-bit DAC install on branch–3.3 code. Mostly works but needs some changes revert before install on L1
- Worked on Dolphin DX (Gen1) tests to confirm ways to prevent front-end model lockups on networked machines during reboots, shutdowns.
- Updated ECR to cover approved changes for mid-station PEM install, additional PEM sensors. Worked to get fabrication started in Pasadena for AA chassis, bare I/O chassis for now-funded ECR.
- Worked with Rolf Bork on plans for procurement of new Dolphin IX (Gen2) equipment, full-scale testing on DAQ test stand.
- More work with Jamie Rollins to solve problems with his Debian upgrades on CIT 40m system
- Worked with Michael Thomas to report over-temperature issues in MSR due to failed A/C unit.
- Investigated user problems with NDS2 data retrieval. Appear related to stale updates on NDS2 servers due to non-optimal operational optimization.
- Recovered trend-writer operation that had failed during over-temp period.
- With Doug Lormand, Joe Betzwieser, trouble-shot problem with IFO slow-controls system. Traced to high error rate on EtherCAT communication module in MSR. Replaced and restored operation.
Carl Adams, Michael Laxen
- Laser T&T Lab: Reinstall of TTFSS into TNT lab
Squeezed Light install
- RF Chassis acceptance testing
- VCO Assembly
- L1-SQZ-R1 Rack assembly
- Testing and repair of spare ADC and DAC adaptor boards
- Planning post O2 work, parts ordering
- OSC DIAGII cable adapter assembly for IFOs, Laser T&T Lab testing
- Interlock cable assembly for IFOs
- Picomotor addition investigation / planning
- PEM: Received Fibox from repair
- TCS Ring Heater troubleshooting: Installed in test rack to trended AA channels to monitor for glitching
- MSR Corner Slow Controls Beckhoff failed module swapout
- MSR AC repair assistance (soldering new sensor)
- Post O2 planning: Cable pull lists
Education and Outreach
- Worked on web-pages detailing activities for recent teacher workshops.
- Conducted Docent Training, where we train SUBR students to become facilitators for public and school field trip visitors.
- Prepared for upcoming Science Saturday, the day we are open to the public on a drop-in basis. Each Science Saturday is themed - and this one is themed to celebrate the Eclipse and educate the public on how to view it safely.
- Conducted two public tours, including a tour from a group of 35 people from Germany. Facilitated by Tien Huynh-Dinh, Joseph Betzweiser, Keith Thorne and William Katzman.
Mechanics, Optics & Vacuum
- Prepared spreadsheet E1700285 for recording tuning of quad BRDs at LHO.
- Prepared list of parts to order for production BRDs for potential use with beamsplitter
- Worked on OFI actuation design and presentation for Systems (with Calum and Eddie)
- Finished talk for SLAC summer school, G1701199, and got it approved by P and P committee.
Continuing the production of vacuum compatible fibers and feethrus for LASTI and LLO. The first fiber feedthru was received and tested at LLO, transmission is as good as before shipment.
Commissioning the setup to test the VOPO cavity
Continuing R&D on fiber feedthrus to improve transmission stability
Completing the design of the insertion tooling for the VOPO suspension (Alvaro)
- Stray light baffles: PRM, SRM, SR2, PR2, MC2, SR3, PR3, Septum, MC1/MC3 and HAM baffles and hardware were inspected and shipped out to be coated with Diamon-like Carbon coating (Alena A., Stephen A., Calum T.) Black nickel coatings provided by different vendors are under evaluation to be used for coating z-baffles and custom washers, brackets. etc.
- CRiMe: Annealing samples of Tantalum Pentoxide coating
Control and Data Systems
- Luis has designed a new circuit for extraction of 3.125MHz signal from OMC DCPD to be used in the Squeezer. He is at the stage of CAD schematic layout now.
- Finished the major work on the Squeezer wiring diagram. Need to seed with location data for cable list.
- Re-did the layout for the demodulator power interface board and submitted it for manufacture. This board will provide whitened monitors of I and Q signals associated with the 2-channel FET I/Q demodulator chassis.
NDS2 (client, server, and server infrastructure)
- Some testing of unit support on nds-dev
- Documentation updates/corrections
- User support
Data Analysis and Astrophysics LabWide Activities
- Write, and submit, LIGO REU renewal proposal to NSF FastLane.
- Present certificates and cash awards to our 2017 Rouse and Blanco summer undergraduate research fellows.
- Many meetings with SURFs Osase, Radha, Rico. Going into the last week!
- Final review sign-off for Observing Scenario Document update.
- LOSC review work.
- Continued to work on migrating LOSC code to use Django 1.11. Have now had some minor
success with getting a few templated pages of text only to display.
- Took minutes and posted for LOSC review meeting this week.
- Took minutes and posted for GSTLAL review meeting this week.
- Attended a number of LSC working group meetings this week.
- Proof read REU proposal for Alan and gave feedback.
LIGO Analysis and Open Data
Kent Blackburn (Jonah Kanner)
- Discussed reworking loscnds to be a Django App with Jonah.
- Some success with using Django 1.11 for LOSC code base this week.
- Attended and took minutes for this week’s LOSC Review meeting.
LDAS Software Systems
- Continued serialization of FrameCPP by implementing: FrProcData, FrDetector, FrMsg, and FrRawData
- Worked on racks to receive S5 nodes
- Removed hard drives for UWM and e-waste
LDAS System Administration
- Assisted Chris and Aaron with the error on their e-mail recipient address
- Activated dcc-dev.ligo.org on vmhost1
- Kickstarted the x2200 nodes from LLO and LHO
- Re-arranged the HP2910 switches from the old black racks to the LHO/LLO racks
- Cabled the ethernet and power cables on the new racks
- Assisted Cody/Chad with their dedicated nodes
- Managed Grid-enabled user accounts
- Requested and installed IGTF, InCommon/Comodo and/or LIGO-CA SSL cert
- Investigated new paradigms for web development - Node.js, W3C XML facilities, data sharing with OpenOffice.
Continued work rearchiving T10KC -> T10KD.
O2 tape validation for last week finished successfully, new pass
40m data in /hdfs up to date.
Between replacing drive 152 and upgrading the firmware on all T10KB
and T10KC drives, all our tape drives have been functioning correctly
for at least a week.
Finished rsync of latest offline h(t) data (hoft_C01).
More work about how hot/warm/cool our home directory data is…i.e. can
we release older files?
Continued to try to figure out why clusterstage fails under SMF.
Started testing with local SSD mm devices on samhome4.
- fixed .htaccess file that was blocking access to a user’s webpage
- pulled Skylake blade and sent it in for repair
- continued testing of HDFS-NFS gateway
- fully populated SL3000 with tapes; labeled and allocated tapes as needed
- LLO LDAS Account Activations (4)
- Changed DCS door code to match that of the control room
Investigating hardware options for GDS filesystem upgrade
Moved locally developed tclnagios package into upstream EPEL repository
Continued validating O2 data tapes
- DCC support: Changed document access levels, removed documents
- Provisioned KAGRA accounts
- Work on a DCC feature
- Worked on RT tickets:
- Tried to figure out how to resolve the PSP access error during the Grouper upgrade.
- Worked on a failed hard disk on user workstation, this ssd disk kept failing I was able to clone it to another disk it seems to be running without any issues.
- More work on getting ready for this weeks SolidWorks upgrade to work SP3.0.
- Loaded a few Mac Laptops to use as loaner laptops.
- Loaded another new mac mini to swap out Downs 3 with to try and fix a video issue. I now have all conference rooms are now swapped out with updated computers loaded with the latest Mac OS.
- Helped Larry out with rack mounting some equipment new equipment.
- Helped with relocating users to different offices.
- Ran security updates on NTSRV's and Linux servers.
- Misc. user support and updated a few software packages for users, plus troubleshooted some issues on Malwarebytes for a few users.
- Added additional rule sets to the spam server to take care of users requests blocking spam/phishing email.
- Work on the internal website
- Updates of ligonews.blogspot.com
- Requested a merchant account for online payment for an LHO public event
- Located a high-resolution image for a publisher, posted to public site’s gallery
- Misc web maintenance tasks
- Wrapped up a couple of purchases. Distributed a number of computer support items. Going through records to make sure all of the contracts for this year have been taken care of.
Ordered a couple of items to extend the hdmi service in the conference rooms.
- Started config and build of a ZFS server. Still a few bugs to work out but so far it is progressing.
Mike and I rack mounted another ZFS server and the setup and config on it will start in a day or so.
- Some minor logistics work with the 40M lab. Hopefully, we will be swapping to a newer edge-switch next week.
- Assisted with a number of different office moves.
- Regular user support, service table modifications.
Patched and updated a number of servers and client boxes.
- Logistics work on some upcoming meetings.
- Spam filter work and updates.
- Some assistance in a couple of lab computer updates.
- Remote assistance to users on travel.
- Filling in for Ryan while he is on travel, weekly patches, …
- Building a display machine for outreach that can work with the high resolution displays.
- Updating outreach displays (With the help of D. Barker) to use the dmt screens from the control room.
- Support for lab wide meetings.
- SolidWorks upgrades for our site engineering workstation and J. Romie
- Intalled robot certificate and configured Raspberry Pi to auto-boot viewing the guard shack camera display
- Replaced end-station access points, decommissioned the Brocade RFS–6000
- Working through auto-population of radius MAC address table from our lab inventory
- Assisted D. Giardina with installing replacement color laser printer in the staging building
- Poster design and printing for Science Saturday
- Installed iMac for VPW staff use
- Attended GC meeting
- Misc user requests.
- converted dSpace disks from scsi (XP) to sata; installed programmable parallel port adapter for dongle
- set up video capture and zoneminder on purple lab computer for Bobby & Nancy
- updated SW to SP3
- working on ldap password verification for resourcespace image delete function
- reinstalled w10 on Alvaro’s machine
- some ilog maintenance
- installed w10 and configured laptop for Slawek for data acquisition
- fixed Xorg issue on gravitation (Matlab cluster)
- new monitors for Fabrice and Mike
- working on wireless vlan for Lee for nw17 lab
- fixed ansys licensing issue - intermittent drops (mac address conflict)
- increased / disk space on maz
- upgraded conference room computer
- some issues with red-pitya for grad students, resolved
Data Computing System (DCS)
- O3 data archive initial shipment received
Data Analysis Optimization
- organized and hosted a two-day joint LIGO / Open Science Grid Computing Optimization Advisory Group Meeting in Syracuse, NY.
- worked with shared computing providers to discuss options for the temporary provisioning of additional High-Throughput Computing (HTC) supply on short notice for for future bursts of computing demand in O2 and O3.
- worked to enhance the LVC data analysis computing usage accounting system, identifying and reporting bugs, incorporating Open Science Grid computing usage, checking our calculation of each LIGO Data Grid site’s computing usage to accurately reflect the use of CPU Hyperthreading, and accounting for dedicated computing resources not managed by HTCondor.
- worked with LIGO and Open Science Grid (OSG) technical experts to identify bottlenecks and inefficiencies in the workflow scheduling of LIGO’s deep offline Compact Binary Coalescence search (PyCBC) on external shared computing resources. Implemented and deployed PyCBC workflow configuration changes to reduce the frequency and cost of unexpected job preemption. These changes are expected to result in a significant (~20%) improvement in the overall CPU efficiency of large-scale PyCBC workflows on OSG, and may also improve the time to result.
- ad-hoc LIGO / Open Science Grid (OSG) systems administration duties
- LSC Computing & Software Committee chairperson duties (organizing and chairing meetings, preparing agendas, following up on requests and action items, etc.)
- prepared travel and logistics for the upcoming LVC meeting at CERN next month.
- Coordinating with Lars Koesterke (TACC) on ECSS project for PyCBC
- Administrative tasks from summer travel/changing positions
- Continued investigation of PyCBC waveform interpolation on GPU
Data Analysis Optimization Operations supported
- Worked with Bob to crane X2200 servers and racks into Synchrotron high bay
- X2200 servers work
*> loaded into empty racks
*> kickstarting, rack elevation guide updates
- S5 node drives prep’d for UMW
- 5412 power cords (verify ordered set work and ordered more)
- Obtained quotes for maintenance and GPU projects
- helped SURF with cWB talk/paper
- EPYC test system talks with vendors
- ZFS server - 4 TB skylake SSD re-purposing options - quotes in progress