LIGO Weekly Report for the week of 16 Feb - 22 Feb 2019
Past Weekly Reports
The LIGO Executive Committee Agenda for Monday 25 February 2019 will be:
(Meeting time: 10:30 Pacific Time/12:30 Central Time/13:30 Eastern Time)
- LSC - Shoemaker/Cadonati
- Commissioning - Fritschel
- A+ update – Zucker
- Operations Management Team (OMT) - Raab
- Hanford - Landry
- Livingston - Giaime
- India - Raab
- Administration - Hansen
- Engineering Optical and Mechanical - Coyne
- Control and Data Systems - Bork
- Lab Computing - Anderson
- Instrument Science and 40M - Gustafson
- Data Analysis and Astrophysics Lab-wide Activities
- Weinstein, Katsavounidis
- MIT - Fritschel
David Shoemaker (Laura Cadonati)
- Low-Latency Gravitational Wave Alerts for Multi-Messenger Astronomy During the Second Advanced LIGO and Virgo Observing Run was accepted by ApJ
LIGO Laboratory Administration
- MS Project Online Training
- Misc HR issues
- Working on various change requests which includes working with budget managers to review budgets and identify funds to be transferred to salaries for new critical hires.
- Various AP issues and coordinating with Caltech to expedite payments
- Quarterly Report
- Working on Annual Work Plan and Acquisition Plan
- Reviewing T&M contracts
Jamie Goad (Hannah Hansen)
- Published revised labor reports
- Meetings with budget managers
- Creating monthly dashboard for budget managers (in progress)
- Reviewed statements and budget responsibility expectations with newly-assigned task managers
- CR development and implementation
- AP invoice approvals
- Cost transfers and expenditure type changes
- Labor distribution adjustments
Procurements and Subcontracts Management
- LLO neoVAN–4s Laser Amplifiers (Complete)
- LIGO characterization of environmental influences (In Progress)
- LIGO India consulting support (In Progress)
- LHO Vacuum System Turbo Controls (In Progress)
- LIGO Workshop and Seminar Support (In Progress)
Technical Writing Services
- Annual Work Plan: Last minute updates implemented.
- Finished FY19Q1 report
General Business Services
- Updating calendars: Vacation/Travel Calendar, Albert
- Working with hotel to arrange for April PAC meeting
- Working on arrangements for April AURA meeting (Barry)
- processing arriving visitors
- Ordering business cards
- Updates to LIGO roster
- Training for Mast Calendar updates
- Processing post doc applications
- Updating comittee members
- Updating office seating
- Processing terminations
- Post doc hire
- Processing report and parameters for employee leave accruals
LIGO Laboratory Safety
Charles Taylor (Richard Oram, Jeff Jones)
- Chaired LIGO Safety Steering Committee
- Created and prioritized safety procedure list requiring revision
- Attended meetings
Attended online webinars
- 3 Top Challenges to Improving Safety Culture
- Fall Protection Anchor Points
LIGO Hanford Observatory (LHO) and Interferometer Operations
After a great deal of work by commissioners, calibrators, detector engineers and visitors, H1 achieved a binary inspiral range of 100Mpc. Key changes to deliver this state included 2dB of squeezing, the reduction of broad 60Hz and harmonic line widths, better calibration reflective of the current actuation state, and mitigation of the 9Mz coupling. See the commissioning and detector engineering reports for more detail.
- Removal of deposited and drifting snow continues, with the majority of the remainder along the arms.
- Approximated proposed wind fence locations with GPS coordinates.
- In preparation of response to DOE Hanford’s annual request of facility replacement costs (used in prioritization of emergency response), coordinated with Associate Director to provide values consistent across several interfaces.
- Provided interface with IT and Computing Systems personnel as HVAC is revised in LDAS room to better accommodate the cold weather.
Daniel Sigg (Keita Kawabe)
The H1 interferometer reached 100Mpc.
For this lock we had squeezed light injection. The dither line used for locking the OMC was increased in amplitude to reduce the broad shoulders around the line harmonics. Also, the 9MHz coupling was particular low during this lock.
The squeezer now has demonstrated 2dB of squeezing. We continue to work on automation, auto-alignment and noise investigations for the squeezer.
More investigation is ongoing on the somewhat mysterious 9MHz coupling. We have implemented a HV driver circuit to bias the second OMC PZT. Since this changes the cavity Gouy phase, it can potentially be used to push higher order optical modes away from the cavity resonance. However, results are confusing since the coupling changes from day to day.
Chandra Romel (Jeff Jones)
Kyle Ryan, Gerardo Moreno
Replaced an end-of-life annulus ion pump on GV9 at mid-Y
Continued commissioning of new screw pump at mid-Y and preparing for vendor visit week of Feb. 25
Crated large ion pump to ship to vendor for rebuild
Attended first aid/CPR refresher course
Reviewed LOPS vacuum budget
Betsy Weaver (Jeff Jones)
- Hosted weekly site coordination planning/meeting, a bit out of order due to continued snowy weather and the Monday holiday. Tuesday Maintenance was shortened this week in order to get back to commissioning quickly.
- Coordinating some face to face time with Joe O from the UK on upcoming A+ Suspension design work.
- Continued basic training with Rahul - started him on the SUS working so OFI and OMCs. Started some contamination control orientation in Jeff B’s absence.
- Working through QUAD Violin Mode Damper review with committee
- Took the weekly ETM charge measurements - handed off to Rahul later
- Poked a lot of people about various ongoing tasks/projects regarding fiber lab projects, tip tilt spares, lab spaces, scheduling, B&K computer software update, etc.
- Spent a couple hours with Travis cleaning out another layer of 20-year old stuff from the OSB optics labs
- Took lab particle counts to check general health.
- Continued work on O4 PSL optics table layout v1
- Writing TSOP for 3IFO FE alignment work in optics lab
- Writing TSOP for SQZ laser test in optics lab
- First Aid/CPR 2 year refresher training
- Prepare for and host weekly PSL maintenance prep meeting
While on site over the past week (only two days), I've been working on analyzing the recent measurements of ETMX's actuator driver electronics, taken 2 weeks ago (LHO aLOG 46754 and 46854). The problem has been that the supposed short-cut method for measuring the coil drivers in analog produced in strange results which caused non-sensical answers (see discussion in LHO aLOG 46927). So I moved toward trying to integrate the results of the coil driver monitor tests, but those proved more complex to analyze, given the amount of things to account for in the measurement.
The former technique for analysis was resulting in 2–3% residual wiggles an intolerable amount so I've spent my time off-site this past week upgrading all of my laptop infrastructure in order to support python, and begin to use IIRrational.
In light of my embarrassingly and regretfully slow progress, I was also asked to just quantify the current level of badness, which I did yesterday upon arrival back on site. The measurement indicated that we're under reporting the sensitivity by 12% (see LHO aLOG 47003 for details).
Other uses of my offsite time included review of "Calibrating gravitational-wave detectors with GW170817" for the PnP https://pnp.ligo.org/P1900042/, completing back log of safety training, and organizing/booking travel for upcoming NIST and LVC meetings.
- Violin modes Sheila asked me to check if the violin modes frequencies have changed over time (between now and June 2018). I will have to update the numbers on the damping filters. This work will also help us with the identification.
- Study the drawings of the OMC and OFI - I have been digging into the DCC to try and gather up to date info/drawings (current and for the A+).
Control Room Operators
Corey Gray (Richard Mccarthy, Jeff Jones)
OPERATOR OFF-SHIFT WORK
- Submitted merge request to locklost repository to fix/enhance saturation plots
- Submitted another merge to correctly handle 3->2 and 10->2 lockloss transitions
- Submitted one more to tell commissioners whether a plot has failed, is still being created, or is unnecessary given the lockloss parameters
- Investigated issues with the locklost overflows plugin
- Started learning how to use the Python module Bottle for web interfacing
- Second round of TCS custom mask alignment (alog #46998).
- Assisted in Optics Lab clean up.
- Worked on minor issues with our Figures of Merit in the control room.
- Organized recently acquired TCS and HWS spare parts.
Work on conversion from TwinCAT2 to TwinCAT3.
Computing and Electronics (CDS, etc.)
Richard Mccarthy (Jeff Jones)
F. Clara, C. Perez, M. Pirello, Ed Merilh, G. Moreno
- Trying to figure out 180Hz noise on IFO we installed and later removed temporary power supplies on the OMC PZT driver. Also installed the original chassis back in the rack. Will look at modifying the new chassis for later installation.
- Looking at having the ability to control the Envirco fans in the PSL enclosure with Beckhoff module.
- Continued working on the AM-AOM Power Stabilization Circuit for the squeezer.
- Made up some cable connectors for TCS temperature sensors.
- Shipped Beckhoff modules to UW for Newtonian Calibrator work.
Jonathan Hanks (Jeff Jones)
- Continuing to bring DTS test stand up to production standard.
- Resolved long-string error with HWS system.
- Writing python code to parse and verify raw minute trend files, checking new h1tw3 data files are correct before making this a production machine.
- Completed filter loading FRS (filter module name too long).
- Worked with LDAS on resolving failed E18–0 pod, which brought h1fw0 down. Configured both MSR E18 to email sysadmin in case of errors.
- General sysadmin (file permissions, svn management)
- Starting on Duo login config/test to the backup login system
- Discussion of plans for DTS and production as we go into ER14 with Rolf, Keith, and Dave.
- Reloading a crashed disk system in the DTS
- Testing newer nds features on the DTS
- Joining in the discussion of Duo two factor auth in the LSC.
Bubba Gateley (Jeff Jones)
Vanessa G. Karen R. Chris S.
Much more snow removal.
Worked with a contractor and Mitsubishi to try and resolve HVAC issues in the LDAS room.
Added more thermostats to individually control all units.
More work on APS Plaque presentation stone.
Performed regular cleaning duties where accessible.
Major Infrastructure Efforts
Timing of LHO site chiller upgrades as a part of PLMP (Property Lifecycle Maintenance Plan) are being assessed this week.
LIGO Livingston Observatory (LLO) and Interferometer Operations
Pre-work for Engineering Run Fourteen ER14 and Observing Run O3
LLO is focused on supporting the planned commissioning period ahead of ER14.
Note ER14 scheduled to commence 3/4/2019 and O3 scheduled to commence 4/1/2019.
LLO had a very challenging week due to a weather related (very heavy downpour) power outage around 6:30 am Wednesday. Power was restored after ~15 minutes outage. This is the longest and most significant power outage in the last three years.
During the the power outage recovery procedure and checklist Operations it become clear that there were some power distortions for a period ahead of the outage which damaged a few unprotected devices. All identified damaged components have been replaced. However, subsequent to the power outage the commissioning team is finding returning the detector to low noise operation challenging. Please see Valera’s commissioning report for details.
Observing Time Loss due to HW/SW/ procedural faults
Summary of LLO Corrective Maintenance issues that have caused Observing Time Loss
We lost about 60+ hours of observing time this week due to faults that required corrective maintenance.
LLO Power Outage: Power restored and recovery still in progress.
Observatory Computerized Maintenance Management.
FRS - Corrective Maintenance
Key Performance Indicator dashboard of issues tracked in the FRS.
FAMIS- Preventive Maintenance
LLO Tuesday Preventive Maintenance Work Orders 2019 (FAMIS)
Scott Mccormick (Richard Oram)
Reporting for the Vacuum Team, Harry, Scott McCormick
- Recovery efforts related to the site wide power outage completed.
- Reorganizing storage areas to accommodate main turbo bundles.
- Setting up test area in the VPW for main turbo bundles.
- GV–11 Actuator forensics in progress.
- RGA software tests remians in progress in the VPW.
- Capital inventory verification efforts in completed.
- TMDS ion generator aperture plate replaced on unit 4.
- Performed scroll pump maintenance for dust monitors
- Updated FRS tickets
- Updated FAMIS tasks
- Attended weekly A+ meeting
Janeen Romie (Richard Oram)
Detector Engineering summary for Stuart Aston, Matt Heintze, Adam Mullavey & Arnaud Pele
Power outage recovery is the priority. See alog 43513. PSL recovery has uncovered some issues with the PSL chillers: the crystal chiller and the diode chiller. See alog 43526 and FRS 12373. As well, the PSL room temperature has changed, and work is ongoing to stabilize it, see alog 43562. Further subsystem checks after the power spikes and subsequent outage include the ISIs, the BRSs and the ALS.
Investigating IMC locking issues, and the configuration of the MC2 suspension M2 and M3 stage coil driver filter state, which had been modified when locking the IFO without the Reference Cavity. Updated the safe SDF snapshot for the current configuration, see alog 43524 and alog 43520.
Checked in to the CDS userapps SVN locally modified models, filters, safe and observe SDF snapshots, since approximately a month had elapsed since the last time. Also, a few new models had very recently been added, such as CALINJ and those for PEMMID, see alog 43492.
Updated SDF Overview and screen to include new PEMMID and CALINJ models, see alog 43483.
Preparing 2 failing OpLev lasers for RMA return to vendor. Working on service contracts for laser vendor.
Procurement of detector spares, including PSL and TCS spares.
Removing electrical interface from HPO in T&T lab, see TNT alog 11412. Beginning T&T lab Beckhoff code overview/updates with vendor, see T&T alog 11411. Beckhoff terminal failure diagnosis and replacement in T&T lab control box, see T&T lab alog 11410.
Closed loop ETMY measurement to better understand cross coupling, see alog 43491.
Centering of EY and IX BRS, see alog 43486 and alog 43471.
Control Room Operators
Gary Traylor (Richard Oram)
Gary Traylor, Mike Fyffe, Doug Lormand, Joe Hanson, Danny Sellers, Tom Evans, William Parker, Raine Hasskew
Doug Lormand, Joe Hanson, Tom Evans
Site safety, security and commissioning support with 24hr work schedule
Completed assigned FAMIS work orders
Trouble shooting HEPI issues after the power outage. Assisted in replacing power supplies for all HEPI pumps
Supported power outage recovery
Assigned FAMIS work orders
Completed assigned FAMIS work orders
Laser safety documentation approval.
Trouble shooting HEPI issues after the power outage. Replaced power supplies for all HEPI pumps
Supported power outage recovery
Attended the monthly safety steering committee meeting
Power outage recovery efforts
Created ndscope templates to trend optic alignments before and after the power outage
Monthly first aid kit inspections and restock
Computing and Electronics (CDS, etc.)
- Power outage recovery.
- Opportunistic patching of GigE camera servers, ldap servers.
- Improvements to the HWS email alerts. Working with Marie and Aidan on HWS software upgrades
- Built python3.4 and 3.6 packages for pyepics and python-ezca
- Patched and rebooted all epics gateways
- Continued debugging of recent tape backup (amanda) failures
- Upgraded nds2-client, ndscope and python*-nds2-client packages on CDS workstations
- Patched and rebooted CDS external facing servers: opsportal, cdswiki, lloepics, daqsvn
- Maintenance task: Checked FE IPMI logs - nothing out of the ordinary
- Renewed lloboy http cert
- Replaced HDMI extender on cr32
- Recovered from SUS HAM front-end with kernel panic. This did not affect interferometer lock, which used real-time components, so waited for lock loss.
- Completed upgrade of PCIe RFM firmware on front-end system at corner-station. Still investigating anomalies.
- Tuesday maintenance checks on CDS systems.
- Helped with DAQ work (GDS broadcaster list) for CAL model updates by Joseph Betzwieser
- Started update to document on CDS recover from power outages
- Spent all day Wednesday recovering from extended power outage. Ordered replacement (redundant) power supplies that failed on two servers
- DAQ Test Stand: Started recovery from site power outage
- Discussions with LHO CDS on completing new DAQ testing, installation schedule
- Worked on details for planned install of large electrical coils in VEAs for PEM testing
Carl Adams, Michael Laxen
- Troubleshooting of failure of Beam-Tube Enclosure (BTE) network at X–1–5
- Baffle PD gain change simulation and bench test
- PEM X-mid troubleshooting - Found bad PEM AA chassis, swap for spare fixed readout problem
- Spare DC power supply noise/acceptance testing
- Corner BRS stations rack wiring/assembly
Power Outage Recovery:
- DC Power, Front Ends, I/O Chassis
- Help trouble-shoot failed HEPI pump controllers
- Slow-controls: replaced failed EtherCAT module on TCS chassis, recover corner, Y-END ALS
- Test VCO for commissioning
- UIM PUM Transfer Functions for Calibration
Timothy Nelson (Richard Oram)
Facilities Weekly Completed FRS report:
Facilities Weekly Accomplishments / Enhancements:
Facilities’s David Barker found the X End chill water pump #1 inoperative during his weekly PM check. After further troubleshooting, a motor wire lead was found to have overheated from a possible loose connection. The lead shorted out. David cleaned the leads and reconnected. Motor is back in normal running condition. This repair was completed with LLO staff saving hundreds in repair cost and reducing possible down time.
Facilities began the Beam Tube Dry Brushing Project on Wednesday 1/30/19. The project is to brush off debris and contaminants such as rodent feces and insect carcasses that could promote the growth of MIC (Microbial Induced Corrosion) in/on the metal of the Beam Tube. Brushing began on the X Arm Beam Tube at the Corner Station and will continue down the entire length of the X Arm as well as the Y Arm. See aLog 43072 for progress details and some pictures of some of the debris we are removing https://alog.ligo-la.caltech.edu/aLOG/index.php?callRep=43072 .
Update: No brushing this week. We will start up again early next week.
LLO experienced a power outage on Wednesday morning 2/20/19 that lasted about 10 minutes. Facilities checked all of their systems for recovery including HVAC, fire alarm systems, power distribution. All recovered to normal operation after power was restored. The PSL HVAC took a bit longer to recover. Temperatures in the PSL are still recovering at this time but, are trending towards normal. We may be 1/2 a degree from normal. David Barker has been working with Matt Heintze to ensure full recovery. See Matt’s FRS 12403 for details https://services.ligo-la.caltech.edu/FRS/show_bug.cgi?id=12403.
Attended February’s Safety Steering Committee meeting on Thursday 2/21/19. We discussed open issues and incident reports and held general discussion. Very productive. See agenda and minutes in DCC document L1900037 https://dcc.ligo.org/LIGO-L1900037.
Education and Outreach
- Conducted six field trips. Facilitated by Kathy Holt, Tien Huynh-Dinh and William Katzman.
- Conducted the monthly Science Saturday when LLO opens its doors to the public. Facilitated by Tien Huynh-Dinh, Kathy Holt, William Katzman, Aaron Buikema and Joseph Betzwieser. Over 450 people attended.
- Hosted one college and one high school tour prior to the Science Saturday.
- Fixed two exhibits.
- Keith Thorne participated in a Skype session with an artist who is looking to use this in an artistic installation.
- Participated in two STEM nights at local schools.
Mechanics, Optics & Vacuum
Minkyun has dimensioned the bottom mass of the suspension, actuators and coil drivers. We should be able to accomodate about 40 times the inertia of the existing HTTS and still meet dither requirements
Working with our partner in Adelaide who make progress on the thermal deformable mirror, and those in Syracuse who plan to test a prototype of the piezo-flexure mechanism
Assembled a Design Requirements Document
Preparing an improved version of the current design for short term needs
Investigating PCF fibers to transmit more power
Looking into acrylate vacuum compatibility testing, and PEEK or Teflon coatings to contain the acrylate (so we would not be restricted to polyimide coated products any more)
- Continued working on SLIC C concepts
- Update A+ Vac. Equipment layout to incorporate new turbo pump stand, update position and allocation of beam tube ports, add isolation GV, etc.
Control and Data Systems
Todd has been working on the assembly of the BRS Heater driver for Luis. He has also evaluated the first article of the new IO chassis that are being constructed now.
Luis has made some good progress on a new lower noise 6.25MHz CLF RFPD. The existing design in service at the sites has an input electronics noise equivalent to the shot noise of ~250uA of DC photocurrent, whereas the new detector has an input elecronics noise equivalent to the shot noise of ~20uA of DC photocurrent. In addition to this work, Luis has been working on the BRS Heater drivers.
Dean has been updating all the documentation associated with the OMC Piezo Driver circuit boards. He has cleaned things up in preparation for ordering new circuit boards to support the recent modifications to the OMC piezo driver at LHO.
The work that is being done by Kavya Sreedar on the next generation Fast Shutter has been progressing. At last report, the shutter would have needed pulses of current of order 1200 volts in order to close the shutter in 1msec or less. After considerable work, the required voltage is now of order 800 volts, and there is a path forward to lower that further. We consider 500 volts to be the target upper limit for the next generation Shutter.
NDS2 (client, server, and server infrastructure)
- Development and testing of a high capacity nds meta data server.
Submitted to testing
- nds2-client 0.16.3
- nds2-client-swig 0.16.4
MacPorts - testing
- ldas-tools-cmake 1.0.7
- ldas-tools-framecpp 2.6.5
- ldas-tools-framecpp-swig 2.6.8
- ldas-tools-utilities 2.6.4
Michael Zucker (Garilynn Billingsley, Hannah Hansen, Rich Abbott, Lisa Barsotti, Ken Mason, Richard Oram, Norna Robertson, Chandra Romel, Bailey Toon, Calum Torrie)
Rich and Luis are working with Dave and Conor (UK Team) toward having a review of the Suspension Satellite Module. A report is being written such that a review can take place. Our goal is to convene a review committee and complet the review within the next two weeks.
Dean Schaetzl had been testing the GS–13 preamplifier boards that are shortly to be shipped to MIT for inclusion into the GS–13 pods.
Data Analysis and Astrophysics LabWide Activities
- Visitor Ryan Magee is here from Penn State. Working on sub-solar-mass CBC search and sub-threshold triggers.
- SURF application review complete. SURF selection in progress.
- Cross-check list of GRBs during O2 for GWOSC data release, with O2_GRB paper writing team.
- Meeting with GWOSC team on final prep for O2 data release. Responding to GWOSC reviewers comments.
- GWOSC open data workshop planning, https://indico.in2p3.fr/event/18313/
- Sanity check re-release of GWTC–1 PE samples after spin bug fix.
- Reviewing GWTC–1 catalog paper issues raised by journal referees.
- Submit detailed review of “A guide to LIGO-Virgo detector noise and extraction of gravitational-wave signals” , https://dcc.ligo.org/LIGO-P1900004
- Reading and commenting on Sarah’s thesis.
- Meet with Tom to go over thesis plans.
- Continue reviewing and interviewing postdoc applicants.
- Testing new ligo-py* conda environments on the LDG and mac laptop.
- Updates to NSF Annual Work Plan for ALTS Detchar.
- More letters of recommendation.
- parsed through all of the calibration files in O2 and mapped each file to an hour-long active segment in O2 (used 3600 s instead of 4096 s because of the way that the calibration files are divided). Now I am adding the calibration files into the hdf5 file structure. I will re-submit my condor runs so that they loop through hour-long segments and can package up the calibration uncertainty at the time along with the corresponding ASD.
- participated in LIGO SURF candidate selection (reading applications, selecting interviewees, interviewing, and proposing a project). We will be making an informal offer for our project by this weekend.
- Started again on my 3G-localization project. I am running ligo.skymap to get statistics for BNS localized with one 2G, one heterogenous, and one 3G detector network (now applying better waveform and noise assumptions). I plan to modify the ligo.skymap cosmological prior to accommodate star formation rate and see whether that prior (and source distribution) affects the demographics of the detected BNS population.
- Our telescope scheduling paper is submitted to ApJ. Now I am in the process of implementing the airmass-weighted scheduling algorithm into the ZTF gwemopt scheduler; it should be incorporated shortly with the ToO marshal, so that we can see how it affects the observing schedule.
- Focused on next bulk data release
- Read several collaborative papers
- Attended weekly LVC working group meetings
- Fixed DMT SegGener segments to generate DC-READOUT_LOCKED segments and allow offline tools to run
- Ran noise subtraction pipeline on recent commissioning data from H1
- Wrote a gating module for gwpy that will be used to show OMC DCPD A/B cross-spectra on the summary pages
- Continued to edit the O2 DetChar paper
- Read SURF applications to look for a summer student
- Organized and oversaw follow-up of interesting things from O2
- Walked through public summary pages with Jonah and moved the configuration files into version control
- Worked on job applications and professional development
- Took electronics measurements at end stations
- Fixed some bugs for pyDARM
- Presented dual-harmonic HMM tracking paper, jointly presented HMM-seedless paper, both submitted to P&P
- Continue working on referee reports for previous papers
- Continue setting up Cyg X–1 search and simulations
- Worked further on a rates methods paper.
- Worked with a Whitman College student on the channel list for Fscans.
- Started looking again at the updates needed to the SLM calibration
Jonah Kanner (Kent Blackburn, Alan Weinstein)
- Fixed a number of minor issues with webpages related to the GWTC–1 release.
- Fixed a number of context related issues on webpages
- Responded to Ian’s comments about the historical timelines
- Circulated final procedures for making bulk data public
- Posted an announcement on public and private web server home pages regarding upcoming
power outage and sent an email to the user group as well.
- Migrated 8.1TB from /big to /loscdata in preparation for public release.
- Performed an exhaustive md5sum check of migrated data against Agata’s original data
and worked with her to resolve the 13 differences found. All good now.
- Worked with Max on several cronjobs that now sync up with git repository regarding
the installed packages on GWOSC servers.
Laboratory Scientific Computing
- Deployed fetching SSH pubkey from LDAP on high numbered pcdevs
- Deployed host based intra-site SSH logins from the submit machines with restrictive access to group on dedicated machines
- Deployed cgroup memory/cpu limit on the submit machines
- Changed the NFS mount points to MDS on the Online_CBC_PyCBC_Live dedicated nodes
- Increased allocation for dedicated Online_CBC_IIR_GPU_x2200 nodes
- Increased allocation for dedicated Online_CBC_PyCBC_Live nodes
- Increased allocation for dedicated Online_CBC_gstlal_Skylake_other nodes
- Provided login access for ryan.magee on cbc.ligo.caltech.edu
- Stood up squid proxy server on squid1 and switched the CVMFS clients to this server via DNS update
- Assisted Danno with SSH login access on the LDRs
- Assisted in powering down the systems in the Synchrotron data center
- 3G Detector Computing requirements estimation
- with Satya M (MIT), diagnosed and debugged LVC computing accounting issues for O2 and O3.
- worked with Open Science Grid staff and management to plan LIGO computing for O3 and beyond.
- worked to coordinate future LIGO and Virgo computing infrastructure plans.
- performed LSC Computing and Software Committee chairperson duties (agenda preparation, chairing calls, coordinating work).
routine user support and maintenance
provisioned users accounts in the cluster
prepare for 2/23 shutdown
define specs for a clone of emfollow to be installed at LHO
Work rearchiving T10KC -> T10KD continues at CIT and LLO, 0 tapes
finished at CIT leaving 19;0 finished at LLO leaving 406.
Did some testing to try to replace SCSI queue problem but unsuccessful
with fewer RAID units and just lots of volumes. Rebuilt test E48 with
lots of mirrors to try to emulate the original problem.
Moved zfsnfs2 to Annenberg.
Lots of other prep for upcoming power outage.
- Completed most of the work needed to have Python and SWIG bindings distributed as a separate package.
- Travel to and attend Parameter Estimation F2F
- Collaborate on GPU port of PE
- Benchmarking and profiling lalinference, and test newer gcc compilation options
- Worked on SCCB issues.
- Worked to coordinate DMT updates.
- Followed up Disk2Disk errors at LHO and LLO.
- Checked on updates to the LDAS HVAC controls at LHO.
Recovered from power outage on Feb. 19 AM that exceeded the UPS capacity
Replaced failed QLogic 5800 fiberchannel switch
One IBM flash unit didn’t boot properly, required a second reboot
qfs-nfs4 could not mount /scratch
samhome4 boot sector recovery required
Opportunistically patched and rebooted headnodes, NAT, and syslog servers.
Routine weekly DCS software updates
Enabled hostbased auth between DCS headnodes
Enabled opt-out for intel compiler
Moved ldas-pcdev6:/local to a larger, dedicated filesystem
Laboratory IT Security
- Patched all CIT Security and AUTH KDC servers.
- Published critical security and patching errata updates
- Worked on quarterly LIGO cybersecurity report
- Configuration meetings with SANS for CBT based training deployment (uploading pilot group information to begin final testing)
- Obtained DUO keys and installed DUO on security test system
- LIAM group work (high availability Identity provider, single sign-on metadata serving).
- Assisting CDS group with multi-factor authentication testing.
- Following up on vulnerability scan issues with sysadmins.
- Applying patches, upgrading OS releases on older servers.
- Providing patch notices to sysadmins.
Laboratory IT Support (GC)
- Coordination of CIT AUTH team tasks
- Continue work on CIT GC team EPR’s
- Coordinate TAR updates with Veronica
- Chair GC inter site team meeting
- Coordinate Downs/Firestone/Synchrotron power outage on Friday/Saturday
- User requests: reactivate users, delete versions, group updates.
- Working on the LVK document ACL changes
- Worked with Alveera on a document cloning feature : we presented a demo of the new feature and received users’ feedback on proposed upcoming projects
- Work on the KAGRA provisioning script
- Work on the DCC account deactivation script
- Worked on the shutdown and restoration procedures for the Feb 23 CIT power outage
- Assist in server shutdown and restoration during the CIT power outage.
- Worked on the shutdown and restoration procedures for the Feb 23 CIT power outage
- Testing the subjectID as primary key in Grouper
- Assist in server shutdown and restoration during the CIT power outage.
- Working on writing a spec for the LAAC Thesis database for migration to wiki.ligo.org
routine user support and maintenance
prepare for 2/23 shutdown
complete tests of ways to share shibboleth session state between SPs => not satisfactory
check corosync/pacemaker and keepalived
worked on zfs replication process for dcc1,2,3
- More TAR maintenance. Updated a number of server side cache files to correct some outdated information. Did some maintenance of the report archives. Ongoing troubleshooting/tests.
- Patched up and backed up cloud instances that I maintain.
- Posted content updates to the Lab public site and to ligonews.blogspot.com. Ran a link check on the 4 public sites, applied fixes.
- Addressed a maintenance issue of one of active authorize.net merchant accounts.
- Ongoing web maintenance tasks.
- Travel part of the week SolidWorks conference.
- Worked on updating NTSRV's and Linux servers I'm responsible for.
- Worked on finishing up loading a new Engineering workstation.
- Loaded server 2016 to replace one of our IIS webservers, this is an on-going project.
- Put together a game plan regarding the upcoming power outage for my servers that I'm responsible for.
- Worked on fixing a dhcp server that had issues after updating the latest OS/security patches.
- Worked with a user securing and fixing a specialty lab workstation, much work on this.
- Other misc. Users support.
- Purchased some low level computer support items.
Worked on a couple of computer repairs, including a bad battery replacement done through the bookstore.
Located some furniture, thanks to Liz, for the work room. Hopefully, we will get it installed next week.
Located some old items to be turned over for disposition. Trying to locate a piece of equipment that has lost its property tag.
Still moving equipment from the move.
- Working a number of issues in relation to the scheduled power outage.
- Assisted/assisting with a couple of different meetings that are coming up in the near future. Presently, gathering items for the meeting support.
- Repaired a couple of different web services.
- Still testing and updating backup scripts. Down to just a few accounts having problems with the copying of their files.
- Worked with a number of users in getting them started for their e-mail transition over to campus.
Tracked down problems a few people were having with their e-mail. Most were client configuration issues but it did take time to track it down.
- Some minor work on lab server modifications.
- Modifications to a number of service tables.
Patched servers and client boxes.
Updates to the spam-filter appliance.
- User support both locally and remotely.
- Working with Carlos on allocating new user systems.
- Planning more campus email migrations.
- Reviewing operations-GC budgets.
- Testing GlusterFS for replicated virtualization host storage (VMs and containers).
- Building replacement systems for LHO-hosted LIAM services.
- Planning migration/replacement of primary identity provider to cloud-based hosting.
- Assisting metadata admins with testing cloud-hosted metadata serving.
- LIGO.ORG DNS cleanup ahead of IdP migrations.
- Porting end-system syslog policy over from GC to security configuration management servers.
- Debugging low-entropy problems preventing identity provider software from starting up expediently.
- Assisting with configuration and testing of campus multi-factor authentication around SSH logins to management systems.
- Reviewing documentation for email server using a smart-host with authentication (to relay mail from CDS systems to outside).
- Weekly patches checked/applied to servers.
- Other misc. user support and daily sysadmin tasks.
- patched and rebooted servers as needed
- hosted vacuum workshop in MPR
- created AWS to CIT IPSec tunnel for DCC work
- assisted with AWS IdP build
- assisted with AWS instance for sympa
- built new Proxmox VE for production use, but still testing shared storage options
- built alog2 as a container on new PVE, configured and testing new alog code base
- fixed issue with Kantech keeping gate open and doors unlocked
- installed push button in reception office to open main gate. electricians pulled wiring, I connected to controller and programmed input to activate relay
- added “detectorimpacting” to severity dropdown in FRS. selecting it will turn the font red on that ticket in the ticket listings
- moved detectorimpacting to top of severity dropdown
- added LLO SEC product in FRS
- assisted a user with printer installs
- added a new custodian to PCS
- assisted Stephen Appert with multiple ICS issues
- assisted A+ group with ICS questions
- added a cell phone to facility alert email list
- assisted a user with email delivery issues
- assisted auth project member with LDAP overlay questions
- renewed a few SSL certificates
- generated new SSL certificate for new server
- replaced a bulging battery in a laptop
- renewed license for our spam filter appliance, installed new license key
- replaced external disk array on hoss with large internal disks
- attended weekly meetings
- usual user support
- worked a few procurement items
- installed w10 on virtualbox on purplecam for data acquisition. Fixed USB–3 camera
connection issue on virtualbox, working now
- issues printing pdf for Mike Z using Xerox, fixed by changing acrobat printing specification
- worked on NTSC/PAL video capture issue for Slawek microscope camera
- issues with Ken M w10 booting after upgrade re: ss drive. Resolved
- some more work on Rai’s new mac re: mathtype/illustrator/photoshop – resolved
- fixed networking issue on Benjaman’s desktop
- fixed crashplan backup issue for Fabrice
- issues with mx_stream and dataviewer cds systems, resolved
- put together list of mac addresses for IS&T for network ip address changeover
- fixed some remote access issues for Alvaro
- worked on 1 sided vs 2 sided printing issues for Erik – not yet resolved
- set up single page scanning configuration on new copier/scanner for Salvo
- helped Carl with a couple of mac desktop issues related to matlab
- worked on some clearone issues in small conference room and interaction area –
seems to be working reliably
- fixed a dropbox issue for Bobby
- misc purchases