LIGO Weekly Report for the week of 30 April - 4 May 2018
Past Weekly Reports
The LIGO Executive Committee Agenda for Monday 7 May 2018 will be:
(Meeting time: 10:30 Pacific Time/12:30 Central Time/13:30 Eastern Time)
- LSC - Shoemaker/Cadonati
- Commissioning - Fritschel
- A+ update (when warranted) – Zucker
- Operations Management Team (OMT) - Raab
- Hanford - Landry
- Livingston - Giaime
- India - Raab
- Administration - Hansen
- Engineering Optical and Mechanical - Coyne
- Control and Data Systems - Bork
- Lab Computing - Anderson
- Instrument Science and 40M - Gustafson
- Data Analysis and Astrophysics Lab-wide Activities
- Weinstein, Katsavounidis
- MIT - Fritschel
This week LIGO bid farewell to Ben Abbott (of LSC papers B. Abbott et al. fame)
after 18 years with the Laboratory. Ben is moving on to a career as expert blacksmith
and judge on the
History Channel program Forged in Fire. We wish him well!
The brothers Abbott – Ben(L) and Rich(R)
David Shoemaker (Laura Cadonati)
- Intensive preparations for O3 continue
LIGO Laboratory Administration
- Attended and presented at NSF Large Facilities Workshop. Link to all presentations from the workshop
- Met with Rebecca Yasky and Kristen Spencer at NSF Offices to discuss documentation required for the proposal cost review. Discussed expectations related to content and formatting of all required documents.
- Making changes to revised CR form and procedure based on input from the manager’s meeting.
- Supporting Caltech Accounting System audit. This audit will be performed by Hamilton, which is an auditing firm. The Caltech Audit Team is the lead and I will be supporting by providing LIGO specific information.
- Finalized and submitted the NSF Large Facilities Data Collection Tool for both Operations CSA and aLIGO.
- Submitted financial slides for Q2 report
- Attended and presented at NSF Large Facilities Workshop
- Met with Rebecca Yasky and Kristen Spencer at NSF
- Researched, wrote, and implemented small change requests
- Completed labor allocations to projects that were worked on by staff
- Created April discretionary account/internal costs/p-card reports
- Worked on draft April financial statements
- Reviewed April financial data for mistakes and requested cost transfers
Procurements and Subcontracts Management
- LIGO - Main Turbomolecular Pump Station & Dry Mech. Pump Bundle (In Progress)
- LIGO Safety Engineer (In Progress)
- MIT Ops. Modification (In Progress)
Technical Writing Sevices
- Waiting for final submissions for FY18Q2 report
- Made significant progress in learning how to use LaTeX for Annual Report generation
General Business Services
LIGO Laboratory Safety
Albert Lazzarini (Richard Oram, Bubba Gateley)
Next Week: LLO Annual Safety Review is scheduled for May 8th & 9th 2018.
- Patched CIT Security Servers (2nd run)
- Patched AUTH production KDC servers (2nd run)
- Meetings with lsc-seccomm on impending ligo.org policy change and password rollout
- Continue background research into GDPR compliance for LIGO/LSC
- Read LIGO retreat documents and NSF June review docs
- Testing, deploying KDC encryption type upgrades.
- Rekeying LHO Krb5 services to test upgrades.
- Removing AEI KDC from infrastructure.
- Building vulnerability scanner packages to test out replacement system.
- Applying patches, upgrading OS releases on older servers.
- Providing patch notices to sysadmins.
LIGO Hanford Observatory (LHO) and Interferometer Operations
- G. Moreno, C. Romel, K. Ryan - Vacuum
- Removed and replaced ion pump # 1,2 isolation gate valves at corner station during vent (Mark and Tyler)
- Final touches put on in-house HAM 6 roughing pump cart (current cart is being used at mid-Y for CP4 bake out)
- Removed and installed several viewports from HAM 2,5,6
- Shuffled pump carts around to prepare for corner pump down on Monday
- Met with Gil Hibbs to discuss close out of current CP bake enclosure and ideas for improving a second unit
- Supported clean and bake loads
- Booked travel to LLO next week to support CP warm up for LGV11 exercise
- C. Carrasco, B. Gateley, K. Roland, C. Soike - Facilities
- Move snorkel and scissor lifts multiple times for APS contractor to various areas of the site for security system installation.
- Work on closing out safety incident reports.
- Replace old worn out RO system with new. Having to re-plumb and re-wire entire system.
- Lots of C & B work.
- Regular maintenance, and FAMIS duties.
- Removed and replaced HAM 4 south door, and HAM 5 north and south doors. Also removed BSC 1 & 3 doors.
From Betsy for Betsy, Hugh, JeffK
1 Week Corner Vent
Launched major effort to revisit corner station chambers to
Install 4 AMDs on each ITM test mass
Install a beam dump in front of the ZM2 suspension which was critical
Modify the OFI Refl beam dump to deflect the beam up a bit more
Move some unused viewports from HAM2 to HAM5 to gain views there
Swap the HWS Scraper Baffle with one with an updated aperture size
This included 10 chamber door maneuvers on BSC1, BSC3, HAM4, and HAM5, and the addition of 80+ steps to the schedule to include the typical entry/exit gamut of First Contact cleaning of the ITM test mass, ISI table rebalancing, transfer function and ESD checks, etc. We started on Monday and finished on Fri at ~4. The pump down of the corner will start early next week.
Betsy also spent some time supporting LLO SUS on their EX fiber welding issues, as well as the LHO EX welding which also took place this busy week.
- W/ P. King, R. Savage: Continue working on mode matching of 70W amplifier to PMC. At COB Thursday the visibility of the PMC is at ~82%.
ETMx Fiber Welding
Provide alignment support for ETMx fiber welding. See LHO alog 41813 and associated comments. Align monolithic masses (UIM, PUM, ETMx) in level, position, roll, PUM/ETMx separation, and PUM/ETMx pitch.
Monitor pitch of masses during welding process.
Provide direction for moving masses as needed during welding process.
Electronics, Control and Data Systems Operations
F. Clara, C. Perez, M. Pirello, Ed Merilh, G. Moreno
- APS contractor continues with Access Controls installation. EY switch appears to be malfunctioning as Ryan is not able to communicate with unit.
- Vault work continues. Had Karl (PNSN) and Ray onsite working on seismometer electronics. They would like to change the IP address of the unit but did not have correct adapter cabling to connect to unit. They are expecting to resume work next week.
- ITMs were HIPOT tested to 1kV, all passed. Troubleshooting ESD driving issue on LR segment. All cabling and signal drive all check out going into chamber. Ran test with Field Fox analyzer for EY ESD and found all segments showing same length in chamber ~39ft, bias ~42 ft. Will retest next week to verify results.
- Added h1guardian1 (new guardian server) to puppet.
- Completed puppet policy for video machines in control room.
- Front panels for new RF patch panel and EOM driver electronics arrived. Plan is to run cables inside the PSL enclosure tomorrow as Peter/Jason/Rick will be out all next week on travel.
Control Room Operators
OPERATOR OFF-SHIFT WORK
- Picked up scrubs, folded, and put away laundered blue scrubs.
- Prep and place Contamination Control equipment for HAM4, HAM,5 and HAM6 vent.
- Install temp dust monitor in HAM4 to cover vent.
- Finished overhaul of dust monitor vacuum pump.
- Staged backup dust monitor vacuum pumps at both end stations.
- Conducted site tour.
- Helped with HAM6 close out last week (ground loop fixes for Tip Tilt suspensions)(alog#41722).
- Assisted with installing beam dump in front of ZM2 for HAM5 (alog#41771).
- Installed modified baffle for the OFI in HAM5 (alog#41823).
- Awarded as one of the recipients for the Humboldt State University 2018 Distinguished Alumni.
- Selected as one of the alumni for California State University 2018 Made In The CSU.
- FAMIS Tasks Completed: 6198
- Ground loop testing at HAM6
- Subbed at ops on 4/26
- Cable dressing DBB in PSL rack
- Welding of ETMx replacement fibers.
- Packing and shipping replacement fibers for LLO.
- Swap out SR2 scrapper baffle skin with a new skin that has a larger aperture for HWS beam (41768).
- Guardian - Merge sustools.py and sustools2.py. While doing that, add the necessary code for the new OFI and OPOS SUS Guardians.
- Continued build of the long-range-dolphin system on the DAQ test stand. Found some timing and pci-bus speed issues.
- Making as-built drawings of DTS using google-docs.
- General sysadmin tasks (maintaining vacuum and FMCS alarms, user accounts on svn).
- Working with Carlos on puppet customization needed for h1guardian1.
- Helping troubleshoot dataviewer issues at the 40m
- Code change to the daqd to make it impossible to generate multi-minute trends (ie 2m trends) when we call everything 1m trends.
Daniel Sigg (Keita Kawabe)
No commissioning due to vent activities.
LIGO Livingston Observatory (LLO) and Interferometer Operations
Post O2 Planned Engineering
This Friday marked the end of week 35 of the Planned Engineering and Commissioning program of projects which commenced Aug 28th 2017. See chart below for outline and progress.
LLO Post O2 Vent Plan
The extended vacuum team made progress on preparations for the X End gate valve (GV11) repair.
The vacuum review board has met to review readiness, timeline and resource availability for GV11 repair.
HAM6 was vented and door removed to allow ingress to conduct inspection and replacement of squeezer fiber interconnect.
Preparations being made for the next site visit to conduct BTE and BT inspection by contractor making a study of potential for and evidence of Microbial Induced Corrosion.
Next Week: LLO Annual Safety Review is scheduled for May 8th & 9th 2018.
Observatory Computerized Maintenance Management.
FRS - Corrective Maintenance
Key Performance Indicator dashboard of issues tracked in the FRS.
FAMIS- Preventive Maintenance
LLO Tuesday Preventive Maintenance Work Orders 2018 (FAMIS)
Scott Mccormick (Richard Oram)
Reporting for the Vacuum Team, Harry Overmier, Justin Schexnayder, Brent Hymel, Scott McCormick, Mike Fyffe
- Staging equipment and tools for LGV–11 repair.
- Relocated the clean room over LGV–11 at the x-end.
- Preparing for cryo-trap warm up at the x-end (CP–4). Troubleshooting VME issues.
- HAM6 pump down in progress.
- Scheduling tasks for LGV–11 repair activities.
- RGA testing to prepare for accumulation test on the x-arm.
- Supported MIC investigation activities.
- Chevron baffle inspections and leak checking completed.
- IP2 replacement in the Vertex completed.
- Clean and Bake activities in progress.
- Updated FRS shipping and receiving tickets.
- On going procurement activities in process for vacuum related tasks.
- Updated FAMIS tasks associated with vacuum tasks.
Janeen Romie (Richard Oram)
Detector Engineering weekly summary for Stuart Aston, Matt Heintze, Adam Mullavey & Arnaud Pele
Vent work is ongoing. The new ETMy had all of its fibers welded this week only to have a fiber break during the annealing process. Glasgow and LHO are laboring to provide additional suitable fibers for us in the near and far term. A lessons-learned meeting was held to reduce risk for the next round of welding.
In the meantime, Scott turned off the turbos on HAM6 to allow Maggie to take some data on pressure dependence on the cavity alignment. The HAM6 door will be removed next week and she will check the fiber alignment and replace the fiber feedthrough. The HAM6 door will be closed again before the end of the week. See alog 38899 for more details.
Both HWS SLEDs were swapped, per work permit 4638 and alog 38906.
Installed new watchdog code for the BSC-ISI to include a ramp down of the actuation signal at the time of the trip, see alog 38888.
Continued installation of the End-X BRS. The beam is now balanced and we are in the process of closing up the vacuum chamber.
Continuing noise budget work for stage 1 of the BSC ISI in preparation for BRS filtering.
Updated IFO Alignment Overview screen (including Perl scripts to save/report/restore alignment) to accommodate new squeezer and OFI suspensions, see alog 38814.
HAM6 close-out work with Carl A and Maggie T, mitigating ground-loops, see alog 38855, unlocking OPO suspension and checking balance after the fiber was replaced and re-routed, OPO and ZM1 TFs were clear, see alog 38873, also ongoing debugging of ZM1 pitch-to-yaw coupling, see alog 38829.
With Arnaud P, investigating the significantly rung-up ITMY suspension after suffering a combination of: an IOP DACKILL on l1susb123 (causing suspension damping loops to shut off), a small earthquake, and still isolated ISI. We tried to understand why the ITMY ISI did not trip, like ITMX ISI did, and found that we were not quite at the trip threshold, see alog 38818.
Control Room Operators
Gary Traylor (Richard Oram)
Gary Traylor, Mike Fyffe, Doug Lormand, Joe Hanson, Danny Sellers, Tom Evans, William Parker, Raine Hasskew
Site safety, security and Planned Engineering support with daily work schedule
Completed assigned FAMIS tasks
- ETMY welding preparations
Weekly LN tank fill calculations for Scott
Subbed in the control for a few hours
Continued investigating some of the issues I am having with the new QEgui
- IAS support of the ETMY test mass exchange
Computing and Electronics (CDS, etc.)
- Checked vacuum calculations for this week’s LN2 delivery. Found that numbers agree if vendor reports in gallons instead of tank level.
- Configured and installed 7 new DAQ Test Stand (DTS) network switches.
- Ran new cables in the communications closet in preparation for DTS move.
- Configured DTS PDUs and UPS units so that they can be monitored by LDAS tools.
- Installed the new DTS KVM in DCS Rack 15
- Cleaned up stale files/empty directories on CDS framewriters
Alyssa Bramley (on leave)
LLO DAQ Test Stand:
- Installed additional PCIe RFM switch. Restore PCIe RFM network for DAQ testing
- Worked with Rolf Bork on propagation delay tests. Configured with one long (5.2Km) arm, one short one.
- Verified behavior of RFM switch web monitoring when disabling ports
- Conducted PCIe RFM routing software start/stop/restart tests. Noticed long ADC cycles on some machines.
- Received replacement PCIe routing computer
- Ordered test stand parts
- CDS Tuesday maintenance tasks
- Recovered corner-station front-ends from RFM interruption during maintenance
- Investigated issues with vacuum control readbacks, raw data at X-end
- Created change requests for front-end computer refresh, addition of dedicated computation nodes
- Helped control-room users with data access issues. Ran daily work-permit meetings
Carl Adams, Michael Laxen
- Vacuum controls CP4 Regen electronics checkup
- Mid-station PEM upgrade: Installed shelves and rails for DC power in racks in EE Lab
- Timing comparator fixes
- PSL TTFSS bench testing
- Squeezed Light installation: Gathered notes to update SUS drawings
- Moved DC power supplies to permanent outlets in DC power room
- Ground loop checking, cable disconnects, HV power supply shutdown for HAM6 work
- Reinstalled X-end VCO after modifications
Timothy Nelson (Richard Oram)
Facilities Weekly Completed FRS report:
Facilities Weekly Accomplishments / Enhancements:
- Facilities continued providing personnel the week of 4/30/18 to complete the 2-a-day cleanings at the Y-End while working around fiber welding schedule as that team wants minimal personnel in the area during this welding and other work.
- Facilities assisted with moving the 5,000 lbs. clean #34 from over the chamber at the X End Station to over GV 11 at the X End Station.
- Facilities continued 2-a-day cleaning this week in LVEA while HAM 6 was open through Tuesday when the chamber door was reinstalled.
- Facilities assisted with the reinstall of the HAM 6 chamber door.
- Facilities and Electrical Engineering continued the annual inspections this week of the 34 clean rooms located around the site. The inspections include a physical check of the clean room as well as a check of all the electrical components installed on the clean room. Each clean room is being tagged and dated by each inspector and inspection sheets are being filed in the DCC. Each inspection tag has a note where to find the inspection in the DCC. 11 inspections have been completed so far.
- Facilities reconnected GV 11 power and control wiring per Vacuum Group request.
Education and Outreach
- Conducted 3 school field trips. Facilitated by Tien Huynh-Dinh, William Katzman & Kathy Holt.
- Conducted a teacher workshop for Middle School teachers from Rapides Parish. Facilitated by Tien Huynh-Dinh & William Katzman.
- Conducted one off-site visit for girl scouts in preparation for a girl scout event at LIGO. Facilitated by Kathy Holt.
- Scheduled summer groups.
- Prepared for upcoming Girl Scout Event on site.
- Conducted one tour and meeting with Walker High School administration.
- Continued work on fabrication & assembly of two electric field meters (EFM) for installation in the end station chambers (1 at each of LHO and LLO) Helium leak tested and then charged with Neon. Leak test of Neon by accumulation to to be done next week. [Rich Abbott, Luis Sanchez, Calum Torrie, Bob Taylor, …]
- Continued work on the design and fabrication of an OFI stray light shroud [Calum Torrie, Alena Ananyeva, Stephen Appert, Eddie Sanchez, Corey Austin]
- Issued review report on the installation readiness review for the LGV11 Actuator L1800108. [Dennis Coyne, Richard Oram, Chandra Romel, Mike Zucker]
- Held CSWG meeting (series A). Included a dicussion on the LSC’s program committee from Gabriela Gonzalez. [Dennis Coyne, et. al.]
- Provided answers to A&E questions and references in support of the pending LIGO_India DCSEM review [Dennis Coyne]
- Provided preliminary documentation on which Detector elements LIGO/NSF are not providing to the LIGO-India project, in advance of the quarterly Systems Engineering meeting with LIGO-India project [Calum Torrie, Dennis Coyne]
Mechanics, Optics & Vacuum
- Gathering info for possible change requests.
- Provided comments on some of the documentation on what we are not providing for LIGO India (for Dennis and Calum.)
- Provided comments to SWG chair on white paper.
- Ongoing work with two undergraduate students. One is working on glue investigations, the other is looking into tuning the vibration absorbers to work at lower frequencies, significant below 100 Hz. (with Calum, Stephen, Alena).
- Ongoing support of fibre pulling/welding issues with US and UK colleagues.
Stray light: working on OFI shroud production and future design versions with Eddie S. and Calum T.
A+ coatings: processing and measuring coated samples and substrates
Control and Data Systems
- Working with LHO staff to configure and install new PCIe network equipment into the LHO DAQ Test Stand (DTS). Once install complete, will load test code to verify system operation prior to installation into the LHO CDS system.
- Continued software testing using the LLO DTS.
- First article PCIe Gen 2 I/O chassis complete and will ship to LHO next week for further testing. In particular, want to test the unit’s electronic emissions to verify that it least it does emit more noise than Gen 1 units. Given that these now have power filtration on chassis fan supplies, hopefully noise emissions will be reduce over Gen 1 units.
- The effort to deliver a fully functional electric field meter to LHO is progressing well. Helium leak checking of the prototype cube was performed successfully and much was learned in the process. We had an electronics failure inside the cube which afforded us a useful exercise in repair and recovery. We believe we understand the cause of the failure, so we are not left too worried.
- Some time and effort was spent researching solutions to a safety related issue concerning communications at the sites prompted by the observation that some areas have limited telephone support. Carl Adams and Dwayne Giardina have some experience with solving this problem and have been added to the team (Richard, Tim, Rich, Carl, Dwayne) responsible for this task.
- Kavya is continuing on a part-time basis to characterize the next generation Fast Shutter. Things are progressing nicely. She now has a standard setup for measuring time of flight and is gradually increasing the coil current while taking data. The short term goal is to ensure our requirement of 6mm travel in 1mSec can be reasonably achieved. She will also be drawing conclusions as to whether there are improvements (mass, magnet geometry, or coil modifications) that might lead to a better device.
- Prepared a calculator that is used to find the physical length of cables in the vacuum system utilizing the measured cable reflection coefficient at RF frequencies. The calculator is in the DCC under T1800199
- Koji and I worked on the next steps for the production version of the IO modulator. Although we have not had time to work on this lately, we have a good sense as to what to do next.
NDS2 (client, server, and server infrastructure)
- built on last weeks work to make the C++ interface a native interface that a C++ developer would use.
- removed apis that are now redundant with the C++ upgrade
- user support
- helping users test the nds2 client with python 3
- code cleanup
- starting a branch to remove sqlite completely from the swig client
- diagnosing a user problem, ended up being caused by a server timeout, filed several issues to help handle this better.
Data Analysis and Astrophysics LabWide Activities
- Running parameter estimation on O1 event LVT151012 for the CBC catalog.
- Got /cvmfs/gwosc.osgstorage.org to mount on my laptop, at last.
- Chasing down an issue with the O1 16 KHz strain files for LOSC.
- Continued review of near-final drafts of research proposals from my SURFs.
- Attending pyastro workshop in NYC http://openastronomy.org/pyastro/, learning how to navigate https://www.xkcd.com/1987/
- Continue review of FGMC rates calculations for O1/O2 BNS and BBH.
- Going over the results of the remaining gstlal box openings.
- Set up a O1 16khz review page under the LOSC(GWOSC) wiki.
- Looked into history of review tests done for O1 4khz and made modifications to code,
then carried out the tests that were documented in the history and posted on the wiki.
- Informed the GWOSC Review Team of the review wiki and requested they begin their
tests per Alan’s recommendation in this week’s meeting.
- Worked with Juan and Stuart to assure a FULL backup of the existing GWOSC VMs.
- Modified the nightly backup scripts to engage a transaction lock before performing the
FULL dataabase(s) backup.
- Discussed dual URL certificate for new development VM with Stuart and in a detailed
email to those involved in this setup.
- Carried out a tests on the GWOSC VMs on two separate ocassions (one for local OS up-
dates, and the other after the hypervisor overseeing all the VMs was updated.
- Looking into reason for 0.2% shift in the about of time associated with No CW hardware
injections during O1 reported between the 4KHz and 16KHz tests.
- Read up on the European GDPR and raised an element with the GWOSC users group at this
week’s meeting. Eric will check on the European side for deeper understanding of what
it means to our users dataset.
- Attended weekly LVC working group meetings.
- Worked on Burst section of data analysis white paper
- Attend population model review call, PE call, Burst call, GWOSC call, LSC council, JRPC, DAC
- Met with Burst co-chairs to plan call schedule
- Met with student Tommy Alford on PE samples work
- Continued to work on visualizations for time-dependent posteriors
- Expanded data selection criteria for TBS
- Met with SURF student and planned project proposal
- Reviewed first draft of SURF project proposal
- Reviewed DetChar MOU FTEs
- Worked on O2 data cleaning methods paper
- Submitted bug fixes to gwpy in order to do data selection for TBS
- Set PyCBC jobs running for BCR testing
- Prepared modified O1 DQ products for rerun with O2 codebase
- Worked on LIGO detchar white paper and Caltech detchar MOU
- Worked on glitch PE project: wiki
- Worked on CBC reviews.
- Worked on an outreach talk.
Advanced Distributed Data Analysis R&D
- Worked to assure full backups of GWOSC VMs in advance of a hypervisor upgrade.
LIGO Analysis and Open Data
Kent Blackburn (Jonah Kanner)
- Focused on aspects of testing the O1 16khz data set and tracking down issues.
LDAS Software Systems
- Implemented support of multiple versions of Python for ldas-tools-al
- Implemented most of the support of multiple version of Python for ldas-tools-framecpp. Still resolving issues with running test scripts.
- Helped install several servers and network switches
LDAS System Administration
- Applied Deb7 to Deb9 OS upgrade on the LOSC hypervisor
- Applied Deb7 to Deb8 OS upgrade on some other key servers
- Recorded the MAC addresses for the Dell C6420s in DHCP
- Kickstart installed the Dell C6420s
- Applied singularity update on the submit machines
- Installed 4TB on the x2200 cluster nodes
- Configured the network switches
- Renewed and/or requested IGTF and Comodo certs
- Provisioned grid user accounts
- Worked with Rittal and CIT HVAC team to complete servicing on LCP 2.5
- GPU node failed drive replacement/ RMAs
- Upgrade more skylake nodes with SSDs
- Added links to DCC documents for safety reports.
- Supported folks updating safety incident reports.
- setup of dcc5 at llo
- study unexpected mysql replication failure mode draft remediation script
- setup and fine tune zrep replication between master and slave
- continue containerizing docdb
- routine maintenance
Work rearchiving T10KC -> T10KD continues at CIT and LLO. Now that
I’ve switched to rearchiving GC backups, only 4 tapes have finished
rearchiving, but not yet been completely checked off, so the remaining
total stands at 85. Network disconnections and other outages mean that
LLO has had many restarts and thus no tapes fully completed;13 L0 tapes
are in process of md5 verification and I’ve switched to L1 tapes (since
I’m not going to md5 verify them, they’ll go faster); 573 to go.
Still working on filling in gaps created by incorrect tape relabeling
in the past at LLO, which also has been knocked over by various problems,
but I’m copying the last of the data types (S6 L1) now.
Again shrunk the amount of lookback for low latency files in /ifocache
after it filled up (after we added the _O2 replay data to the mix)–turns
out I had calculated the numbers right, but changed the wrong value in
Replaced failed disk controller in narchive2.
Dealt with an flashcard failure and associated fallout.
Changed all Flash based zpool ZILs to mirrored.
Dealt with 10.14 network failure by following Juan’s directions about
which switch connection to unplug.
Did some test on /home8 as a 6 array config, then reconfig’d as 4 arrays.
Moved samhome5, qfsnfs3 and the /home5 E48 over to the new SAN.
Moved the rest of the T10KB drives to new SAN.
- Worked to debug the new LHO OSG compute element.
Swapped SSDs and HDDs in cluster nodes. Finished provisioning new O3 cluster nodes and reinstalling upgraded Skylake cluster nodes.
Installed new ADMIN LAN cables in the DCS room
Fixed typo in intel compiler setup script
Applied latest SL7 updates to DCS hosts
Created new LDAS users
Submitted the next batch of failed disks for RMA
- Working on the provisioning script changes (KAGRA user provisioning) and implementing a new LVK group
- Worked with Alveera on code updates for bulk metadata changes.
- Made several documents public, updated group assignments
- Testing the LVK group in the test system
- Assisted Albert and Eric with uploading back-dated DCC documents
- Updated several group changes.
- Assisted in the patch/reboot of the master Kerberos system and testing LIGO.ORG key rollover
- More investigation in Grouper on the name change process
- Verified the clean up process for the LDAP discrepancy and ou=group.
- More work on syncing the new PDMPRO Archive server.
- Troubleshooted new PDMPRO server with a database connection to new SQL server this took much time to get these two servers talking to each other correctly in order to create a Vault for the PDM setup. This has been resolved.
- Maintenance on current PDM server.
- Restored a server from LHO. "Old Yacolt" to setup as a backup server for aLIGO business.
This is still an ongoing project.
- Worked with tech going over Calum's laptop.
- Upgraded hardware on an Engineering workstation, upgraded a monitor to get a dual setup working for this user, plus I loaded additional Engineering on this workstation as well.
- Worked with Larry installing a video camera in SCR plus ran updates and made some modifications on the conference computer is that room as well.
- Worked with Larry swapping out the conference equipment West Bridge 2nd floor and clearing out the old equipment, also ran some test on Teamspeak to make adjustments mics and audio on the new equipment. I also installed a Polycom conference phone in this room as well.
- Loading a new Engineering workstation with many Eng. packages, plus other additional software.
- Swapped out some toner cartridges and other supplies on a few printers.
- Troubleshooted and Fixed a network in a users office, had to replace a network switch that went south.
- Backed up a users computer that is leaving the project.
- Working on loading a workstation and laptop for a new hire starting next week.
- Updating all NTSRV's and Linux servers I managed with current OS/Security updates, I’m still working on this.
- Some work on spam filters.
- Other misc. User support.
- GWADW2018 web support
- NSF review web support
- Addressed issues related to ligo.org sandbox upgrade/migration per Webcomm
- Running some JS API tests
- Ongoing misc web maintenance
- Purchased a few more items for the conference rooms.
Received support items for a couple of people and delivered them.
Arranged for battery replacements on macbook and a couple of PC units.
Some minor updates on budget.
- Worked on a couple of license updates. Worked through one issue to where Mike can now access the support area of one of the packages.
- Quite a bit of logistics work for the GWADW meeting coming up. Got the costs down and hopefully the level of support needed, down as well. Still some shipping issues to work out.
- Mike and I spent time updating items in the conference rooms. The webcam installed in the SCR still needs to be properly mounted but is working with its present setup.
Still some minor testing and tweaking needed for the sound in the conference rooms but there has been a improvement since installing the new Revolabs units. A lot of wiring has been cleared out as well.
- Worked logistics with Mike on a couple of the engineering projects.
- Some work in the server rooms. Mostly, recabling and racking up of computers.
- Assisted a couple of students with some s/w setups and modifications to their accounts.
- Patched server and client boxes. Modified a number of the service tables.
- Regular user assistance. Spam appliance modifications.
- Some assistance in tracking down issues with a couple of security boxes.
- Replacement KDC policy work, testing/deploying process for upgrading encryption types across LIGO.ORG KDC infrastructure.
- Assist UW / PacNW Seismic Network staff with network setup.
- Upgrading VM infrastructure management server and investigating replacement system for newer servers.
- User email migration to campus O365.
- More server operating system upgrades (or upgrade by replacement of server).
- Configuring new multifunction printers and print server.
- Complete migration of backup scripts.
- Cleaning off a server to use for upgraded CAD drawing vault.
- Migrating Subversion server backend storage to ZFS and rebuilding logical backups.
- Access control system network configuration changes as contractor progresses through installation of equipment.
- Weekly patches checked/applied to servers.
- Security server maintenance, alerting/incident follow-up.
- Other misc. user support and daily sysadmin tasks.
- bits of review and testing with Ryan as he did major kerberos updates to the LIGO.ORG kds.
- patched and rebooted servers and workstations as needed
- assisting with troubleshooting Quadera performance problems.
- ordered and received laptop for new operations specialist
- requested LIGO.ORG credentials for new operations specialist
- setup mail alias for new operations specialist
- added new ops specialist to mailing lists
- set SEC doors to open for a Saturday tour
- added new resolutions for IIET tickets
- fixed a bug in FRS/IIET quick search where “ACCEPTED” tickets were ignored
- fixed a bug in FRS/IIET where “ACCEPTED” tickets have a strikethrough placed on the text where links to those tickets appear
- cleaned printer in Staging Building to remove streaks from printouts
- updated group membership in ICS for a user
- ordered two laptops for RGA usage
- delivered new laptops to 2/3 of SEC staff
- worked a couple other procurement items
- attended weekly meetings
- misc user support
- Weekly patch cycle
- Fiber testing in advance of PEM install at midstations
- Provisioned laptop for new operations specialist and gave tutorial
- Attended GC meeting
- Attended weekly LIAM meeting
- Minor spam filter modifications
- Misc user requests
- testing migration from mbox to maildir format for email
- installed new backup esxi machine (amd fx 8 core vs intel)
- w Lee, installed new cds machine using new custom rpms
- recovered some ancient data from old machines for EK
- some more work on superstring web cgi for Salvo
- some a/v issues for Nancy re: chromebook, resolved
- some distributed matlab issues for Evan
- fixed a disk issue for Ken
- added more robust code to vmware/ups shutdown scripts for guest machines
- working on ansys and comsol renewals
- working on specs for new lab printer
- installed os/software on new laptop
- misc purchases
Data Computing System (DCS)
- O3 GPU devices received.
- Remainder of O3 CPU cluster nodes received and installed.
Data Analysis Optimization
worked with LSC Data Analysis Council and LSC working group chairpersons to review LVC science priorities and O3/O4 computing supply & demand.
with Josh W. (CIT), reviewed progress optimizing the PyCBC software pipeline running LIGO's flagship deep, offline search for compact binaries to run on consumer GPU hardware for production use during O3.
with Virgo computing chair (Michele P.), discussed opportunities to enable Virgo scientists to assist in adapting more LVC data analysis pipelines to run on Virgo computing resources.
- Testing and debugging configuration of OSG submit host for CIT cluster
- Implemented sine-gaussian veto for GPU on PyCBC, necessary for O2 equivalent workflow to run on GPU
- Continued debugging waveform interpolation on CPU; segmentation fault somehow introduced in latest changes
Data Analysis Optimization Operations supported
O3 cluster installation
- worked with vendor and S&R on pallet delivery timing/location
- with Ed’s help, installed and detached the balance of O3 chassis (qty 16)
- networking and power cabling
- O3 ToR switch installation
- X2200 GPU card upgrade support - metal plates taken to Carp shop, arrange for X2200 loaner return, etc.
- Follow up with vendors on outstanding O3 orders