LIGO Weekly Report for the week of 9 October - 13 October 2017
Past Weekly Reports
There will be no meeting of the Executive Committee this coming Monday due to other engagements
Last week Eric Gustafson was elected Fellow of the American Physical Society for 2017.
He is joined in this honor by fellow LSC colleagues Jolien Creighton, Daniel Holz, Vuk Mandic. Congratulations to all for this recognition!
David Shoemaker (Laura Cadonati)
Intensive preparation across the LSC (and Virgo) for the release of papers on new results and preparations for a press conference on 16 October to announce those results.
LIGO Laboratory Administration
- Reconciling final V29 budget to approved CRs. Modifying CRs as necessary and will reroute for electronic signature.
- Working on closing out FY2017
- Published revised job description posting template and instructions.
- Julie updated Staffing Committee website link.
- Large engineering project for LHO/LLO Baffle components (ongoing/5 weeks) specialty packing, shipping and tracking
- Completed Caltech PS end of fiscal year requests
- Completed capital equipment screenings
- Managed logistics activities across the laboratory, “hands-on” special facilities requests, shipping & receiving issue resolution
- Continued transition of property records to electronic card files
- Researched standard shipping practices and options to update LIGO processes
- Continued progress on updating S&R documented processes
- Continued S&R training with Logistics staff across the laboratory, highlighting recent issues
- Attended LIGO Techmart “refresher” training
- Continued progress on scheduling EHS hazardous shipment training
- What-if budget scenarios for FY19–23
- Building a more flexible what-if budget scenario tool
Procurements and Subcontracts Management
Eric Garcia - CIT, LHO and LLO
- LHO- Fire and Emergency Services Review for Bailey (Complete)
- LHO- Clean Room Garb (Complete)
- LHO- Business Analyst Support (In Progress)
- LHO-Bake Enclosure Fab and install (In Progress)
- LHO-Septic Services Mod (In Progress)
- LHO-Pest Control Mod (In Progress)
- LLO- NeoLASE (Draft SOW submitted to LLO and LHO team for review)
- LHO- SOW for Landscaping (in process submitted to Bubba for review)
- Procedure Development
Melanie McCandless LLO
- LLO-Windstream Mods. (In Progress)
- LLO-Woodard Mods. (In Progress)
- LLO-Blahut Mods. (In Progress)
- LLO-LSU Ethernet (In Progress)
- LLO- NeoLASE (70W Builds in process)
Bailey Toon LHO, CIT
- LHO- Fire and Emergency Services (Complete)
Technical Writing Sevices
- Assembling FY17 annual report
- Writing story for Lab website
General Business Services
- Calendar updates: Albert Lazzarini, Travel calendar, LIGOMeetings,
- Processing, reconciling Cardquest for Hiroto, Adhikari, Conf & Meetings card
- Outside calls for Barry Barish/Public Outreach
- Processing in coming external affiliates
- Event planning (October 16 announcement)
- Preparing Staffing Committee meeting agenda for October 30
- Processed expense reports.
- Allocated Pcard transactions.
- Answered inquiries from internal customers.
- Worked with external customers to facilitate travel accommodations.
- Updated CIT and LHO travel calendar.
- Tracked status of visitor’s through the approval process.
- Processed payments for the LSC activities program and sent payment acknowledgments.
- Tracked payments for the LSC Activities program and notified LSC Spokesperson.
- Investigating automated parsing of vulnerability scanner logs and combining with IDS logs for easier summaries / reports.
- Preparing replacement/rebuild/upgrade of Lab backup IdP (login2).
- Reviewing and cleaning up KDC Puppet policy, planning replacements/upgrades.
- Follow-up on IDS summary email notices.
- Assist Randy with FY2017 close-out reports, work plan.
LIGO Hanford Observatory (LHO) and Interferometer Operations
- C. Carrasco, B. Gateley, K. Roland, C. Soike - Facilities
- We began installing padlocks on the beam tube enclosure doors.
- Bubba, Chris, Karen, Christina & others attended a refresher training class for First Aid/CPR.
- We assisted the SUS team with re-install of the ITMx lower structure into BSC3.
- Completed some FAMIS tasks.
- Tagged some equipment with Bailey for property services.
- Weekly and daily systems checks.
- Apollo resumed some of the HVAC Controls Upgrade.
- Removed both doors on HAM 4.
- Still troubleshooting H-2 Electronics Bldg HVAC.
- Installed portable air compressors at end stations for pressurizing potable water tank.
SEI Hugh Radkins and Jim Warner:
SUS and Vent Management
- Changing filters on the Corner Station HEPI Fluid system:
flushing system, changing filters, autopsy of filters
- Post-O2 Vent activities/support: 5-way Cross install at HAM6
of Squeezer feed-thrus for 1064nm fiber optic cables into
- HAM5/6 Septum Window Viewport install for Squeezer.
- Track the HWS and AWC parts for both LHO and LLO through
C&B, through machine shops for various mods, and into kits
for use next week, just in time for Aidan Brooks’ visits to
install and align. This required a few hours in the lab
cleaning the fussy gold coated copper reflectors.
- Reinstalled the ITMx lower quad into the chamber using the
install arm. Still to do is reattach the chains, re-suspend,
rebuild what was removed for the re-weld, and realign to the
- Usual weekly maintenance, alog is 38963.
- The NPRO noise eater indicator on the PSL Laser (and FSS)
MEDM screen was indicating the noise eater was off. Repeated
attempts to reset the noise eater were apparently unsuccessful.
Alog is 38966,
which includes a link to the FRS ticket. P. King, while in the
enclosure for plumbing work with J. Bartlett, checked on the
operation of the noise eater and found that it was working as it
should; alog is 38990.
The thinking at this point is there is something wrong between
the field box for channel H1:PSL-MIS_NPRO_RRO_OUTPUT and EPICS.
Investigation is ongoing.
ITMx Fiber Welding
- Clean up after successful fiber welding. Final alignment
numbers are posted in alog 38952.
Alignment of SLiC Baffles
- Worked with G. Billingsley on setup and alignment of the
system that will be used to align the new BS and ITM elliptical
baffles. Basic concept aligns a laser to the centers of the
ITMy and P/SR3 optics. This beam can then be used to align the
baffles. The procedure for this alignment can be found on the
DCC at E1700341.
Contributions from Betsy Weaver, Jeff Kissel, Jason Oberling,
Travis Sadecki, Hugh Radkins, and Angus Bell.
(Tuesday) We removed, bagged, and tagged mirrors and associated
hardware no longer needed.
(Friday) Both doors have been removed in preparation for next week’s
ITMX and ITMY
(Monday) ITMx has been successfully hung as a newly welded
monolithic. The assembly welding was completed by Travis, Jason, and
Angus. The new ITMx was welded and the violin modes of the fibers
measured in air with the PUM clamped. The welding went quite
smoothly apart from a couple of points:
1) The epoxy around the sapphire prism on one side of the PUM (+Y
side) has charred due to heat from the welding of that fiber.
2) The lengths of the horns on the PUM after fibre removal were
not all exactly 6.0 mm above the ear, and this was not compensated
for exactly with each fiber. The consequence is that the break-off
values for one of the fibres (+X-Y) will be higher at the PUM and a
bit lower at the ITM
(Tuesday) ITMx: The first half of Tuesday was spent collecting and
removing our welding and IAS equipment in the LHO Vertex “Bier
Garten” and converting the space from a laser enclosure welding area
to a install/assembly area. Travis then replaced the middle wire
segment (from top mass to UIM) that was broken during the extraction
of the lower structure. He also added the UIM non-magnetic blade
dampers (NMBDs) to the main chain suspension blades and removed the
unused copper ECD blocks as per E1600011.
ITMY: With Filberto we worked on TCS Ring Heater issues on ITMY.
Garilynn and crew commissioned the new baffle alignment equipment on
ITMy. We got as far as swinging back the ACB and installed face
shield protection on the optics.
(Wednesday) Travis and Betsy put the ITMx main and reaction chains
back together and bolted them up. This process included re-seating
the UIM and PUM magnet flag assemblies to their proper controls
configuration and reattaching them to their 8 respective locations.
Betsy and Travis applied First Contact to the CP-HR side and the
ITM-AR side, then pealed both sheets prior to pushing the structures
together to their nominal 5 mm separation. The unit is still
sitting just outside of the chamber, waiting for install later this
GariLynn and Betsy spent time installing the new AUX alignment
system,that Garilynn procured for us, in the ITMY chamber. It is
currently installed and clamped to the in-chamber stool pointing
(Thursday) With help from Apollo’s Mark and Tyler, Betsy and Travis
re-installed the ITMX lower structure into BSC3. Thursday
afternoon, they worked on re-suspending it to begin assessing
(Monday) Installed the SR3 RoC Heater. The heater is sitting ~6mm
away from the back of the SR3 optic and it is plugged into the feed
(Thursday) In preparation for installation of the Output Faraday
Isolator, Gerardo and Norna finished installing and cabling up the
temporary OSEMs onto the current OFI suspension for testing.
With GregG in chamber and JimW helping outside, a new HAM6–5 Septum
Window was installed.
On entry to HAM6 a cracked black glass beam dump (for OMC Refl Beam
Diverter beam) was found. The dump was broken in a relatively clean
vertical fracture … and when reconstructed appears to show a small
pock-mark from an apparent small laser blast.
Electronics, Control and Data Systems Operations
F. Clara, C. Perez, M. Pirello, Ed Merilh, G. Moreno
- LVEA DC power distribution clean-up continues. Going thru LVEA racks and
removing “temporary cabling” and moving power to electronics to
permanent DC lines. Illuminators/PD amplifier/Op Lever whitening chassis
- Continuity test for in-vacuum SR3 ROC heater. Found issue with in-vacuum
cabling. Power pins at feedthru not seated correctly. br>
- Tested in-vacuum cables for ITMX ring heater/RTD. Issues with in-vacuum RTD at the RTD. Cabling looks good but RTD shows OPEN. Besty looking for spares.
- Continue procurement, assembly and testing of various squeezer electronics.
- Verified functionality of signal chain for testing of OFI at chamber. A couple of temporary chassis installed to facilitate this.
Control Room Operators
OPERATOR OFF-SHIFT WORK
- Completed final round of in-person interviews for Operation Specialist position.
- Assisted IO group with installing/removing parts in HAM2. (alog #38975).
- Looked for parts related to upcoming Tip Tilt installation and also gathered parts for other installation work.
- Sent out optics to MIT.
- Outreach: Prep work for participating in LHO’s local press conference for the 10/16 announcement. Received request to give a LIGO talk at Heritage University in November.
- FAMIS Tasks completed: 6168, 6169, 6919, 6544, 4749
- Ongoing I/O work in HAM2/3
- Subbing at ops 10/10
- Scanned in completed squeezer electronics test plans
- Recorded and aLogged curent IOO values/positions for all optics in HAMs 2 and 3.
- Monolithic welding of new ITMx.
- In-air violin mode measurements of ITMx.
- Installed NMBDs on ITMx.
- Working on reinstalling ITMx into BSC3.
- Wrote EPICS device support for OPC UA using the open62541 library.
- Start of a Python script to parse a TwinCAT3 TMC file into an EPICS database.
D. Barker, CDS Computing
J. Hanks, CDS Computing
- Attending ICALEPCS conference
- Experiments with EPICSv4 for data transport
Daniel Sigg (Keita Kawabe)
No commissioning activity due to vent.
LIGO Livingston Observatory (LLO) and Interferometer Operations
Post O2 Planned Engineering
This Friday marked the end of week seven of the Planned Engineering program of projects which commenced Aug 28th 2017. See chart below for outline and progress.
LLO Post O2 Vent Plan
The main achievements related to the work plan this week are:
The laser team conducted further alignment and tuning of the 70W amplifier on the PSL table including optimizing mode matching into the pre mode cleaner. The output power of the Front End Laser which suddenly dropped off has been recovered. The 70 W laser is up and running and power stability is being check.
The install teams have installed the newly designed and fabricated baffles to reduce scattered light in HAM 5 and HAM 2. Chamber Doors were put back on as part of storm preparations
Thankfully our region received only slight rain and wind as a result of Hurricane Nate.
LLO was closed over the weekend and storm preparations were made. However, LLO was opened for normal operations to commence Monday morning.
Many thanks to all staff for completing the extra work involved in storm preparations.
The main activities for next week are:
Recovery from storm preparations, powering back up many systems that were switched off as a precaution to power outages.
Vacuum Team commenced pump down of Vertex vacuum
There are a number of work permits that are activities that are “dirty” tasks in that they will generate particles i.e. pulling cables, optical fibers and hoses in overhead cable racks that are planned for next week after the doors are back on.
There are plans for afternoon DRMI commissioning work.
Observatory Computerized Maintenance Management.
FRS - Corrective Maintenance
Key Performance Indicator dashboard of issues tracked in the FRS.
Tuesday Maintenance Period Work Permits opened and or executed this week.
FAMIS- Preventive Maintenance
Tuesday Work Permit Summary
LLO Tuesday Preventive Maintenance Work Orders 2017 (FAMIS)
With the vertex pump down underway the input mode cleaner was relocked and the vertex interferometer (DRMI) alignment and locking recovery has started. The DRMI and possibly full interferometer locking is expected to continue for the next three weeks until the remaining parts are ready for the vertex in vacuum installation. The HAM1 chamber is still at air - the investigation of the scatter/clipping in the reflected beam path is expected to be finished on Monday morning. The output mode cleaner was relocked with a single ITM bounce beam. The power in the OMC transmission is consistent with the lower transmission of the SRM: 32% vs 37%.
Scott Mccormick (Richard Oram)
Reporting for the Vacuum Team,
Harry Overmier, Justin Schexnayder, Scott McCormick
- Commenced pumping down the Vertex for commissioning activities.
- Resumed vacuum bakes on RGA manifolds and vacuum bake ovens post site closure due to inclement weather forecast last week.
- Dry air compressor was reactivated to support HAM1 door removal and in chamber work.
- Diagnostics carried out related to LGV8 annulus ion pump alarm status resulting in a bad power supply.
- Auxiliary turbo pump station maintenance performed on multiple auxiliary turbo carts.
- BSC2 annulus ion pump diagnostics remain in progress.
- Attended operations meetings focused on Post O2 vent activities.
- Clean and Bake activities on going.
- Organized storage areas and technical cleaning tasks in the VPW.
- Updated FRS shipping and receiving tickets.
- On going procurement activities in process for vacuum related tasks.
- Updated FAMIS tasks associated with vacuum tasks.
- Attended the weekly vacuum team meeting.
Janeen Romie (Richard Oram)
Reporting for: Stuart Aston, Matt Heintze, Adam Mullavey and Arnaud Pele
Last Friday, the site prepared for Hurricane Nate by putting all of the doors on the open chambers; HAM 1, HAM2 and HAM5. See Richard's alog 36196 for all of the storm preparation. The SEI and SUS systems were damped, see alog 36199. The PSL and CO2 lasers were powered down, see alog 36193.
Because the chamber doors were all installed for the hurricane preparations, the decision was made to start commissioning on Monday, instead of opening the BSC chambers to fix the alignment of the compensator plate behind ITMx and to top gun the input optics, as per G1701508-v5
This week, planned maintenance occurs from 8am 12pm daily and commissioning occurs from 12:01pm 8am. Planned maintenance this week includes pulling cables for the squeezer work, further alignment and mode matching of PSL components, electronics maintenance tasks and organization of LVEA after in-chamber activities.
Working with the commissioning team, we recovered the alignment and locking of the
IMC after the recent vent. We also realigned the vertex and checked the IFO
reflected beam in HAM1, which is still high and clipping on the first wave-plate. Although there are still doubts over this alignment, we plan to raise this wave-plate a few millimeters as we think we have the range. See alog 36283.
The PSL PMC heater servo had issues that were debugged this week, see alog 36360. After this was fixed, the PMC is much more stable, see alog 36264. Mode matching from the 70W into the PMC has been good. The FSS is working well, still working in ISS alignment.
Prior to the storm preparation, the HAM2 optics were inspected and top-gunned, see alog 36159. HAM2 ground-loop hunting identified and mitigated 3 ground-loops, that likely arose during the baffle installation and vent activities see alog 36151. Close-out TFs on all suspensions verified no rubbing or other
Issues, see alog 36227
Last week in HAM5, verification that SRM and SR3 suspensions are healthy and clear of rubbing, after being locked down for ISI balancing, see alog 36141. The HAM5 ISI was balanced, see alog 36116. Ground loops were hunted, identified and mitigated, see alog 36119.
Updating ECR documentation to v4 to cover a request from commissioners
to have the capability to switch linearization for individual ETM ESD
quadrants, which would allow the IFO to acquire lock, then be
transitioned to low-noise using a single test mass,E1500045-v4 Also, this change will remove the non-utilized charge balancing EPICS
records from the linearization for clarity. This work was implemented yesterday, see alog 36261.
Continuing monitoring of charge on the ETMs, after bias was flipped to
attempt to reduce charge on ETMY, see algo 36250.
Monitoring ETMY suspension vertical drift after
experiments with changing the VEAY set-point temperature by 4 F. Also,
identified an unanticipated coupling with RGA vacuum bakes that have
been causing the ETMY to height to sag, see algo 36241.
Debugging of automated matlab transfer functions and awgstream, see FRS ticket 9215. Running overnight TFs on HAM 2 and HAM 5 ISIs to confirm, see alog 36285
Control Room Operators
Gary Traylor (Richard Oram)
Gary Traylor, Mike Fyffe, Doug Lormand, Joe Hanson, Danny Sellers, Jeremy Birch, Tom Evans, William Parker
William Parker, Doug Lormand
- Site safety, security and Planned Engineering support with daily work schedule.
Assigned FAMIS work orders
Completed assigned FAMIS work orders
Storm preparations by replacing removed HAM doors
- 70W Laser install ongoing
Testing and training lab water manifold testing
Assisted with HAM1 door removal, with Danny, Joe and Radar.
Coordinated with Michael Laxen about cable pull, no help was needed because it was a short pull.
Coordinated with Scott and Harry about lift and TMDS table move. We can proceed with the lift once personnel are available, maybe Monday.
Sealed holes in the acoustic enclosure with Danny, Joe and Mike F.
Cleaned acoustic enclosure with Danny, Joe and Mike Fyffe.
Joe Hanson and I took samples of HEPI fluid and sent for analysis.
Removed HAM1 door
Sealed and cleaned Vertex acoustic enclosure
Stored parts and tooling in main building optics lab
Cleaned main building optics lab
- Performed fire extinguisher inspect for October
Computing and Electronics (CDS, etc.)
- Continued tuning the new frame filesystem. Increased the metadata space after getting a better estimate of what is required.
- Kicked off the latest monthly tape backup.
- With Carl Adams, added hardware to EtherCAT Chassis for Squeezed Light installation. Updated TwinCAT System Manager configuration to see the new hardware
- Obtained Windows 7 Pro Activation key and re-actived Windows on Y-end TwinCAT computer
- Patched and rebooted backup fileservers
- Upgrade workstations cr41 and cr43 to SL7, Created autostart script for cr43
- Archived tape B00025 from backup3 and replaced with a new tape
- Cleaned backup vtape 16
- Continuing troubleshooting of user monitor/graphics issues in the control room. Related to nVidia driver, SL7 updates
LLO DAQ test stand
- Received test set of updated PCIe RFM cards from vendor.
- Completed RCG changes to fix incompatibility with EPICS Sequencer 2.2 (also fixed PSLPMC model shutdown issue)
- Completed Debian 8 kernel 3.2 bootserver and stand-alone OS disks, made copies to send to Rolf Bork
- Reassembled x2iscex I/O chassis. Reconfigured x2susex I/O chassis as x2iscey for long-distance RFM tests
- Recovered L1 front-ends from glitch of Timing Master input caused by Squeezed Light installation work, DC power change in that rack
- Updated existing MATLAB nds2-client wrapper for faster data retrieval
- Got quotes for accelerometers for LLO PEM upgrade, mid-station PEM install. Seeking accelerometer signal conditioner quotes.
- Outreach: Tour, interview for Austria Presse Agentur
Carl Adams, Michael Laxen
Timothy Nelson (Richard Oram)
Completed FRS Update:
Projects and Enhancements
Facilities was able to repair several electrical problems around the site including the control for the X End Station Electronics area lighting as well as bulbs and ballasts in Shipping/Receiving. All work was completed with LLO staff and spare supplies saving several hundred dollars in repair costs and minimizing down time.
Facilities was able to complete repairs to the failed hydraulic steering cylinder on the site’s all terrain 6,000 lb. capacity forklift. Repairs were completed by LLO staff saving thousands in repair costs and minimizing down time.
Previously I was able to establish a PM schedule for the site’s 2 large portable generators and the 1 built in generator. Due to unsatisfactory service from the vendor I decided to make a change to a new vendor. This change will give LLO better service for the generator PMs and repairs. The change also gives us a savings of almost $1,500.00 a year.
Facilities continues 2 a day cleaning for all open chambers.
Facilities continues summer time PMs on all HVAC systems including condenser coil cleaning on systems for VPW, Staging Building, SEC, and other outlying buildings.
Facilities continues to complete all FRS requests and other daily duties including, custodial cleaning of the Corner Building, SEC, and Staging Building as well as technical cleaning in accessible garbing areas.
Facilities also completed routine FAMIS tasks including daily check and recording of the site potable water chlorine levels, weekly generator checks (3 total), plus other equipment checks like forklifts and manlifts.
Facilities continues to complete as many requests in house in order to save money including HVAC service and repair, electrical service and repair, plumbing service and repair, and general maintenance tasks.
Education and Outreach
- Conducted seven school field trips. Facilitated by Tien Huynh-Dinh, William Katzman, & Kathy Holt.
- Exhibited at a STEM Festival. Facilitated by Tien Huynh-Dinh & William Katzman.
- Updated website.
- Facilitated 4 press visits.
- Conducted one tour for a group of senior citizens.
- Conducted one public tour.
Mechanics, Optics & Vacuum
At LHO this week testing Beamsplitter elliptical baffle alignment tool.
The test was mostly successful, we will require small re-work of the laser and microscope mount. The system is currently in place and aligned to within ~2 mm for use by TCS alignment next week.
Many thanks to Betsy, Jason and TJ for help in assembly and test.
- At LHO Monday to Wednesday. Worked on OFI suspension actuation with Gerardo, Jeff K and Betsy. Also helped Gari set up some alignment equipment.
- At LLO Friday to take part in filming by BBC.
we have completed the testing of the cavity with measurements of non-linear gain, escape efficiency and higher order modes separation. Though we are still refining the interpretation of results, we are confident that this OPO will perform adequately and we are preparing for shipping it to Livingston.
completed the assembly and polishing of optical fiber feedtrhu for LLO and Lasti. Performed optical transmission measurement, leak checking and final cleaning.
Completed the cleaning and baking of the tooling to insert the suspension in HAM6 at LLO.
- Revised SLiC Barrel baffle mounting bracket and provided CAD data to local vendor for fabrication
- Prepared quick sketch of OFI magnet base and ordered replacement parts to replace non-magnetic bases.
- Re-balanced WHAM5
- Designed SI compatible PreMode Cleaner, KINEMATIC PEDESTALS for KAGRA per Rick. S request
- Continued working on the integration of VOPO SUS into HAM6, and prepared detailed system layouts of VOPO SUS + TIP/TILTS
- Worked on the integration of SRM AWC thermal lens assy. into HAM5, and prepared detailed system layouts.
- Worked with Aidan and Stephen on the relocation of M1 in HAM4
- Updated production drawings for the Chevron Baffle
Control and Data Systems
The focus this week has been on modifications to the CDS real-time code generator (RCG) to produce not only hard real-time kernel objects, but also identical software, less kernel specifics, that can be run as an executable in user space. A first cut at this was completed this past week and initial testing is underway on CDS standard Linux and Debian 8 Linux. This version of the RCG will produce both kernel and user space code whenever ‘make’ is invoked in the RCG build area. Early functional checks indicate this generated user space work properly, able to do most everything the real-time kernel object can, including data acquisition and the running of transfer functions via diagnostic tools. This code is by no means ready for ‘prime time’ release, but good progress. Some foreseen uses of this user space code include:
- Offline control model testing on any Linux machine, including one’s laptop.
- Control simulations, using data feeds from file or DAQ systems.
- Testing of models on line, in safer user space, before running the real-time kernel object version.
- Using newer Linux features, perhaps eventually run in a hard real-time mode without the requirement of the present CDS Linux kernel patch. This would make it much easier to move to newer versions of Linux within the CDS computers, as changes to Linux can break the CDS patch, which in turn requires expertise to update the patch.
- Luis has assembled and tested the whitening filters for the Squeezer Homodyne detector.
- Squeezer in-vacuum cables due to ship from manufacturer today.
- 75 new AA boards are back from assembly. Zach has tested about 50 of them so far. A new batch of an additional 50 or so boards is soon to arrive from LHO.
- Todd has been testing the FET Demodulator power boards.
- Fiberoptic feedthroughs and cables have been researched and ordered for us in the new I/O chassis design.
- Wrote a test procedure for the Homodyne whitening board.
- Met with representatives from other projects regarding epics over websockets.
Vacuum System Sustainability
Jon Feicht (Michael Zucker)
GV–11 Task:Vacuum components received at LLO, electrical components shipped to motion control vendor. ISI weldment received at LLO. Reviewing tools, rigging etc. necessary to R&R bonnet.
GV–8: Plan is to leak check GV–8 stem week of Oct 17. Sustainability task priorities will be evaluated following outcome.
TMP replacement task: SOW being generated.
Ion pump repair: All 4 units shipped for repair (2/site).
Data Analysis and Astrophysics LabWide Activities
- Big push to get GW170814 out the door for LOSC.
- Lots of Collaboration wide telecons this week plus usual working group meetings.
- Lots of publication materials to read this week as well.
- Work with burst group on O2 analysis papers
- Work with Tom Callister and LWA team on radio follow-up program
- Attend burst, all-sky, and DAC telecons
- Read/reviewed several papers
- Worked on noise subtraction
- Submitted CBC DQ paper to arXiv (1710.02185)
- Worked on teaching/research statments
Advanced Distributed Data Analysis R&D
- Identified a number of issues with working with LOSC data in the DCC. All had work-
arounds. Hope to spend some time discussing with Melody when time opens up a bit.
LIGO Analysis and Open Data
Kent Blackburn (Jonah Kanner)
- Mainly worked with getting LOSC data sets into the DCC and interfacing with LOSC web
- Big push to get GW170814 out the door this week for the announcement.
- Continuing to develop new tools for LOSC use.
LDAS Software Systems
LDAS System Administration
- Skylake node support (bad drives and CPU isolation)
- Preparing a skylake node for E3–1240v6 testing (bios, CPU)
- RMA’d failed drives (4,5,8 TB) from assorted node types
- GPU node support
- SimplyDouble install, cabling, etc.
- LLO Oracle Support contract finalizing
- SMC–5 failing power supply support (adjustments to be made across phases and PDUs)
- Supported Melody Araya’s installation of new versions of MediaWiki on dcc, dcc-backup, dcc-lho and dcc-llo. In all cases the database format required patching.
- TRIED to get dcc-dev to behave - It insists on throwing errors about unterminated js, css, and perl code. Something is really wrong.
- Supported Melody Araya with Bugzilla update on dcc. The only real “issue” was that there had been a sendmail upgrade, and the local sendmail no longer knew that it needed to masquerade as dcc.ligo.org.
- Tested exposing DCC public files to port 80 by adding an extra Apache Rewrite rule.
- Upgraded server angel.ligo.caltech.edu from Debian Jessie to Stretch. Initial Apache configuration difficulties arose due to collision with default /etc/apache2/apache2.conf file. Lots of time wasted trying to make sense of log entries that were clearly WRONG, but were in fact right because of the config collisions.
- Documented build of ZFS–0.7.2 on Debian Stretch: https://wiki.ligo.caltech.edu/sysadmin?ZFS-0.7.2_Build_On_Stretch
– SW updates
– DNS adjustement
– finer ACL
Draft solutions to give lsc members who do not have an LDG account “guest access” to jupyterhub
Evaluate options to share notebooks amongst jupyterhub users
Load test DCC and S3 in preparation for Oct 16 announcement
evaluate containerization for apps like DCC and associated mediawiki
Work rearchiving T10KC -> T10KD continues.
40m data in /hdfs up to date.
Work rearchiving files not accessed in at least 1 year finished for
/home1 and /home5. Work continues on /home3, /home4 and /home6.
Continuing to double check that everything was restored to /backup
Exported old unused mm zpools on samarchive
Gave some pointers to Michael Thomas and Alyssa Bramley at LLO on how
to deal with their malfunctioning disk array.
Sent Solaris partitioning instrunctions to Michael Thomas, for adding
local SSD partitions as mirrors for all the mm devices.
Helped Alyssa Bramley with getting the LLO gateway fully functioning
again after rebooting for an SRU update.
Installed SRU 220.127.116.11.0 on ldas-gc and gateway at CIT.
- Discussed LDAS maintenance activities.
- fixed issue with docker on dgx–1
- patched and rebooted all servers with a GC connection
- upgraded head nodes to Condor 8.6.6
- installed CUDA 9.0 and cuDNN 7.0
- installed new SMC storage server
- applied CVE–2017–9798 fix to httpd
- patched and rebooted h1ldasgw0 (CDS QFS writer)
- finished rearchiving the bulk of /backup off T10KC tapes and started reading remaining files off 9940 tapes
- replaced failed DIMM in node816
- Replaced failed disk in Nexsan chassis 48_3 (Disk 7, pod 2) & returned the faulty disk
- Applied latest SRU to gateway
Archived September puppet reports
Racked new NVidia DGX–1 unit.
Recovered the home2 filesystem after multiple hardware failures.
Validated that the newly rebuilt kickstart server can still kickstart a cluster node.
Migrated all remaining SL7 hosts to the new puppet server with a minimal set of rules. Continued migrating the old puppet rules to puppet v5.
- DCC support: Changed document access levels, removed documents, reactivated accounts, make documents public, assigned DCC access to several accounts
- Work on a DCC features
- Assisted with Mediawiki upgrades
- Participated in the patch and reboot of the DCC production server
- Updated the DCC wiki for Hannah and Dennis
- Investigate several DCC anomalies.
- Several meetings with Philippe regarding the new DCC servers
- Support: account reactivation user support, sponsored NSF_Reviewers accounts
- RT support: PAC wiki access, reactivation and transfer of a LIGOLab member (separate tickets).
- Change the myligo server passwords
- Worked with a few users fixing auto sync issues on GoodSync, plus added a few more users to the server.
- Looking into a Firefox issue for a user, due to Firefox crashing. Did a complete reload on Firefox, restored her profile. Then I did a refresh on Firefox.
- Updated all conference rooms computers with latest patches, plus had to make some adjustments on the OS's as well.
- More work trying to get samba mapped drives to work on computers that stop working.
- Put away conference all hands equipment into its own case to be ready be ready for the next all hands meeting. We now have two setups for conferences one with mics and a mixer and the other is a quatro setup for smaller meetings.
- Gathered up equipment that was used for our past Surf students, plus cleared out a few offices with equipment from users that have left the project.
- Setup an office with computer, monitor, plus other peripherals for a new user in Downs 3.
- Fixed the Downs display case due to the OS freezing up. May have to load another mac mini with an updated OS and software to swapped this one out.
- Working on reloading Macs that were turned in from users that have left the project.
- Been working with MacOS 10.13 High Sierra loading GC software plus additional software the project uses like Comsol, Mathematica, Matlab, to see what is compatible and what software is upgradable.
- Ran patch and security updates on NTSRV's, Linux servers, and Conference room computers.
- Working on getting the Goodsync server backed up, due to Samba server is failing to map a drive to backup server. On going project.
- More work on Spam filters for users.
- Other misc. User Support.
- Most time spent on preparation for an upcoming news event
- Published a media advisory on Lab public site and ligonews.blogspot.com
- Demo’ed a content publishing solution on sandbox for Lab web team
- Addressing the issue of SSL implementation on ligo.org
- Backup of ligo.org production instance
- NSF Review web support
- Ran a link check of Lab public sites, applied fixes
- Updates of Lab youtube channel
- Updates of ligonews.blogspot.com
- Just some delivery of support items.
Tracked down some information for property group.
- Worked logistics for a number of meetings and upcoming news events.
- Helped Veronica with a few ligo.org items.
- Debugged a few computers in different labs. One needed a number of additional libraries and drivers, the others were just some configuration tweaks.
- Patches on a few servers and client boxes.
Updated service tables.
Modifications and updates to spam-filter appliance.
- Minor work on ZFS appliances, both old and new.
- Updates on reports for Stuart.
- Regular user assistance.
- Assist Carlos and Dave with a test installation of cron job health checks web service (replacing “OK!” email notices).
- Planning network core swap-out for all–10G/40G fiber.
- Upgrading more servers to Debian 9.
- Testing new DCC server installation Kickstart scripts.
- Planning virtualization host and storage replacements.
- Testing mail server replacement configurations and migration processes.
- Weekly patches checked/applied to servers.
- Security server maintenance, alerting/incident follow-up.
- Other misc. user support and daily sysadmin tasks.
- patched most GC servers, rebooting when needed
- monitored LIGO public website during Nobel announcement and ensuing press conferences. Requests peaked at 2,099 per minute.
- registered new domain name
- more work in AWS with LIGO public website, including use of S3 and CloudFront.
- created new DNS zone, added some entries for new hosts
- a user has a Dell laptop that is out of warranty that is experiencing intermittent display issues. I have ordered a replacement laptop.
- a user has a Dell laptop that is out of warranty and is experiencing issues booting the laptop. I have ordered a replacement laptop and given a loaner for now.
- removed jammed paper from multi-function printer/scanner in shipping and receiving.
- restarted streams of guardshack cameras
- configured a user’s new iPhone for email
- showed a user how to use webmail on shipping and receiving kiosk computer
- aLOG NTP time sync issue prevented users from accessing aLOG. manually synced time. need to investigate further.
- removed a user from mailing lists, and ICS groups
- renewed SSL certificate for a cdswiki
- added new LSC fellow to mailing list
- replaced mouse connected to Raspberry Pi unit that displays guard shack camera video feeds
- reviewed emergency power plans prior to Hurricane Nate
- ordered and received new SM and MM fiber SFPs for new switches
- worked a few other procurement items
- attended weekly meetings
- other misc user support
- Continued configuration of replacement site router / core switch
- Installed replacement wireless ap for the PSL enclosure and tested connectivity
- Noticed that a cooling fan in CUR 3 wasn’t turning. Removed power and investigated. Found an electrocuted insect of some sort. Reported to electrical staff, replaced fan
- Printed / edited many posters for SEC staff
- Setup for LSU Physics Executive Committee meeting here onsite
- Brief overview of AWS instances with D. Giardina
- Attended LIAM meeting
- J. Romie's laptop having intermittent trouble booting. Still investigating, but ordered her a replacement while troubleshooting
- Normal Tuesday patching of GC hosts
- Misc user requests
David Shoemaker (Stuart Anderson, Sharon Brunett, Peter Couvares)
- Interactions with NSF on a Change Request to align budget with a revised plan.
Data Computing System (DCS)
- Worked on Quarterly Report
Data Analysis Optimization
- Continued to work with search group chairs to estimate post-O3 data analysis computing demand.
- Discussed gstlal optimizations with computing staff to inform O3 hardware purchases at UW-Milwaukee.
- Refined weekly data analysis computing reports for LSC science working-group chairs and Data Analysis Council.
- Worked to enhance the LVC data analysis computing usage accounting system, identifying and reporting bugs, auditing results for accuracy, incorporating dedicated and shared computing usage, etc.
- LSC Computing and Software Committee chairperson duties.
- LSC Executive Committee duties.
- Updating and testing pycbc benchmark suite
- Continued investigation of pycbc GPU rare errors in chi-squared calculation
- Researching parameter estimation techniques and possible analytic marginalizations
Data Analysis Optimization Operations supported
- EPYC gstlal_inspiral CPU pinning investigations
- Conducting Phi reruns with Intel 2018
- Preparing for Xeon Gold PyCBC runs, using just provided inputs from Josh (thanks!)
- Vendor meetings for updating Xeon and AMD platform options (plus configurable budgetary quotes)