LIGO Weekly Report for the week of 14 September - 18 September 2015
Past Weekly Reports
The monthly LIGO Staffing Committee meeting will preempt the Executive Committee meeting this coming Monday, 21 September 2015
The Advanced Detector Era (ADE) began this week with the start of the first observing run, O1, using the H1 and L1 Advanced LIGO interferometers.
Gabriela Gonzalez (Marco Cavaglia)
- The first Advanced LIGO Observing Run began Friday, September 18, 2015 at 8:00 PDT. Thank you to everybody who made this possible!
- Matt Evans (MIT) was reelected as chair of the LSC Advanced Interferometer Configuration working group. Thank you to all who participated in standing for these elections.
- Two new groups have joined the LSC: Kenyon College (PI: Madeline Wade) and the University of Alabama-Huntsville (PI: Tyson Littenberg). Welcome to the LSC!
- Badri Krishnan has been appointed co-chair of the Publications and Presentations Committee, replacing Steve Fairhurst. >* Patrick Brady and Peter Couvares have been appointed co-chairs of the newly formed Computing and Software Committee.
- Lisa Barsotti has been appointed chair of the newly formed Run Planning Committee (replacing the Operaitons Committee, chaired by Fred Raab).
- Jeff Kissel and Maddie Wade have been appointed co-chairs of the Calibration Committee.
- Andy Lundgren (replacing Josh Smith) and David Shoemaker have been appointed co-chairs of the Detector Characterization Committee.
- Joey Key has been appointed chair of the Education and Public Outreach, replacing Szabi Marka.
- Stan Whitcomb has been appointed chair of the Detection Committee, replacing Peter Saulson.
Congratulations to all new co-chairs and many thanks to past co-chairs for their service!
LIGO Laboratory Administration
Jodi Fauver (Dwayne Giardina, Nichole Washington)
- Worked on req approvals and a couple of procurements (garb laundry and dust monitors).
- Consulted with Ernest (CIT Property Services) about projects listed on M1500001 and a path forward. Hannah is doing some follow-up because we are still not on the same page. Also supplied CIT PS with information about new capital equipment.
- Firmed up our understanding of the GSA van for LLO: GSA POC in Baton Rouge will order the van for us in the Oct/Nov time frame, GSA POC will let us know tentative monthly cost when he knows it, I will be primary POC so I can pay the bill, Tim Nelson will be local POC for communication of recall or other information.
- Communicated with R. Abbot WRT asset management software candidates.
Reports & Editing
- Collecting annual report contributions
- Researching website feature stories
- Post doc advertising submitted
- Visitors/Guests processed
- Roster updated
- Staffing Committee meeting preparation
General Business Services
- Preparation of US Mail, FedEx, UPS, etc
- Daily sort/distribution/collection of mail and packages:
- Logistic support: moving, packing, reorganizing, etc
- Tracking request: L. Wallace Oracle incoming order, G. Billingsley incoming order, D. Kovak sent out package to LLO,
- Receiving packages, logging package information, packing slip handling and notification
- Meeting: LIGO meeting
- Office supply order request, reorganizing, restocking supply cabinets
- Special projects: TMDS packing with R. Taylor assistance using the crane move crate up to Synchrotron, finished shredding old files, Cart Maintenance new mirror, clean up of G. Billingsley lab 15 moving to synchrotron, C. Torrie- label property of cabinets in Synchrotron and start inventory cabinets, Updating cabinet keys labels
- Provided matrix support in supply inventory and vendor research
- Submitted payment requests for reimbursements.
- Deposited travel and reimbursement checks to the credit union.
- Provided Cardquest support to LIGO pcard holders.
- Reconciled pcard transactions.
- Completed and submitted paperwork for set up of new suppliers.
- Attended travel training presented by CIT new travel management company.
- Processed rush payment request and mailed the check to vendor with required original
- Calendar updates: LIGOMeetings, vacation & travel calendar
- Processing, reconciling Cardquest for Adhikari, Hiroto, Marx, Barish
LIGO Laboratory Safety
The LIGO Safety Audit Team conducted the LHO 2015 Annual Safety Audit this week. LHO should feel proud of their safety record this past year, as well as the overall state of the facility.
An all-hands safety meeting was also conducted by David Nolting that covered all incidents reported since last years audit, from 9/2014 to 9/2015, as well as encouraging staff and visitors to report Good Catches and the benefits of doing so.
During the LHO audit, the LHO management and audit teams discussed the progress of the LIGO Observatory Vacuum System Hazard Analysis. The goal is too have the hazard analysis completed and formally approved before the 2015 Christmas holiday break.
Reminder: The Monthly Safety Message is on Good Catches. Please read and send David Nolting an email indicating that you have done so.
Attended the LIAM face-to-face meeting Thursday and Friday.
Posted a phishing advisory.
LIGO Hanford Observatory (LHO) and Interferometer Operations
- B. Gateley, Facilities
- Fabricate and install motor mount reinforcement bracket on LSB chiller fan.
- Investigate SF-3 fan noise and vibration. Disable fan and lock out.
- Attend LHO annual safety audit.
- Adjust control room temperature settings.
- Daily system checks.
- The Coherent Mephisto NPRO we shipped back for repair is being
shipped back to us. Coherent found the problem as an
intermittent failure of a protection relay in the laser head and
has fixed it. Since it is coming from the UK it could take up
to 10 days to arrive. We will test it out as soon as it shows
- PSL Front End laser diodes are getting close to needing their
currents increased, in accordance with standard PSL maintenance
procedures (relevant DCC document: T1200560). We
(PSL team) are currently making plans to perform this procedure
sometime in the coming weeks.
- We plan to realign the in-use SUS optical levers.
- From E. King: Issue with the ETMy ring heater due to the
End-Y Beckhoff EK1100 EtherCat Coupler failure on 9/15/2015.
See LHO alog 21557
for full details.
- IFO Configuration
- Control SDF, observation run quality snaps
- Alarm Handler finalization
- Consult with Ops on earthquakes
- Consult with LLO for Tidal Relief
- Maintenance preparation and coordination
- HAM3 0.6hz investigation
- Ran a few sets of charge measurements on ETMX, processing
those and new ones taken during large earthquake downtime.
- Control room support - fixing ESD trips, monitoring
configuration control with SDF, assisting with troubleshooting.
Spent some time with operations on the effort to understand how
injection and event notifications come into the control room.
The HARDWARE INJ medm screen needs an update.
- Set up all SDF to monitor lock-to-lock stretch settings
changes via an standard set of OBSERVE snap files.
- Continuing effort on procurement of spare silica ears for
QUAD test mass and PUMs. Received back 4 that had the bonding
surface re-polished. One ear was chipped at the vendor during
- Finished hunting down and then sent one of each type of blade
from each subsystem (minus SEI which is being sent separately)
to CIT for further inspection/testing. 21 blades sent from BS,
QUAD, OFI, OMC, HXTS, SLC, TMS. Auditing missing blade records
- Calibration Group activities in preparation for O1.
Electronics, Control and Data Systems Operations
Had EtherCat chassis End Station 2 fail on Monday night. Unit was pulled and replaced with spare. IFO was able to relock using spare unit, but had some issues with TCS ring heater. Faulty unit had coupler that had failed. After replacing coupler, unit was tested on H2 teststand. Had some communication errors coupler terminals, but were able to resolve after power cycling chassis. Swapped unit back at EY, but had communication errors with chassis. Unit had to be powered cycled twice to clear all errors. Ellie verified that ring heater issues had also cleared.
Continue to work on backlog of chassis that need repair. Repaired S1105311 AI chassis that had issues with 18-bit DAC Anti-Image Interface Board. Optocoupler not switching states for DAC WD. Re-soldered all pins for all optocouplers which resolved issue.
Repaired S1201328 BSC ISI Interface Chassis. Replaced U1 regulator and moved jumpers on P7 and P6 to bypass relays on theISI Interface Back Board.
Repaired Coil Driver Test Box, replaced U3 LT1125
Vacuum Control System Upgrade is making progress with the assembly of 20 96 pin EtherCat Adapter boards.
Started installation of Solar Power system at the –250m port on the beam tube for powering instruments associated with the ION pump at the port.
Contacted Commvault and Arkeia for new CDS tape backup solution (Expecting quotes)
Removed dead hosts from Ganglia and reinstall the client and sections on new imaged and installed computers
Control Room Operators
OPERATOR WEEKLY OVERVIEW
Operator team continues to run H1 in the closing days of ER8 & offer H1 assistance where needed.
On 9/10 we had a training by Mike Landry on checklists for triggers & the Rapid Response Team.
OPERATOR OFF-SHIFT WORK
- Continuing Conlog troubleshooting
- Helping diagnose Beckhoff errors at the Y End Station
- Investigated issues with astronomical event updates from GraceDB
- Added and removed tests for the DIAG_MAIN Guardian node
- Discovered inconsistency in the use of the cdsutils avg module. This inconsistency prevents its use in the critical diagnostic node (DIAG_CRIT). Investigation ongoing.
- Began planning VerbalAlarms V2. It has grown much larger than the original design intended, this has limited its adaptability to new items and has given it a sub-par loop time.
- medm work on Ops_Overview and starting on new Inj/Event medm.
- Follow up work to PEM Injection coupling
- ETMy glitch investigations
- Outreach design work for LIGO Outreach Black Hole Master game
- Finished packaging and shipping of Quad spare parts to CIT for upgrade testing.
- Inventoried spare SUS blade springs and sent one per type to CIT for testing.
- Began working on assembling spare AOSEMs.
- Making travel arrangements for upcoming QUAD Pipeline Upgrade work at CIT.
- PSL/TCS Work: Helped Jason bag and remove spare PMC from optics lab.
- Ordered the last of the parts needed for the dust monitor upgrade.
- Worked with Jim B. on development of upgrade code for dust monitor operation.
- Work on order for manufacturing D1200971 OMC Lower Wire clamps spares.
- Review Garb cleaning RFQs and vendor responses with Jodi.
- Working on presentation for upcoming keynote
- Offered guidance on operator work schedule to LHO operator team
- Working on operator schedule with regard to new needs (2nd Operator on Maintenance Days & on-call operator for Rapid Response Team during O1).
- Working on travel for upcoming work trips.
- J. Hanks, CDS computing
- Closed bug 905 - dealing with SDF output precision. The SDF system now keeps a full precision copy of the data so that proper output is done.
- Working on a SDF IOC for slow controls monitoring. This is involves updating the common epics sequencer and investigating changes need in the build process.
- D. Barker, CDS
- Configuring LHO hardware injection machine to be same as LLO.
- Confirming all front end code and settings are under full configuration control pre-O1.
- The O1 observation run was formally begun 08:00PT on Sep 18. Turning over run coordination from ER8 was Vern Sandberg, who has done outstanding work in marshaling forces to bring the detector to a stable and sensitive configuration for the run. Observation mode was established in the ER8 phase, and we have been now running several weeks in this configuration. All the observatory staff including the detector engineers, operators, computing personnel, LSC Fellows, visitors and commissioners are to be congratulated in reaching the milestone of observation running with the new detector.
Education and Public Outreach
LHO outreach participates in a four-person panel that advises IceCube on education and outreach. The panel had its yearly meeting with IceCube personnel on 9/16/15.
LIGO Livingston Observatory (LLO) and Interferometer Operations
L1 Detection Coordination
The first observation run started this week. Many thanks to
all the local staff for getting us ready for observations.
The second quarter LSC Fellows are on-site and working both on their own
projects and on Data Quality and Detector Characterization issues
related to the O1 data.
The fellows are in the control room 16 hours a day during O1 (with help
from some local LSU and University of Mississippi colleagues).
In addition the Fellows will be part of the team that does rapid
follow-up for any alerts that we will send to our EM partners.
The fellows are:
Guillermo Valdes from University of Texas at Brownsville. Guillermo will
be working on monitoring TCS performance
(working with Aidan Brooks remotely). Mentor: B. O’Reilly
Jess McIver from Caltech. Jess will be studying seismic transients.
Mentor: J. Giaime
Ryan Goetz from the University of Florida. Ryan will be looking at RF
amplitude modulation of the beam exiting the input
mode cleaner. Mentor: M. Heintze
Filipe daSilva from the University of Florida. Filipe will work on noise
regression using PEM channels. Mentor: A. Effler
Alex Urban from the University of Wisconsin, Milwaukee. Alex will be
working on EM followup, GRB alerts and
Calibration. Mentor: J. Betzwieser
Control Room Operators
Gary Traylor (Richard Oram)
Tom Evans, Joe Hanson, Danny Sellers, Mike Fyffe, Doug Lormand, Jeremy Birch, Guimin Wu and William Parker.
The Operators Have Supported:
Commissioning, Calibration, ER/O readiness and site safety with 24hr work schedules. (Mike Fyffe, Gary Traylor, Jeremy Birch, Danny Sellers, Joe Hanson, Tom Evans, William Parker)
Edited the GRB EPICS record database so that the channels could be renamed during the maintenance Tuesday work. These channels will also be trended after maintenance along with another channel. (Doug Lormand)
HEPI maintenance checks in the corner were completed. (Jeremy Birch)
Preparation for HPO repair and inventory of usable components and researched components to assemble the remaining top guns for distribution around the site. (Guimin Wu)
Preparations for the start of O1. (Gary Traylor)
Computing and Electronics (CDS, etc.)
- Updated local GRB alert EPICS IOC for proper recording in DAQ.
- Rebooted failed Kerberos Domain controller (KDC)
- Helped Keith Thorne learn more details on network switch configuration
- HW injection - Answered user questions, tracked upgrades at LHO
- Timing monitoring - Identified few remaining items to be completed for pre-O1 rollout
- PEM - With William Parker, verified that readout of weather stations to EPICS database using Modbus was working. This still needs to be put in final, operational form.
- Operations - Recovered DAQ from mx driver errors on data concentrator. Finally tracked old bug with failed test points to host file errors on NDS server. Remote assistance with vacuum rack failure (bad UPS)
- Sys-admin - Configured new network switches. Web site updates for O1, helped Dwayne Giardian with logbook work. Prepared access and code update logs for cross-checks. Ordered supplies for raw trend RAID upgrade.
- Annual reviews for subordinates. Helped Carl, Michael with EOM connection to front-ends ADCs.
Carl Adams, Michael Laxen
- EOM Testing and Install Preparation
- PEM equipment repair and installation prep (Infrasound, rooftop antenna receivers, new microphones)
- DC Power Supply Monitoring investigation
- Shutter RMA and return
- Power supplies for NEG/Inficon vacuum gauges
Timothy Nelson (Richard Oram)
Fault report for this week. Each item listed has been identified by Facilities with a resolution. The requests have been completed or the work has been scheduled and is pending completion.
3641 LLO Facilities general WORKINPROGRESS — Install phone in pre-control room area by sink around the auditorium 08:07:56
3640 LLO Facilities general WORKINPROGRESS — Top cooling unit in guard shack seems inoperable - important for Science Saturday 10:53:12
3639 LLO Facilities general WORKINPROGRESS — Analog phone in control room not functioning 11:01:43
3622 LLO Facilities general WORKINPROGRESS — Replace Eyewash Station in VPW 07:52:05
3621 LLO Facilities general WORKINPROGRESS — Replace/upgrade Eyewash Station at X End Station 07:53:36
3604 LLO Facilities general WORKINPROGRESS — Turning off A/C, some banks of lights (but not fans) in TMS lab at y-end station 2015–09–10
3603 LLO Facilities general WORKINPROGRESS — One of the beam tube doors wide open 2015–09–10
3602 LLO Facilities general WORKINPROGRESS — Replace bulbs in back shipping and receiving bay 2015–09–10
3514 LLO Facilities general WORKINPROGRESS — Fountain Pumps 2015–09–04
3491 LLO Facilities general WORKINPROGRESS — move 2015–09–09
3434 LLO Facilities general WORKINPROGRESS — Y-Arm door(s) issues 2015–09–04
3394 LLO Facilities general WORKINPROGRESS — A/C unit in the guardshack 2015–09–10
3237 LLO Facilities general WORKINPROGRESS — fluorescent lights in exhibit hall & office need replacing 2015–09–10
3638 LLO Facilities general CLOSED RESOLVED Remove dead turtle from pipe outside of main conference room 07:50:08
3631 LLO Facilities general CLOSED RESOLVED Stock garb and gloves in racks outside of optics lab in Staging building 11:02:23
3629 LLO Facilities general CLOSED RESOLVED Y25 booster dehumidifier Wed 12:12
3625 LLO Facilities general CLOSED RESOLVED Flat tire on golf cart Wed 12:10
3617 LLO Facilities general CLOSED RESOLVED Move boxes to DCS storage 11:02:52
3572 LLO Facilities general CLOSED RESOLVED Assign time estimates for UPS famis tasks 2015–09–11
3545 LLO Facilities general CLOSED RESOLVED A sign added to the Receiving Hours sign on LIGO Lane saying “Dead End, No Turnaround” 2015–09–11
3464 LLO Facilities general CLOSED RESOLVED volume control panel not functioning
Projects and Enhancements
- Attended annual safety review in LLH this week. This was a very productive meeting. The great success of the safety program and safety awareness was highlighted and ideas and opportunities discussed to further heighten site safety awareness and aid in a continued positive safety record.
- During weekly HVAC PM, Facilities found a noisy air handler fan bearing on the X End. The bearing was lubricated then which helped, but the bearing will need to be replaced. This work is scheduled for the next maintenance period on 9/15/15. This is another good catch from our inspections helping to reduce issues that could have a negative impact on science. A window of opportunity opened up on Wednesday and the repairs were completed. There was noticeable reduction in sound being detected after the repairs were completed.
Major Infrastructure efforts
Construct new toilets in staging building 2nd floor (LLO)
Status: In progress. Completed preliminary design and drawings for Male/Female restroom in staging building. We received approval from the NSF (07/17/2015). Architectural Engineering firm in progress developing competitive bid package. Developing RFP.
Education and Outreach
- Conducted three on-site field trips for students. Kathy Holt, Tien Huynh-Dinh and William Katzman facilitated.
- Presented at the Girl Scout’s BIG event. Tien Huynh-Dinh facilitated.
- Helped facilitate film-maker’s presence across the weekend.
- Conducted one Science Club presentation. Kathy Holt facilitated.
- Ordered parts for exhibit repair.
- Presented at the Covington Rotary club. Amber Stuver presented.
- Conducted one tour for a Senior Citizen group. Tien Huynh-Dinh & Kathy Holt facilitated.
LIGO India Update
Arun Srivastava, Secretary, AEC, and Head, International Collaborations and Planning Division, our LIGO-India contact at the DAE of India, did not have further news last Thursday on the status of the cabinet note at the Prime Minister’s Office.
I am currently awaiting from Tarun Souradeep an indication of the status and plans for release to LIGO of the engineering consultants’ reports on two potential LIGO-India sites. This could affect the timing of a potential fact-finding trip to India.
The RRCAT team is preparing to review the mechanical drawings completed by the IPR team for the LIGO-India vacuum system to ensure compatibility of interfaces for future detector installation. Additionally the team is working on a laser pre-amplifier with a goal of achieving 35W operation by July 2016. A Reference Cavity is also under construction. Outside the LIGO-India team at RRCAT, there is an internal RRCAT committee that is vetting construction plans for their prototyping laboratory.
The IPR team has already sent vacuum system drawings to LIGO (MZ and JW) for comment. They will be sending these same drawings to RRCAT for their review in parallel.
There are now bi-weekly instrument science telecons occurring in India to share and broaden knowledge related to LIGO-India instumentation.
- Preparing annual report input for sustainment engineering activities in the Systems, Optics and Mechanics group [Dennis Coyne, Calum Torrie]
- Revised the conceptual design for a chevron baffle assembly for the ion pumps (see D1500305). [Eddie Sanchez, Calum Torrie, Mike Zucker, Dennis Coyne]
- Approved ECRs this week:
||ECR: Update GDS broadcast list for O1 (addition of PEM channels)
||Add PEM channel to science frames at LLO
||ECR: Remote monitoring of TCS chillers
||ECR: Addition of inline check valves for TCS chillers at LLO and 3rd IFO
Mechanics & Optics
1) Air Nozzle Vacuum & Optic Illumination System
Working on commissioning the vacuum system for the above setup. Torrie, Gushwa and Taylor.
2) Bounce roll mode damper (BRD)
Working on commissioning the vacuum system for the above setup. A single BRD is mounted inside our 4 ft vacuum system in order for us to measure the Q. Building a single pendulum setup for this effort too. Torrie, Robertson, Taylor.
3) Chevron Baffle
Working with Coyne and Zucker on the Chevron UV baffle for Varian 2500 l/s ion pumps. Sanchez and Torrie.
4)Lower quad in vacuum.
Working on setting up the feed-throughs, data loggers and electrometers that will mounted to the lower quad inside our 4 ft vacuum system. Torrie, Baric, Guswha. (Weaver coming to support us on build and install of lower quad.)
- The Request for Quote for gold coating of 2 Compensation Plates and 6 Annular End Reaction Masses was issued.
- We are coordinating vacating the 015 labs with the Division
- 4 ETMs have shipped to LHO for ear bonding - thanks to Nichole and Elizabeth!
- Currently working at University of Glasgow, catching up on what is happening here.
- Prepared input on the LIGO visitors program for the annual report.
- Preparing a talk on suspension-related risk-reduction work to give to the Glasgow group next week.
- Liaising with Betsy and Glasgow colleagues on ear production, and assessment of existing ears.
- We have completed the documentation for the preliminary design review.
- We are working on the final version of the optical layout and noise budget, which we need to proceed with the final design of the suspension.
- We are modeling the suspended table with optics and ballasts.
- Designing the oven and VOPO cavity optomechanics
- Preparing the drawings to send the RFQs
- Received a new fiber feedthru (4.5" CF, 1310nm and 633nm), which is being tested.
Control and Data Systems
- Preparing some modified ETM ESD drivers for shipment to LHO. The modifications will bring these three units up to the latest revision providing enhanced dynamic range and better frequency response of the monitoring channels.
- Evaluating various technical solutions to inventory management including:RFID, 1D and 2D barcodes. These technologies would be married to a software solution.
- Reviewing materials for Squeezing review this Friday.
NDS2 (client, server, and server infrastructure)
- Transitioned to the new O1 hardware for the production nds server at LLO.
- Identifying some DNS issues that cause access problems to the LLO NDS2 server. LLO is fixing the problem.
- Changed nagios test parameters to stop TIME warnings on the LHO server.
- Fixed an issue w/ the remote screen automatic updates at LHO.
Exploring effect of arm cavity resonance on DRMI signals.
modification of AA filters for the LSC to increase phase margin; its unnecessary to have a ~10 kHz AA filter when our digital system acquires signals at 64 kHz. We plan to measure the noise spectrum of the RF demodulator outputs out to 100 kHz to place requirements on the AA filter attenuation.
we continue to have hourly Frame builder crashes (this has been going on for several months); we are gearing up to replace the frame builder computer to see if its an issue with overload of our old Solaris system due to the increased channel load from various upgrades.
using a 2^14 frequency divider from Wenzel, we are making a digital (coarse) and analog (fine) frequency discriminator system to allow for low noise, high bandwidth, analog arm stabilization using our KAGRA-like ALS lock acquisition signals.
starting to incorporate Hang Yu’s python suspension balancing scripts to replace the Matt Evans / Vuk Mandic F2P scripts written in CSH/Expect.
Data Analysis and Astrophysics Lab-Wide Activities
- Lots of work on calibration review. Prepare report to JRPC:
- Lots of staffing issues.
- Lots of work on gstlal pipeline for ER8, with Kent, Surabhi, and the team.
- Detchar DQ shift all weekend, for L1 Sep 10–13, 2015; present on Monday detchar call:
- Lots of work on detection checklist development.
- Continued ER8 GSTLAL BBH Offline daily and weekly search pipeline runs.
- Requested access to Haswell head node at CIT after discovering it was inaccessible.
- Tested Makefiles for using the optimized GSTLAL and LALSuite build on Haswell hardware.
- Found some tools were missing and reported to system administrators.
- Updated Veto Definer File to new version after communicating with Laura Nuttall.
- Discussed some of the basics of GSTLAL with Rory Smith.
- Worked on issues with travel expenses and accuconference with admins.
- Provided bullet-ed list for astrophysical data analysis section of Annual Report.
- Commented on computational matchmaking spreadsheet, providing some recent accounting
Finishing up final runs for search methods paper (74h on-source regions for M82 host galaxy).
Created functionality in SMEE to run using condor.
Added 3D injection waveforms to SMEE.
Met with Siong and Jade to discuss updated arrival time prior based on oLIB/cWB triggers.
Ran distance and 10kpc runs for time domain PCs with updated prior.
Began preparing slides for LIGO seminar.
Reviewed Carl & Eve’s final SURF reports and got back with comments;
Ran several checks on scalar-tensor Crab injections and recovery–still waiting for some jobs spanning higher injection strengths to finish;
Started Crab agnostic search test run;
Still trying to figure out why TIGER doesn’t take advantage of Savage-Dickey ratios.
Advanced Distributed Data Analysis R&D
- Began working on the Haswell platform over the weekend to kickstart the evaluate
optimization of BBH offline gstlal search pipeline.
aLIGO Analysis and Open Data
- Building LOSC data pipeline for producing observation runs
- Working on the LOSC code for producing data release
- Building next release of myLIGO
- Attended auth Face2Face meeting
LDAS Software Systems
- Worked on fixing –data-valid option for framecpp_verify
- Added option –data-valid-all for framecpp_verify
- Integrated framecpp_compare into 2.4.2 release.
- Prepared release of ldas-tools 2.4.2 and deployed for Solaris systems and submitted to SL6 and Debian builders
- Continuing on packaging of the split version of the LDAS suite of tools
LDAS System Administration
Continued working on tape space issues at LLO.
Still working to fix the mistakenly (due to a misuse of sfind options)
relabelled tapes at LLO.
Threw away some unneeded boxes from Sync 212.
Got Oracle to install our new T10KD tape drive.
Oracle replaced the SL8500’s front panel.
Set up archiving for O1 at CIT and LLO.
Started to debug the problem with CIT’s drive 135, but can’t figure
out how it got like this. I have an email out to Marty from Oracle to
see what he knows.
Have been copying commissioning frames from the LHO/LLO to CIT using
Dealt with the situation where all NFS exports from samarchive ceased
working…tracked it down to a segdb database backup. Have asked Ryan
Fisher for more info, but for now /backup/segdb is not mounted on his
Notified user that was filling up /home5
(he’d added ~23.5TB to his home directory in the first few weeks of
Sept, for a grand total of ~45TB;he was able to clean almost all of it
up after being asked).
- Ongoing work on GW Event interface for Mike Landry.
- Had a couple of extended meetings with Mike Manske about coding practices standards.
- Ran further tests of running gstlal_compute_strain to produce hoft
- Set up copying of raw data, and generation of rds and aggregated hoft
data for O1.
- Verified trend, raw, rds, and aggregated hoft are up-to-date at LHO,
LLO, and CIT.
- Tested offline running of SegGen to update analysis-ready segdb flags.
- coordinated with Jonathan to bring new nds2 server into production
- configured x5–2 servers
- updated archiving configuration
- opened SR to replace failed E18 disk
- 9940 tape validation: temporarily on hold
Deployed new dcs-loghost on O1 hardware. Reconfigured all other ldas/dcs hosts to point to the new log host. Verified that log messages still make it to the Caltech log collector. Enabled fqdns in log messages so that they can be disambiguated at the central CIT collector.
Ran new ethernet and fiberchannel cables to the new Oracle servers. Updated ILOM and BIOS on all 6 servers. Installed OS and HSM-QFS on all 6 servers.
Migrated puppet reports to future new puppet server. Started archiving reports monthly to reduce disk space usage.
Continued working with Evan O. on new ldrslave server.
Added ER8 commissioning frames to archiver
Current 9940 tape files validation progress: 30.7% complete, 177/195 tapes, 11050727/36014311 files. (no progress from last week, but should resume next week)
- Answered questions, removed versions, reactivated accounts, delete documents, make documents public
- Work on the KAGRA provisioning script
- Work on RTs - reactivations, transfers
- Investigate why the new version of Grouper is not working at grouper-test.ligo.org is failing for enrollment.
- Participate in the F2F this Thursday and Friday.
- IT conference this week.
- Other misc.: Continued nsite software/phone support.
Failed hard disk on an Engineering workstation, Installed a new disk and reloaded this computer from scratch.
Worked in Lyuan's lab helping to get a lab computer setup for him to get connected to the laser.
Plus installed additional software for him as well and setup some drive mappings for him as well.
Started loading additional Engineering workstations
Order more supplies/equipment for GC.
Continued work on spam filters.
More that usual onsite/phone/remote user support, plus other daily sysadmin tasks.
Answered questions about using public vs non-public sites.
Help with the CMA use, made edits on the CMA.
Research on redundancy/recovery planning.
Research on performance testing tools. Forwarded info to Dwayne about an open source tool for load testing AWS hosted web applications.
- Pushed ligo.org updates to the production server.
- Prep work on migration of Lab roster to uranus.
- Ongoing web maintenance tasks.
- Wrapped up a number of maintenance contract renewals. Still have one more that is in process.
Purchased a number of computer support items. Most items have been distributed.
Wrapped up paperwork for reconciliation.
Attended the cardquest training for travel.
- Worked with a number of users in transferring or retrieving home account data.
Closed out a few accounts and setup a new account, went through orientation on new users.
Worked out equipment configurations for a couple of new users.
- Worked with Sharon on network equipment in Annenberg. Installed a couple of switches and some configuration work on the core 10G switch. Some minor setup and testing for the 40G ports on the switch.
- Minor logistics work on different meetings and conferences.
Update the BBB test box and will be running tests on the new version in the next couple of weeks.
- Tracked down a couple of boxes that had IPMI running on the regular network. Certain boards will change or reactivate the IPMI setup when network port connections are changed. Working with Abe to find a method to keep things locked in.
- Worked on issues with the ZFS appliance units. A board replacement has been received and now working on scheduling some downtime to replace it. Will also have to schedule a second short down time period to update the patches on the unit. These will be scheduled for after hours.
The temporary backups have finished. Also, have a new tech. working on the replication issue.
- Some minor cleanup and work in the different computer rooms. Also, working on some cleanup of one of the storage rooms.
- Reworked some items on the new spam-filter appliance. It is ready to install, just looking for a time to schedule the swap.
- Regular user assistance, spam work and table changes. Extra work performed on getting items resolved for those accessing the campus oracle services.
- Migrating business group data from CIT.
- Deploying replacement staff laptops.
- Warranty fixes on staff laptop.
- Updating OS X image for staff / conference room machines.
- Cleaning up old phones from VOIP system and preparing replacements for staff.
- Investigating VM performance issues.
- End-of-year PCard reconciliation.
- Reorganizing inventory in comm rooms.
- Cross-training CDS/GC with Carlos.
- CDS patches and investigating replacement config management infrastructure.
- Weekly patches checked/applied to servers.
- Daily work on spam filters searching for false positives and setting up new rules.
- Other misc. user support and daily sysadmin tasks.
- Setup for the LLO all hands meeting
- Auditorium mixer install
- Gathered access logs from Kantech
- D. Giardina training on downloading guard shack cameras
- Various user support
- Daily spam clearing
- upgraded ram on esxi hosts and added some more additional 2 port net cards
- redeployed ligo-sesimon from centos 7 to sl6 platform to allow install of ldas software
- continued work on epics server for Sebastien
- installed new w7 on xeon machine (and ordered ram to up to 32GB) for Alvaro, as he is
having issues opening up large solidworks models
- working on svn issue for Keith Thorne and others at LLO
- some git issues on lunar for Bobby, resolved
- cron issues for Michael on ligo-seismon, required install of nscd, resolved, working now
- misc purchases
- set up local mac backups for one of the grad students (CCloner via usb)
- working on understanding new clearone system setup
- upgraded ansys on bates machine
- chasing down some inventory items from property office audit
- worked on qnap cds backups
David Shoemaker (Stuart Anderson)
- Processing Procurements for the O1-reqiured DCS complement.
- Monthly report for August prepared and submitted
- Permission received from the NSF for procurements
Data Computing System (DCS)
- Orders placed for the last 2 LHO DCS facility items after receiving NSF approval (fire suppression and UPS)
- Tuning priorities of analysis jobs on computing clusters.
- Switched data services from ER8 to O1 at the start of the first Advanced LIGO Observing Run.
- Added data archiving services for the larger ER8 and O1 Commissioning frames until the Science frames are updated with additional requested channels.
Data Analysis Optimization
Worked with search chairs and pipeline developers to understand, document, and plan aLIGO computing demand and performance optimizations, and to compare recent usage with earlier projections. Appointed as co-chair of LSC Computing and Software Committee. Discussed aLIGO and LSC DA computing management plans. Worked with OSG and Georgia Tech LSC group to test cWB pipeline with OSG interface to Georgia Tech HPC resources.
- PyCBC management and telecons
- cpuopt clustering investigation
- Interaction between condor and CPU affinity
Data Analysis Optimization -- Operations supported
- Fiber routing/dressing for X5 and Nexsan systems connecting all HBAs to 6510-[1,2] in Annenberg
- elevation maps updated
- multi-mode optics verified to show link lights, HBAs show link, cables prep’d for systems in Synchrotron
- Brocade 6510-[1,2] switches installed, with Larry’s help, and labeled. Finiky power supplies but new power cables resolved the issue.
- continued 7750ICX testing with borrowed NICs and optics for 40G testing with HEP
- CUDA 7.5 build of pycbc+cuda completed - initial tests don’t break anything but I would like a more robust test so have contacted Syracuse.
- upgraded bios on microcloud node 631
- pulled motherboard from pcdev2, RMAd
- looked into TSX enabling issues - seems safest to just disable the rtm hle flags until Intel issues a microcode fix
- CardQuest Travel Portal training session (90 minutes)
- routine vendor contacting for HP 5412 v3 cards