LIGO Weekly Report for the week of 17 April - 21 April 2017
Past Weekly Reports
The LIGO Executive Committee Agenda for Monday 24 April 2017 will be:
(Meeting time: 10:30 Pacific Time/12:30 Central Time/13:30 Eastern Time)
- LSC - Shoemaker/Cadonati
- Commissioning - Fritschel
- Operations Management Team (OMT) - Raab
- Hanford - Landry
- Livingston - Giaime
- India - Raab
- Administration - Hansen
- Engineering Optical and Mechanical - Coyne
- Control and Data Systems - Bork
- Lab Computing - Anderson
- Instrument Science and 40M - Gustafson
- Data Analysis and Astrophysics Lab-wide Activities
- Weinstein, Katsavounidis
- MIT - Fritschel
David Shoemaker (Laura Cadonati)
The Council voted in favor of the position of Deputy Spokesperson for Laura Cadonati along with an initial assignment of responsibilities. Further discussion of the motion for changes in the bylaws to establish this position in the long term will be needed.
Jo van den Brand of NIKHEF Netherlands was voted the new Virgo Spokesperson and will take on the role 1 May.
LIGO Laboratory Administration
- Reviewed purchases for proper expenditure type designation and completed equipment screenings
- Processed equipment receiving, inspection, identification-tagging, movement, storage and disposition requests
- Managed all logistics activities on campus
- Assisted with international imports from Germany and UK
- Hands-on assistance with ICS training, transfer of e-waste, tracking of GSA documentation
- Preparation of capital inventory at LHO
General Business Services
- Calendar updates: LIGOMeetings, Travel calendar
- Travel reconciliations: Barry Barish, Rana Adhikari
- Processing, reconciling Cardquest for Barish, Conf & Meetings card, Hiroto, Adhikari
- Staffing committee meeting prep and after
- Reformatting Committees lists
- Reviewing Staff Service Awardees list
- Processing Diversity Committee minutes
- Reconciling LVC expenses and income
- Data entry for postdoc data base
LIGO Laboratory Safety
Signed work permits for LLO task.
The LLO 2017 Annual Safety Review was conducted on April 18–19. A walking tour of the facility revealed very few findings…mostly housekeeping issues that are common. (AL note: many thanks to the LLO team for hosting this year’s review and being diligent in safety matters).
It is most important that whenever a task is to be performed, no matter how common or repetitive, it is always smart to first submit a work permit describing the activity. Submitting a work permit is a great opportunity to communicate among the work groups and team, of your intentions so conflicts may be managed and to ensure that the correct staff are involved in the activity.
Report “Good Catches” to your supervisor or at the following URL: https://safety.ligo.caltech.edu/index.html. When we report a “Good Catch”, we give ourselves the greatest opportunity to correct unsafe conditions, equipment or even risky work behavior before an incident occurs.
- Investigating JupyterHub security
- Rewrote the site Cyber Security Training presentation
- Participated in 2nd round interviews for postings 16396/7
- Preparation for LLO visit the week of 4/24
- Continue updating LIGO Open Source software policy
- Review CIT web server allow/deny policy
LIGO Operations Management Team (OMT)
OMT reviewed the status of preparations for the commissioning break during O2 and the planning process for post-O2 work. The break is on track to begin the week of May 8. The break will likely affect current plans for the post-O2 work, which will be followed further.
LIGO Hanford Observatory (LHO) and Interferometer Operations
Hugh Radkins and Jim Warner
- LVEA floor tilt/pinning study continues
- 3IFO & H1 ICS for SEI
- Seismon work and Blend switching upgrades–JWarner w/
(DET ENG) MAY 8th Vent Planning
- Writing the Vent Plan.
- Working with Gerardo to bond ears to spare test mass ITM06.
S3 was successfully bonded with an ear yesterday, we plan to
work on the second ear next week. Note, this optic will not be
ready for any monolithic work until ~May 25th, however, there is
no plan to need it.
- Prepping tools and kits needed for the vent.
- Reviewed procedure, then cleaned and First Contact’ed the HWS
in-vacuum lens needed for the vent.
- Worked through resurrecting the Clean and Bake Ticketing
system with Calum and Chandra in order to ensure user interface
was easy and useful.
- Worked with Peter K to inspect the 2“ optics on the HAM4
in-chamber HAM table via looking in through a viewport. This
was in attempt to convince the team that we do not have any
other possibly errant 2” optics installed in the HWS path. We
could not make any 100% certain statements, but we did find
evidence of things being correct with the inspection. Results
alogged. Attended weekly call.
- Attended weekly call regarding upcoming baffle plans for
- PSL maintenance. See LHO alog 35611
for full details.
- Increased HPO pump diode currents.
- Diode Box 1 continues to decay faster than the other 3 HPO
pump diode boxes.
- Temperature tuned HPO pump diodes.
- Tweaked beam alignment into PSL PMC.
- Participate in review of PSL/TCS cooling systems with an
- Weekly reset of PSL power watchdogs.
- Monitor health and performance of system.
- More issues with new ITMy oplev laser. See LHO alog 35614
for full details.
- Went out to tweak output power of laser to eliminate
glitches, found laser running very warm.
- Suspect this laser is drawing more current than other oplev
- Checked other oplev lasers in LVEA, none running warm.
- Will replace this laser at first opportunity, as the
temperature it is running at is not good for the laser and
makes eliminating glitches extremely difficult.
- Will likely use laser SN 189–1, as so far it is showing no
signs of suspected ~0.44Hz feature (see next bullet point).
- Lab testing of laser SN 189–1 (removed from ITMy oplev for suspected
feature at ~0.44Hz).
- To date have found no evidence of ~0.44 Hz feature in the
- Measuring spectrum of laser output with a signal analyzer,
all clear so far.
- Completed testing of 9 3IFO oplevs lasers for functionality.
- Results: 5 are non-functional and 4 turn on but exhibit very
low output powers.
- Getting quote from manufacturer for repair/refurbishment of
- Monitor health and performance of system:
- ETMx and ITMx oplevs still glitching.
- Once ITMy oplev is taken care of, will move on to these.
- Participate in review of PSL/TCS cooling systems with an
- Monitor health and performance of system.
- Continued work exploring detuning of the IFOs response to
- Installing / commissioning further infrastructure
- Attempted manipulation of detuning to test tracking
- Updated documentation with as built methodology
- Restarted high-frequency sensing function sweeps via relevant
guardian code change
- Finalized draft of Calibration section of 2019–2023 Lab Coop
- Cleaned up / reorganized requirements documents in the aLIGO,
CAL branch of the DCC document tree
Control Room Operators
OPERATOR OFF-SHIFT WORK
- Inventory gloves
- Working on plan to remove and replace expired gloves
- Ship repaired cleanroom sock to PCS for cleaning
- Assist LLO with getting the LHO dust monitor code
- Check and adjust dust monitor vacuum pumps (FAMIS)
- Building supports to mount PSL canister filter
- Receive parts for and build the PSL Chiller Air Bleed/Trap
- Getting PSL Chiller Air Bleed/Trap parts ready for shipment to LLO
- Finish the BOM for the PSL Chiller water manifold.
- Attend chiller systems water quality conference call
- Needed to fill an Owl Shift.
- Completed a personal work outreach project of filming Sharon Yellowfly (my mother) speak the Siksika (Blackfoot) translation of the GW150914 LIGO Press Release. The video is now on the LIGO Virgo YouTube page here.
- Continued HWS alignment work
- Wrapped up TCS maintenance guide document – will continue to update it as I go
- Generic TCS maintenance
- PCal End X calibration
- Worked on fiber display for Outreach
- Tooling prep work for upcoming vent
- Finished and tested the python SEI blend switching script that will be deprecating the old perl script. The new debian machines cannot run perl and since the medms call the perl script, we need to replace this asap. If tests go well for LLO and Stanford we will implement the change next week.
- Working on scripts to generate a grid of Guardian screens. I’m trying to plan ahead for the large number of new nodes that will be added post O2 and the lack of space on the Guardian Overview screen.
Daniel Sigg (Keita Kawabe)
No commissioning due to O2.
This week we have spent about 10.5 hours for planned commissioning in coincidence with LLO. Observation mode duty factor of H1 from Fri. Apr/14/2017 00:00:00 to Thu. Apr/20/2017 24:00:00UTC was 78.2%.
GDS calibration code that was put in place on Apr/11 for a bug fix introduced another bug where a momentary ETMY timing error makes the calibration status latched to bad even after the problem goes away until DMT calibration process is restarted. Due to this there was about 6.5 hours of good observation time that was not analyzed by the low latency analysis (alog 355569). After failing to quickly revert back to the old version on Apr/18, for now the operators are instructed to restart the DMT calibration code as soon as the problem is detected (alog 35675) though this is not something that happens frequently. We'll revert back to the old version during the next maintenance period. In the meantime, a new version with long term fix was developed and is under testing on a separate DMT machine.
Education and Public Outreach
- Attended monthly Collaboratory council meeting
- Exhibited at the annual Safety Connect event, booth facilitated by Richard McCarthy, TJ Shaffer, Mike Landry and Amber Henry
- Hosted 2 field trips facilitated by Amber Henry, Kim Burtnyk, and Elizabeth Castrellon
- Hosted private tour facilitated by Eric Garcia and Amber Henry
- Attended 2 elementary school science nights
- Conducted a job shadow facilitated by Dave Barker, Elizabeth Castrellon, Mike Landry and Amber Henry
- Attend community science celebration booth facilitated by Jeff Kissel
- Completed lobby photo display with help of Chris Soike
- Completed instillation of video monitor for drop in visitors with the assistance of Ryan Blair
LIGO Livingston Observatory (LLO) and Interferometer Operations
L1 Detection Coordination
We fell short of 70% duty factor from Wednesday to Wednesday.
The main impacts were the failure of the ETMX PUM (Penultimate
Mass) driver which took ~28 hours to recover from (mentioned in last
weeks report) and an extended commissioning break to re-align the
Hartmann Wavefront sensors
I have been working on mode matching calculations for the experiment
that suits the latest optical layout. The calculation for the optical
path to the SHG is based on Lee's solution with a couple of small
adjustments to allow for a laser with a slightly different waist. The
calculation for the path to the other components are based on the
manufacturers specifications and documentation from MIT. The final
optomechanical mount for the in air table top experiment has been
designed sent for fabrication.
I have started to learn about the
lock-loss tools but have yet to use them productively, I'll talk to
Shivaraj and/or Adam today about the best strategy for their
I added some more noise sources to the noise budget summary page. I also
made a python module to help me add the rest of the noise sources, so
hopefully that will speed things up a bit.
With the suspension watchdogs project, I have been working on trying to
access larger quantities of data. Ideally we would access about 2 years
of data. I spoke to Duncan about this and he showed me some ways of
getting the data more quickly. The first step was to move my code onto a
computing cluster, which I’ve now done. Then I started looking at minute
and second trends rather than the full data, which speeds things up a
little more. Next I need to edit the code to use some of the gwpy tools
that Duncan showed me, which should speed things up substantially.
- This week I have wrapped up the ISS glitch investigation. The conclusion being that we
think the dropouts were as a result of the ISS loop being close enough
to instability that it occasionally fell over and caused a lock loss. Two
months ago the gain of the loop was decreased at the same time that the
diffraction power was increased this fixed the problem but the analysis
suggests it was the change to the gain that provided more stability. The
outcome is that we will lower the diffracted power to get a slightly
more power to the interferometer and monitor the ISS for about a week
and see if the same glitches return.
- This week I made substantial changes to the pydv codebase to improve
the efficiency of pulling multiple channels from the NDS server at once,
and expanded the command-line syntax to allow multiple channels to be
plotted on each subplot in the tool’s output. I’ve started working on
tests for my foton and simulink parsing tools, which revealed a few
problems in the code which I’ve been working to correct, and I’ve
implemented some methods to reduce the complexity of the graph which is
generated from the Simulink model, to allow paths to be identified
through it faster. I was the on-site fellow supporting the DQ shifter at the
Janeen Romie (Richard Oram)
Reporting for the last couple of weeks for the LLO Detector Engineering Group: Stuart Aston, Matt Heintze, Adam Mullavey, Arnaud Pele
Supporting O2 interferometer alignment, debugging and locking efforts. On-call Detector Engineers are available to Operators 24/7. Provided general support in maintaining and commissioning the interferometer. Recovery of IFO after Tuesday Maintenance.
Planning commissioning break tasks and post O2 vent tasks, see E1700109, which is a work-in-progress. Reviewed with OMT Tuesday.
Aligned the corner station Hartmann Wavefront Sensors (HWS), per T1700163. Images and technical details may be found at alog 33205 and alog 33230. The SLED power for the “X” arm was much less compared to the “Y” arm. The MEDM screens indicated about a factor of 2 and using a power meter it actually appeared to be even worse than this. This meant trying to align the x-arm was definitely harder than Y. This data will tell us if we have any large spot absorbers like they see in Hanford see alog 33228
Aligned the FSS double pass AOM, as the amount of light transmitted through the reference cavity was about 3.3V max as opposed to the nominal 4V max. After realigning both the FSS double pass AOM and the reference cavity the FSS_TPD locked, per the MEDM screen was 4.4V and the visibility using FSS_RFPD_multi meter values = (1-(0.2/2.0))*100 = 90%. Details in alog 33179
Debugged and resolved an issue with ALS Y not re-locking, see alog 33137. The real issue was a problem with the cdsutils avg function that Keith hopes to fix next Tuesday. However, the ALS guardian should be coded to handle such issues.
Tested a new OMC ASC scheme, designed to give us more suppression at the microseism (0.2Hz) and the quadruple pendulum frequency (0.45Hz), which are the main contributors to the RMS (alog to be posted soon). Unfortunately, this new scheme also injects large broadband noise into DARM. At this point it's not clear where this noise comes from. We will continue our investigations next week.
T&T Laser Lab work is progressing. Water cooling lines were installed for the 300W power meter head and 70W amplifier diode box which is cooled in series with the front end laser. Laser fiber inspection is ongoing, see TNT logbook 11230. Interlocks were tested and qualified. Starting alignment of thee NPRO into the 35W laser.
Processing and analysis of continuing charge measurements to monitor the charge on the test masses. On this occasion to verify that charge situation had not changed following a nearby lightning strike and power glitch. Details may be found on alog 33202
Tweaking the ITMY OpLev operating power to try and avoid glitching looks to have been successful, but will continue to monitor, alog 33204
Investigate HTTS OM2 UL weaker OSEM channel. Suspected electronics chain in CER, but routing through known good (OM1) Coil Driver made no improvement. Therefore, suggesting the fault resides higher up the electronics chain with the AI chassis or DAC. It was decided to postpone further debugging until post-O2, but to monitor the affected channel for any further degradation in performance. Details may be found in alog 33162.
Updating and preparing B&K equipment for pending vent work at Y-end. Gathering components for the preparation of the installation of L4Cs into HAM1.
Recommissioned the SEI huddle test stand to test the L4Cs Joe H finished assembling recently, as well as the one that will go in HAM1 chamber, see alog 33184. All tests show that the 6 L4Cs we tested are healthy. Work is ongoing to get the SEI pod neon leak test stand back up and running.
Installed filters that Brian L. provided in SEI alog 1144 in order to test common low frequency control of the corner station platforms. Wrote a script in order to switch to the new configuration in one command line.
Computing and Electronics (CDS, etc.)
- Switched from ntp to chrony on SL7 hosts
- Helped recover from brief power outage. Identified, debugged and fixed bash issues with CDS setup scripts that caused autostart problems with EPICS gateways.
- Built new SL7 rpm for gwpy–0.4
- Started testing Amanda backup software on slow-controls computer
- Updated awgstream-matlab package to include prebuilt .mex file
- Upgraded X end station HWS computer to SL7
- Cleaned vtapes 13 & 14 on tape backup
- Installed SL7 to new opsportal server and set up two-factor authentication. Replaced the old opsportal. Created wiki documentation for setup of opsportal
- FAMIS Tasks - checked the FE IPMI sensors
- SL7 workstation updates. Pushed out awgstream-matlab update on SL7 workstations
- Installed python-psutil package on SL7 workstations
- CDS Tuesday maintenance work (Filter loads, Guardian checks)
- As part of maintenance, rebooted Guardian script server to remedy slow memory leak issue.
- Made improvements to MATLAB nds2-client scripts to improve robustness based on user testing.
- Finished restarts for VME vacuum controls to remedy large ion pump read backs
- Purchasing for CDS operational supplies, Updated CDS webpage with links to LIGO/Virgo working-group logbooks
- Got SEI 'huddle' test stand back online for L4C testing
- Helped recover SUSAUX front-end from DC power trip due to power transient from thunderstorm. Recovered a SUS front-end from interruption due to electronics investigations
- Traced problem with slow NDS query in Guardian script to debug prints in pre-production nds2-client installation. Will be remedied in maintenance/commissioning break.
- Supervise Electrical Contractor in performing Electrical Maintenance Tasks at site:
- Convert Fluorescent fixtures to LEDs in Laser T&T Lab and (2) Offices per Facilities
- Replace Photo Cell on Wall Flood Light on outside East Wall of LVEA
- Review Electrical installation and order Material for the installation of the two A/C Units at SEC
- Pull Electrical Materials from Spares for the installation of the Condensing Unit at the CSB SHP/RCV. Order additional material from Vendor
- Attend Laser T&T Lab Meeting. Review LASER Lab Interlock documentation submitted for approval by Matt Heintze and provided comments
- Address Grounding Issues at X-End: Josh smith reports no Glitches seen in DARM after the minor change to the Grounding made on Apr 11th. Closed FRS and Work permits
- Attend All Hands Safety Meeting: Give talk regarding Electrical Safety with emphasis on Hot Work Permitting procedures
- Check SEC power for possible surge when operating Stage lights. No apparent issues found - will keep issue open for a few days and monitor
Carl Adams, Michael Laxen
Laser T&T Lab Work:
- Mod interlocks to allow HPO and 70W to operate concurrently
- DB1 slow-controls troubleshooting
- Update slow-controls code for HPO and 70W operation
- Investigate using new slow-controls PC for Laser T&T specific code
- Cables, Cable mapping / documentation (BOM)
- Qualified interlocks
- IFO Troubleshooting - PUM Coil Driver X-End
- As a test. moved I/O chassis fans to separate power-supply to attempt noise mitigation. Avoided need for front-end reboot.
- Squeezer Electronics - RF Distribution Amp Assembly
- RMA of equipment to vendors
- RF Measurements at the end stations
Timothy Nelson (Richard Oram)
Fault report for this week can be reviewed through aLog link below:
Each item listed has been identified by Facilities with a resolution and requests have been completed.
Projects and Enhancements
The 15 ton HVAC system was not cooling in the Staging Building Optics Lab. Facilities’ Dave Barker found an overheated frequency drive creating a fault condition which shut down the system. The drive’s cooling fans are inoperative. Dave installed temporary cooling fans and restarted the system. Permanent repairs will be completed once parts are received. All repairs completed by LLO staff generating a cost savings of several thousand dollars and reduced down time. See FRS https://services.ligo-la.caltech.edu/FRS/show_bug.cgi?id=7930
Facilities found and repaired an ongoing “supervisory” fault with the site’s fire pump monitoring system. The fire pump #2 power module had failed, causing the fault condition. The module was replaced and the system is normal. All work was completed by LLO staff saving several hundred dollars and prevented extended down time of the site fire alarm monitoring. See FRS https://services.ligo-la.caltech.edu/FRS/show_bug.cgi?id=7830
Completed 2017 Safety Review at LLO. With the exception of a few very minor issue, all site conditions including work areas, equipment, and tools were found to be in safe, working order. An “All Hands Meeting” was held after the Safety Review with the entire LLO staff to go over all findings and discuss ongoing, good safety practices.
Education and Outreach
- Conducted two K–12 school field trips. Facilitated by Tien Huynh-Dinh, Kathy Holt, William Katzman & Amber Stuver.
- Conducted a tour for the participants in the Southeastern University Research Association meeting. Facilitated by Joseph Giaime, Tien Huynh-Dinh, Kathy Holt & William Katzman.
- Conducted a tour for college students. William Katzman & Kathy Holt facilitated.
- Scheduled summer groups.
- Conducted the monthly Science Saturday, where LIGO Livingston is open to the public. Facilitated by Tien Huynh-Dinh, Shivaraj Kandhasamy, Richard Oram, William Katzman, & Kathy Holt.
LIGO India Update
Upcoming visitors from India:
Week of Apr 24
- P. Sreekumar, Director of Indian Institute for Astrophysics and member of the LIGO-India Scientific Management Board, is coming to LHO for the PAC meeting, but arriving several days early to learn more about the nature and scope of observatory operations.
- We are hosting a visit by the MP from the district containing the primary selected site for LIGO-India and other officials to learn more about LIGO. Ajit Kembhavi, Director Emeritus of IUCAA, who was in the US for other matters, will join us as well.
- Later during the O2 run, we expect to host members of the DCSEM team designing the LIGO-India facilities.
- We expect significant numbers of LIGO-India colleagues as long-term visitors to both LIGO observatories during the post-O2 work activities, to use these activities as a training opportunity for future LIGO-India installation and test activities. First step of identifying the visitors will be followed up by identifying how to integrate these visitors optimally in the various work tasks.
Ongoing work within Lab:
- Export control preparation;
- Risk registry refinement;
- Refinement and assignment of LIGO-India support roles;
- Estimation of LIGO-India support for remainder of this Cooperative Agreement and the FY19–23 CA.
Mechanics, Optics & Vacuum
- AERM06 Harness bonding is complete.
- CP04 AR reflection and absorption measurements are complete. The total reflectivity of CP04 is around 200 ppm, about twice the value we would like to have. Side one reflectivity has changed since it was measured by LMA, this is being investigated.
- Worked on funding status and job adverts for two summer students.
- Tuned BRDs to match the prototype beamsplitter here at Caltech and measured their quality factors. Also worked with Eddie on a revised design of BRD steel springs with thicker pyralux which potentially will give us more damping if/as needed.
- Worked on answers to questions for an article to appear in “Science Scotland”, a publication of the Royal Society of Edinburgh.
- Stray light baffles: preparing laser damage experiment (Alena Ananyeva, Calum Torrie)
- LIGO Voyager: looking for options to test emissivity of coating samples within 5–50 microns range (Alena Ananyeva, Calum Torrie)
- CRiMe: Annealing, inspecting, weighting and shipping coated waters and substrates (Alena Ananyeva and Liz Natividad)
Control and Data Systems
- Developed a set of Delphi scripts to work with Altium. These scripts make it easy to automatically generate rack and component location reports in wiring diagrams, as well as to generate a cable pull-sheet with cable lengths.
- Continuing squeezer hardware build
- Luis has tested a low noise 4 channel digital potentiometer for general purpose chassis use and is ready to send out the main board files on a new PDH locking box.
- Luis continues to support the SR3 heater testing in the 40m lab.
- Working with Todd and Rolf on the cheaper IO chassis. Rolf has verified that the new boards are functional within a prototype chassis. A power distribution board was designed and built, and is now integrated within the IO chassis design. There are no indications of the timing glitches that seemed to be a problem in the last design.
NDS2 (client, server, and server infrastructure)
- Getting the client ready for the release
- Working on build issues
- Responding to sccb queries
- Working on server questions and the alternate server implementation
- Improving error messages when we have ambiguous channels
- Working on rate limits and timeouts
Data Analysis and Astrophysics LabWide Activities
- Finishing FY17Q2 report from data analysis group and LOSC.
- LOSC Event tutorial updated in preparation for planned release in the next month.
- Continue assembling waveform templates for LOSC with best fit parameters.
- Working with SURFs and Caltech undergrads on data analysis projects.
- Meeting with grad students about thesis work progress, candidacy exams, etc.
- Staffing work.
- Continuing to learn my way around LOSC with tutelage of Roy.
- Met with TJ to discussed his issues posting summary pages on LOSC.
- Continued analysis of large memory leaks in PE codes.
- Began working with the PE nightly test DAG as it had been reported as a reproducible
run for seeing the memory leak, and confirmed it is.
- Discovered that both nest and mcmc inference codes are affected, but not to the same
- Reported finding from memory leak testing at this week’s PE group call.
- Ran nightly test DAG without ROQ (required finding and modifying source code) and found
that in this case it is the use or ROQ that triggers the memory leak problem.
- Succeeded in integrating lalinference_nest into Intel’s InspectorXE memory tool; this
is running now to build an analysis of memory issues.
- Looking into ways to integrate the MPI based lalinference_mcmc into Intel InspectorXE.
- Attended usual set of weekly working group meetings.
- Investigated data quality issues at LHO and LLO
- Updated working draft of LOSC summary pages
- Followed up loudest PyCBC background events
- Worked on updating CBC DQ paper
- Started writing single-detector search code
- Worked with Tomas, an undergrad, on running Hveto to look for source of MAINSMON glitches
Advanced Distributed Data Analysis R&D
- Continue to work with PE group to track down the memory leak issue in their pipelines.
aLIGO Analysis and Open Data
- Teaching new member of LOSC dev team (K. Blackburn)
- Designing new authorlist web for myLIGO with M.Manske
LDAS Software Systems
LDAS System Administration
Continued work rearchiving of T10KC -> T10KD and added some T10KB ->
T10KD as well.
O2 tape validation up to date as of last week, this week’s run started.
Got a script going to rsync (pull) 40m data from nodus to /hdfs.
Testing right now, but should be able to put it under cron soon.
Lots of work to get /home7 in production, including rediscovering that
forcedirectio is bad for NFS performance.
- Stress tested vmhost1 to possibly identify bad DIMM, if any
- Installed and enabled puppet on samhome7
- Allocated dedicated Online_CBC_PyCBC_Live nodes for Alex
- Re-mirrored the replaced Samsung SSD drive on ldas-gridmon
- Decommissioned losc-notebook.ligo.org and updated local DNS accordingly
- Patched and rebooted the LOSC systems
- Assisting Jamie with additional setup tasks on jupyter
- Fixed syslog forwarding on login2 to boo
- Created machinelearning account and assigned group membership specified by Marco
- Updated local monitoring for the segdb and segments-s6 systems
- Migrated figaro and its services (DHCP, NTP, http, mail relay) to the old-jiminy
- Rebuilt the vacated figaro system as popeye running SL7 to become the primary DNS server and local LDAP with overlay
- Managed Grid-enabled user accounts
- Requested and installed IGTF, InCommon/Comodo and/or LIGO-CA SSL cert
- Tested the latest version of the calibration code.
- Verified raw, trend, and hoft data are up-to-date at LHO, LLO, and CIT.
- Followed up on several data issues.
- applied workaround to fix kernel vulnerability while avoiding having to reboot nodes
- reserved a Skylake node (lldetchar) for benchmarking tests
- created shared account for machine learning group
- updated LDR monitoring page, removing ambiguous labels
- patched and rebooted dgx–1, installed VNC and supporting software for easier off-site access
- continued repackaging Hartmann data for transfer to Caltech
- LLO LDAS Account Activations
- With Michael T, installed new l1dmt3 machine
Successfully rearchived and validated 2 ER10 tapes that were reporting errors.
Created machinelearning account and initial set of users who belong to the group.
Enabled tape drive autocleaning in acsls
With Alyssa B, installed new DMT server to help validate new releases of the DMT/calibration software.
Started copying original CDS raw trend files into HDFS for final merging.
Applied additional ssh security settings for shared accounts
ER10 tape validation: 100% complete.
O2 tape validation: Up to date modulo incomplete tapes
- Changed document access levels, updated document permissions, make documents public
- Troubleshoot user access
- Upgrade software in the backup systems.
- Set up the PnP master-slave database
- Assisted users with various DCC related questions
- Talked with Fred Raab regarding LIGO-India documents
- Worked on RT tickets: assisted with member transfer process, reapplied for a user with a misspelled username
- Cleared up some space in authcore
- Met with Roy and Mike regarding the proposed author list generator
- Participated in the panel for East LA College’s forum on Women in STEM.
- Upgraded to the latest version of SolidWorks on the Main / Backup PDM servers.
- Upgraded LHO PDM server SW2017.
- Upgraded SolidWorks on most of our Engineers that are using SW at CIT, plus I was able to fix other issues with additional software packages including Windows updates. - Updated SW2017 on Pleione and install latest security patches.
- More work on configuring a network switch for Ruckus wireless.
- Misc. user support onsite/phone and taken care of issues remotely.
- Continued work on blocking spam and adding additional rules to the Spam filters.
- Work on/testing of various components of the internal Lab website.
- NSF Review webpage updates, initiated the process of setting up an authorize.net account for the event.
- Work on the new Lab roster.
- Amaldi12 web support.
- Ongoing web maintenance tasks.
- Had to address an issue of reducing the size of my inbox as it was nearing its limits.
- Multiple support item orders. Finally able to get a maintenance contract canceled, the company wanted justification for not renewing the contract.
Ordered a new notebook computer for one of the users. It should arrive next week.
Worked on a number of property items and now tracking down items to update property info. records.
- Some h/w troubleshooting on a couple of servers, mostly cabling and disk issues. Also, some work on a couple of notebook computers, disk issues.
Minor printer work.
Reset cabling and switches in a couple of the offices to clear up problems people were having.
Some minor computer moves and h/w modifications on security systems.
- Worked on conference rooms. There is a temporary webcam setup in Downs1, presently looking at different cameras for the conference rooms. Did find a 30ft. usb2 cable that would work with the mac-mini and camera setup.
Power strip for Downs3 conference room has been ordered.
Mike is building a test box, so we can update the OS on the mac-mini’s in the conference rooms in the next month or so.
- Quite a few changes on the service tables.
- Updated and patched the spam-appliance. Also, updated a number of rule sets.
- Assisted a couple of users with their mail-client configurations.
More phone support than normal, this past week.
- Patched a couple of server and client boxes.
- DNS server config changes ahead of migration and cleaning up cross-site zone transfers.
- Migrating applications off of older CentOS machines (site admin, cron-mail, DNS, webmail).
- Diagnosing mail server performance issues.
- Converting users from POP mailboxes.
- Reconfigure multi-purpose room AV for upcoming events.
- Briefly testing some LSC and CDS software builds on Debian testing release (Stretch at this time).
- Building display computer for Outreach LSB video screens.
- Replaced user desktop.
- Complete replacement of one of CIT IDS boxes.
- Diagnosing performance issues with other IDS machines.
- Building and testing filters to allow IDSes to ignore bulk data flows (LDR/GridFTP traffic).
- Investigating KDC kadmin/kpasswd limits.
- Puppet4 installation for security, GC infrastructure at campus.
- Converted logging to machine-consumable format for automated, live indexing.
- Reviewing IdPv3 upgrade/new installation recipe in advance of swapping out login5 or login3 (both at LHO).
- Weekly patches checked/applied to servers.
- Security server maintenance, alerting/incident follow-up.
- Other misc. user support and daily sysadmin tasks.
- patched and rebooted most GC servers and workstations
- received replacement core switch.
- brought contractor back in to provide upgrades / new features for ICS. this consisted of SOW writing, review, discussion with contractor, and asking Melanie McCandless to submit the PO. first batch of upgrades was deployed 4–21.
- submitted request to have contractor’s LIGO.ORG credentials reactivated.
- investigating a problem with export to CSV from FRS
- guard shack cameras web page was not showing the stream from the cameras. I patched and rebooted the server that creates the streams. page works.
- added an LHO user to the LLO spam filter list of white-listed senders
- modified SSL settings on two servers to increase their score
- with Matt Cowart, investigated problem with projector in auditorium. DVI over Cat5 converter will likely need to be replaced soon.
- fixed a problem with the work permit site not printing “clean room move” checklist when work permit has “clean room move” selected.
- gave overview of AWS setup during GC meeting.
- gave two demos of work permit system to LHO.
- set up work permit site for LHO usage.
- in ICS, corrected an assembly record that had incorrect status
- planning MPR projector and screen replacement
- set SEC doors for Science Saturday, returned to work hours after.
- assisted a user with FRS email preference settings
- upgraded the server running FRS and the work permit sites to a newer OS. Had to troubleshoot config file changes, and a few missing packages after the upgrade. minimal downtime experienced. no users reported errors.
- found the reason the replacement ICS server was randomly giving stack trace errors. changed a few settings and tested thoroughly. I will move the production version of ICS to the new server next week.
- found the reason a user was reporting problem creating links in aLOG posts. the browser was loading a cached version of the “new report” page that was several weeks(months?) old. a forced cache refresh cleared the issue.
- worked a few other procurement items
- attended weekly meetings
- other misc user support
- Continued work provisioning new wifi equipment
- Upgraded SolidWorks around the site
- Troubleshooting psl laptops and remote access
- Support for two presentations
- Attended weekly GC meeting
- Misc user requests
- some teamspeak work
- configuration work on Erik’s new laptop
- yum update issues EK workstation – centos 5.1 obsolete
- set up a couple of skype/teamspeak conferences for postdoc interviews in interaction area
- configured/ordered ws for Bobby/Nancy video capture purple lab
- took apple monitor to hw services for stand repair
- some work on seismon and epics
- linux ws in matlab cluster bad os hdd, getting replacement ssd from thinkmate
- issues w Ken workstations; updated, fixed registry issues, stable now
- set up thunderbird for Rai
- specing out notebook for Nancy
- updated linux kernels
- fixed some latex issues on sqwiki wiki (moin moin)
- upgraded solidworks and vault to 2017 sp2
- set up Fabrice’s laptop for use during his vacation (solidworks, Ansys, remote connections)
- some laptop issues for John, resolved
- authentication issues on qscan; ldap & shells; fixed
- updated cds new fileserver
- tweaked nfs on reinstalled (ubuntu -> sl7 ) cds workstations
- moved cds vpn from ws4 to server
David Shoemaker (Stuart Anderson)
Helped with the quarterly report
With Dave and Stuart and the NSF, established that there will be a 6-month no-cost-extension of aLIGO to allow the best use of computer procurement funds.
Data Computing System (DCS)
- Interviewing candidates for optimization position.
- Provided information to the NSF on aLIGO procurement options
- Finished FY17Q1 NSF quarterly report.
Data Analysis Optimization
- coordinated current and future LIGO optimization work with data analysis pipeline developers and optimization team staff, including the PyCBC offline CBC search, the SkyHough pulsar search, LALinference CBC parameter-estimation, and the PowerFlux burst search.
- participated in hiring interviews for two open LIGO computing positions.
- worked with Open Science Grid staff to discuss current issues and future plans related to their support of LIGO data analysis computing
- worked with Satya M. (MIT) to enhance the LVC data analysis computing usage accounting system, identifying and reporting bugs, incorporating Open Science Grid computing usage, checking our calculation of each LIGO Data Grid site’s computing usage to accurately reflect the use of CPU Hyperthreading, and accounting for dedicated computing resources not managed by HTCondor.
- continued to enhance the new LVC Data Analysis Council science priorities spreadsheet
- met with Syracuse University IT management to discuss future GPU hardware plans
- met with Georgia Tech High-Performance Computing center (PACE) staff to discuss technical and policy issues with the their LIGO data analysis cluster use via the Open Science Grid, including the transfer and storage of LIGO data, job scheduling, and support for publishing internal LIGO data analysis results.
- agreed to speak at the HTCondor Week 2017 meeting in Madison, WI in May.
- agreed to serve on the Steering Committees of two proposed SI2 Software Institutes of major High-Performance Computing (HPC) centers.
- discussed the upcoming Open Science Grid “Summer School” with OSG staff, strategized who to target within LIGO, and publicized it internally.
- ad-hoc LIGO / Open Science Grid (OSG) systems administration duties
- debugged HTCondor scheduling issues on the LIGO Data Grid
- prepared for a LSC Computing face-to-face meeting this summer to assess future LSC computing needs and plan directions.
- worked with the National Center for Supercomputing Applications (NCSA) LSC group to help them run the PyCBC offline CBC search pipeline on the BlueWaters supercomputer via the Open Science Grid.
- reviewed a useful LSC policy exception to allow the use of PEM channel data in a computing tutorial.
- debugged service monitoring issues with the aLIGO Segment Database (segDB).
- received word that the LSC’s most computationally-demanding and best-optimized search pipeline (PyCBC) was written up in the March 2017 issues of Physics World: http://tinyurl.com/k5mkdfx.
- received word that the “Data Access for LIGO on the OSG” paper I contributed to was accepted to the ACM’s PEARC17 conference.
- Extensive consultation and assistance helping Eliu Heurta at NCSA run PyCBC on the Open Science Grid from the Caltech cluster, as a trial of using that mechanism to run on Blue Waters.
- Debugging problem with PyCBC bundled executable in release 1.7.0 (missing a needed library)
- Continued work on SkyHough optimization, this time combining the sparse matrix optimization with the previous memory optimization (still in progress).
Data Analysis Optimization Operations supported
- Dell FX2, TwinPro2 and Skylake servicing (disks, memory, kickstarting)
- Dead Datanode maintenance
- Vendor RMAs
- Finally found proper power cable for 4th GPU card to reside in pcdev11, will deploy and test 4/21
- Interviews (on site and via skype) for two LIGO job postings
- Continued working with vendors for custom X2200 risers, PCIe x8 custom card options, etc.
- Continued working with vendors and CIT facilities for strainer blowing and flow balance for row 5 in Annenberg
- cWB _avx_norm_ps loop studying for upcoming SURF project
- Final stages of pycbc timings/graphs (round 0)