Power supply units destroyed in surge
A lightning strike to an overhead electrical line caused a campus-wide power and Mudd Data Center outage Aug. 18. An additional Data Center shutdown is scheduled for tonight through Saturday morning, and another will occur overnight Sept. 9-10, Director of Networking, Telecommunications and Operations William Deigaard (Wiess ‘92) said.
The scheduled outages will affect all core campus services including Internet, e-mail, file service, ESTHER and network services on and off campus, Deigaard said in an e-mail sent to the All-Departments listserv Tuesday. IT staff and server owners will begin to shut down systems at 9 p.m. Systems will be fully operational by 7 a.m., the e-mail states.
The shutdowns are necessary to replace two 20-year old Uninterruptible Power Supply units the power surge destroyed, Deigaard said. The UPS units supply power to computers, networking and mail servers in the Mudd Data Center. The old UPS units were scheduled to be replaced even before the outage, Deigaard said. The two new units together cost about $150,000, in addition to installation costs. Deigaard said the new units are more technologically advanced, will be easier to service and will have better surge protection. Since the UPS units were destroyed, battery backups in place could not prevent the outage, he said.
The power outage occurred at 4:28 p.m., when the power surge through an overhead electrical line tripped the main circuit breaker at the main high-voltage switchgear building. The Rice-owned line is one of the few exposed utility lines on campus, Assistant Vice President for Facilities Russell Price said. A backup generator allowed campus phone lines to remain operational, Deigaard said.
Price said Facilities, Engineering and Planning may consider ways to further protect the exposed line from lightning.
“We’ve had a project in the past to replace that overhead line — maybe bury it or try to protect it somehow — but it’s such an enormous project, it’s been one of those things [we] just have been putting off,” Price said. “We’re going to look into that again and see if there’s any way to protect that overhead line from lightning.”
Power was restored to most of the campus, including the colleges, in about 30 minutes. Power was restored to the entire campus by 6:45 p.m., Price said.
Restoring power to the west end of campus — Rice Stadium, Rice University Police Department, Rice Media Center, Continuing Studies and Autry Court — took the longest because the overhead line that was struck carries power to that part of campus, Price said.
“We always have to check the system out thoroughly before we re-energize, and that’s why sometimes it takes time,” Price said. “We just don’t want to turn the power back on [and cause more damage].”
Power was restored to Data Center computers by 1:30 a.m. Aug. 19. All the servers had to be rebooted, and most systems were running by 3 a.m. The system was made operational by wiring temporary bypasses around the disabled UPS units. However, some systems still ran slowly, and lost data needed to be reconstructed from backups.
Deigaard said the abrupt shutdown of the computers is responsible for any lingering problems.
“[The unplanned shutdown] is one thing that’s caused the difficulties of this past week because the systems were not shut down cleanly — because they had the power yanked out from under them,” Deigaard said. “There have been various straggling problems here and there, and it’s been a little slower than it needs to be.”
Other problems were caused by the power outage. Several people were stuck in elevators at Fondren Library, McNair Hall and Sid Richardson College, Price said. The emergency power in McNair Hall malfunctioned, leaving stairwells and the underground garage dark. The Central Plant, which provides water pressure, chill water and steam, took several hours after power was restored to become fully operational, Price said.
Orientation Week activities were not affected significantly, Campus-wide O-Week Coordinator Alex Paul said. The Aug. 18 Athletics rally was cancelled because of the power outage, Paul, a Jones College senior, said. A brief Data Center outage Aug. 19 disrupted new-student class registration for about an hour.
Price said the timing of the power outage — just before the end of the work day — allowed staff members to work on the problem immediately.
“During these emergency situations, it’s good to have good technical people on our staff,” Price said. “We teamed up quickly to troubleshoot the situation and determine what was wrong and what was not wrong and restore the power very quickly.”
Other news stories
- All buildings to have wireless Internet within 18 months
- Almost 300 volunteer in Outreach Day activities
- Hypnotist entertains at Big Owl Bash
Sports
- C-USA provides competition, rivals
- Soccer hopes to win C-USA title
- Volleyball picked to win Conference USA
Arts & Entertainment
- 'Song' proves sex and rock 'n' roll a volatile mix
- Artistic passion lies at core of thrilling 'Heart'
- New 'Potter' installment lives up to hype, sure to win over fans of all ages
- Summer belongs to reality divas