This article is more than 1 year old
US schools' IT systems drop out after weekend firmware update misery
Three-day outage sparked by bungled software rollout for data-center gear
A botched network switch firmware update caused school IT systems to fall over in the US state of Washington, it is claimed.
WSIPC, which provides technology and software for classrooms in the Pacific Northwest, said its Skyward and IEP Online services were both out of commission for three and a half days this week while it remedied some problems within its data center.
"WSIPC continues to make progress in restoring full system functionality but it is still not operational," the provider admitted on Tuesday.
"You may experience access to certain systems, but until we can confirm that all systems are stable, please refrain from use."
Today, systems were apparently restored:
WSIPC is happy to announce that all system operations are now online and available for your use. You may still encounter some residual errors connected to ancillary functions like CAS or Print services for the short term but this will not impede your ability to access your systems.
We understand the inconvenience this outage has caused and appreciate your support and patience as we worked to resolve the issue as quickly as was possible.
A Reg reader familiar with the situation told us the outage hit school services, such as email and student record-keeping for some Washington schools.
Weekend IT upgrades
The cause for the issue, it is claimed, was a weekend firmware update to some of the Dell networking hardware WSIPC uses in its data centers. When things didn't go as planned, services went down, and apparently it wasn't easy to clean up, as WSIPC said it would actually need to replace the knackered switch gear.
"Two emergency response support teams, Dell Co-Pilot and Microsoft Premier Support, will be onsite and working closely with our Technology Services team to install and configure the new equipment," WSIPC told customers on Tuesday morning.
We've asked Dell for comment on the matter, including some insight as to what exactly went wrong and whether other enterprise customers should be concerned of the same thing happening in their datacenters. So far, the Round Rock giant hasn't anything to say on the matter.
Similarly, WSIPC has not given any further guidance beyond its blog posts, failing to respond to a request for comment. Any admins or IT staff in the Evergreen State with info on the outage and its impact on schools are welcome to contact El Reg anonymously. ®
Biting the hand that feeds IT
