Edge computing platforms provide double redundancy and reduce system failover time by 90 per cent
23 March 2022
It’s the vital dual service that virtually all residents of any municipality rely on every day — but that hopefully goes unnoticed: drinking water consumption and wastewater disposal. Water utilities are the foundation of any city or town, but for the District of Columbia Water and Sewer Authority (DC Water) in the US, the challenge is magnified by that fact that it services some of the nation’s most important buildings, including the White House.
The scope and scale of the mission have led to a massive infrastructure that pumps more than 99 million gallons of water to 681,000 residents through a web of 1,350 miles of pipes, four pumping stations, five reservoirs, three tanks, 43,000 valves, and 9,500 fire hydrants.
With a total service area of roughly 725 square miles, DC Water also treats wastewater for 1.6 million people in the District as well as neighbouring Montgomery and Prince George’s counties in Maryland and Fairfax and Loudon counties in Virginia. The centrepiece of those efforts is the 150-acre Blue Plains Advanced Wastewater Treatment Plant, the largest facility of its kind in the world. It treats an average of nearly 290 million gallons per day — with a peak capacity of nearly 1 billion gallons per day.
Thanks to a sophisticated SCADA system operating on Stratus Edge Computing Platform, the agency can achieve the availability goals that its large water system requires. To make this all possible, InSource Solutions, a digital transformation company and Stratus distribution partner, facilitated DC Water’s new solution.
Critical importance of SCADA to DC Water
Operating one of the largest water utilities in the country, it’s no surprise that a strong supervisory control and data acquisition (SCADA) system is a non-negotiable requirement for DC Water. The agency relies on the AVEVA System Platform for secure, real-time automation, operations, and control of its water pumping stations, storage facilities (tanks and reservoirs), sewage pumping stations, stormwater pumping stations, and fabridams on the Potomac River.
The AVEVA implementation spans databases called Galaxy Repositories, two redundant AVEVA Historian instances, two application object servers, and two terminal servers that run AVEVA InTouch® HMI for visualisation. DC Water operates these applications in three control rooms.
Resilient edge computing architecture
Unsurprisingly, with so much at stake, DC Water has designed a sophisticated edge computing infrastructure to minimise or eliminate any downtime. “The one thing we absolutely cannot tolerate is a failure in any of our servers,” said Karen Green, SCADA manager for DC Water. “If we’d implemented regular servers and experienced any sort of failure, we would be dealing with downtime — pumps and valves not working, water not reaching our customers. That’s obviously unacceptable.”
Each DC Water site — from storage facilities and tanks to pumping stations to sewage facilities — has multiple programmable logic controllers (PLCs) from both Rockwell and Schneider Electric that feed data to a central server through a control network running on private fibre optic. To build in high levels of system redundancy in its computing architecture, DC Water operates a separate management network driven by two physically separated Stratus edge computing platforms.
Optimised for distributed edge-in architectures, Stratus edge computing platforms deliver the performance needed to support advanced processor- and data-intensive applications, while also providing the fault tolerance, security, analytics, and manageability DC Water requires.
“Our AVEVA Historian instances are for data collection,” Green explained. “We push data from our Tier 1 Historian to a Tier 2 Historian where users across the authority can access various data including pump statuses, water pressures, water analysers and power information. We’ve configured them as store-and-forward servers, so that if there’s any interruption, the data will backfill when the connection is restored.”
Complete and valid data
Naturally, the frontline users are the dozens of people who operate the equipment at remote sites using AVEVA InTouch® HMI. SCADA analysts like Green configure the various SCADA nodes and analyse the data to optimise performance. “Our analysts and mechanical and maintenance teams also use that data,” Green said. “The maintenance team, for instance, uses the historical data and alarms for troubleshooting and to prevent recurrences. We configure the reports, and they are automatically emailed to engineers and managers.”
Members of the water quality team can monitor their equipment and receive alarms and notifications on their phones if variances or failures arise — an important consideration for people managing water and wastewater systems. “Of course, we also have upper management who are looking at the KPIs, and regulators like the EPA want to see our data,” she said, “so it’s crucial to have reliable and valid data that’s complete and timely. This all comes from Historian and our SCADA software that reside on our Stratus edge computing platform.”
Stratus edge computing and support
DC Water operates a Stratus Edge Computing Platform, backed up by another (physically distant) Stratus edge computing platform, creating two layers of dual redundancy. This configuration gives DC Water’s sprawling operation exceptional levels of reliability.
“With Stratus, we have fault tolerance and redundancy,” Green said. “If one server fails because of a power outage or communications disruption, the other one takes over from another location and there’s no downtime. This wasn’t always the case. In previous iterations, it might take 20 minutes for the entire system — AVEVA System Platform, InTouch®, and Historian — to failover. Now, with a different configuration, our software/network failover is negligible.
“We’ve found that virtualisation is very helpful to our agency. For example, if we need to perform maintenance, updates, or repair work, we can easily move one virtual machine to another server without any interruption. In fact, the other day, we had a problem with some PLC software. We just loaded it on the Stratus edge computing platform, and that made things very easy for us. With virtualisation, we don’t have to maintain an individual box for each server. We just have, basically, one large box that can run many virtual machines. We get one level of redundancy because Stratus edge computing platforms have dual processors working in parallel. We can also backup, duplicate, and restore the VMs whenever we need to.”
Green also underscored the importance and value of the support she’s received from Stratus. “That’s important — very important,” she said. “In our line of work, we need responsive support, and Stratus’s engineers are right alongside us on every issue. Even if it’s a non-urgent issue, they will call me to follow up until the issue is resolved. They’re very responsive and diligent — and that’s just what we want.”
More information on Stratus Technologies may be found here.
Contact Details and Archive...