Blog Post by Rick Pierro, Superior Controls president
As Superior Controls’ automation engineer, Mike Demboske, plunged his luggage into the backseat of his waiting Uber, he thought about his upcoming lengthy and exhaustive trip to Wuhan, China. As he crawled through morning traffic towards Terminal E at Boston Logan Airport, a strange mixture of adrenalin, anxiety and despondency surged through his body.
The trip would start with a mind numbing 21-hour trip to Beijing, punctuated with a six-hour layover in Hong Kong Airport. Mike would then sit in Beijing Capital International Airport for two hours before boarding the 737 for an additional two-hour flight to Wuhan International Airport. After landing in Wuhan and running the one-hour Chinese customs gauntlet, he would climb into the taxi for another 45-minute battle through traffic and construction to the Wuhan, Hilton. Once checked in, Mike would collapse from exhaustion onto his hotel bed as he contemplated the finality of his grueling 33-hour, door-to-door journey. And then, just six hours later, he would begin another three-week Site Acceptance Test of the process automation system Mike and his team had so carefully designed and programmed that would grow and produce critically needed therapeutics for his Wuhan based biotech manufacturing customer.
Site Acceptance Tests are attached to the Biotech world in the same manner that Super Bowls are attached to football. It’s the final trial, the culmination of all that work and effort; a solid, documented, definitive and unambiguous proof that the automation system functions exactly in accordance with the customers specification. All control valves open and close at the right moment, heating systems maintain their desired set points within preset thresholds, pumps and flow rates are properly controlled and monitored and finally, all critical data and alarms, called GMP data, are properly collected, stored and made available to the manufacturing operators to ensure that the product is being produced properly and consistently. Site Acceptance Tests are important. They are the Super Bowls of automation projects.
And so, when the pandemic hit, coincidently originating from Wuhan, Superior Controls engineers could no longer travel to the customer site and had to find another way to test. Another way to play the game.
With critical automation projects scheduled for Site Acceptance Tests throughout the world, Superior Controls engineers had to find a way to remotely implement and document them with the same certainty, confidence, and thoroughness while simultaneously avoiding the possibility of interference from malware, software viruses or cyber interference. Furthermore, the documentation and formal signatures signifying acceptance of each process test by multiple parties located in different states or continents had to be arranged.
More recently, while Boston was still in pandemic lock down, Mike’s teammates completed a two-week startup in China without leaving Massachusetts. There was no 33-hour grueling trip, no sitting in Hong Kong for six hours and no battle through traffic and construction to the Wuhan, Hilton. In fact, the automation team never even left their house.
The Superior Controls automation team had three computer screens set up in their home offices. One screen displayed the formal written 80-page Sequence of Operations and 48-page Site Acceptance Test. These documents had been downloaded to DocuSign and were also being simultaneously reviewed by the customer in China, the engineering firm in Pennsylvania and the consulting firm in Boston. DocuSign provides software that enables companies to manage and sign electronic agreements that can be stored in the cloud and signed by multiple parties in multiple locations. As each test in China was performed and the correct results noted, all four witnessing parties signed and dated their document, presumably from the confines of their comfortable home offices.
The second computer used a ruggedly secure Virtual Private Network (VPN) to communicate directly with the primary operator interface in the customer’s control room. This computer replicated and displayed everything the onsite customer could see: graphics, recipes , events, alarms, the state of batch reactors, the state of all instrumentation and sensors in the facility. As each test was performed at the biotech facility, all four parties -customer, consultant, engineering firm, and Superior Controls – could see what the automation systems said was happening in the factory.
The third computer was connected to a portable camera held and guided by an onsite manufacturing technician during the entire acceptance test. This computer also used Microsoft Teams to enable all connected parties to see and talk with each other during the testing. With the camera continuously active, all parties were able to see the positions of valves, motion of pumps, and read the pressure, temperature, and flow rates from the local instrumentation and sensors to confirm that the readings matched the computer generated operator display readings.
In two weeks, the customer, consultant, engineer, and automation systems integrator, all residing in various locations around the world, successfully tested and established that this new manufacturing facility worked properly. Furthermore, they were able to provide the customer with the signed legal documentation that the FDA, and most governing bodies for therapeutics, accept as proof that the system was tested properly.
The Pandemic has certainly changed our life temporarily in many negative ways. But perhaps, there will be some permanent benefits coming from this misfortune. Superior Controls engineers have already implemented multiple successful remote, Site Acceptance Tests at Chemical, Food and Beverage, and Biotech facilities throughout the world. Grueling 33-hour traveling marathons for a Site Acceptance Test may become a thing of the past.