This everyday piece of hardware crashed an $11 million drone

It’s the little things that make all the difference, like when an $11 million MQ-9 Reaper drone crashed out of the night sky because of a small metal washer that you can buy at a hardware store.

The washer was not on the aircraft itself, but inside the ground control station, where it came loose and created an electrical short that sent erroneous commands to the Reaper, throwing it into an uncontrolled final landing in the early morning of April 10, 2019.

The good news is the incident should be the last of its kind: newer ground control stations have a different configuration to prevent loose metal bits from screwing with its sensitive hardware.

“This discrepancy has since been remedied on the affected equipment to prevent similar incidents,” Alexi Worley of Air Combat Command Public Affairs told Task & Purpose “Additionally, the newer GCSs have a different [Control Console Assembly] configuration that eliminates this condition.”

All seemed well at first when the Reaper took off at about 8:30 p.m. somewhere in the U.S. Central Command’s area of responsibility, which covers the Middle East and central Asia. It’s pilot and sensor operator, meanwhile, were sitting at a ground control station at Creech Air Force Base, Nevada, home of the 432nd Wing.

Little did they know that at some point before the mission, two flat, metallic washers and a nut had come loose within the ground control station. It’s not clear how that happened, but after the mission the bits of hardware were found resting in between sensitive pins on the station’s control console serial module (CCSM), the mechanism which converts analog signals from the pilot’s control stick, keyboard and trackball into digital commands for the Reaper.

Still, everything was fine at first. The aircraft flew normally for more than five hours, then entered what’s called a point-and-click loiter, where the Reaper’s autopilot keeps it flying around a specific position while supporting a mission.

It was at this point, shortly before 2:00 a.m., that the CCSM sent several erroneous commands to the Reaper, causing the engine to shut down and the propeller to stop. The pilot and sensor operator initiated emergency protocols to try to regain control of the aircraft, but the Reaper was still receiving erroneous commands from the faulty CCSM, and soon the crew’s screens went black as the aircraft plummeted to the ground.

Related: The militants fired an RPG at a C-130. They didn’t expect the MQ-9 Reaper to respond

What went wrong? An investigation after the crash revealed that two washers and one nut had come loose and fallen onto the CCSM itself. There is no cover on the CCSM to prevent loose metally bits from falling onto the sensitive equipment, the Air Force investigation board found.

Furthermore, that corner of the ground control station is inspected only every 168 days. There are other regular inspections that happen at shorter intervals, but they do not require opening the CCA. In a cruel twist of fate, this station in particular was due for its next 168-day inspection the day of the crash, investigators said.

Investigators noted that the Reaper’s software fault logic also contributed to the problem. It failed to recognize or reject the conflicting commands as a failure within the CCSM, which then made it impossible for the pilot to regain control of the aircraft.

The good news is that the new ground control stations moved the CCSM from under the pilot’s controls to behind the station’s monitor, where it hopefully won’t be rained down upon with loose hardware.

The story is a classic example of Murphy’s law: whatever can go washer, will go washer. Plan accordingly.