An in-depth study of data-breach problems last year where hackers infiltrated 312 businesses to grab gobs of mainly customer payment-card information found the primary way they got in was through third-party vendor remote-access applications or VPN for systems maintenance.

"The majority of our analysis of data-breach investigations -- 76 percent -- revealed that the third-party responsible for system support, development and/or maintenance introduced the security deficiencies exploited by attackers," the Trustwave report published today states. The vast majority of the 312 companies suffering the payment-card breach were retailers, restaurants or hotels and they came to Trustwave for incident response help because Visa, MasterCard or another payment-card organization had traced a batch of stolen card cards to their businesses, demanding a forensics investigation within a matter of days.

In fact, only 16 percent of the 312 companies managed to detect the payment-card data breach on their own, says Nicholas Percoco, senior vice president at Trustwave and head of its SpiderLabs division. Most of the time, sophisticated analysis by the payment-card organizations of a large volume of fraud reports from customers about unauthorized credit-card use was the trigger for the call from Visa or MasterCard to investigate a suspected breach.

Percoco said forensics investigations did show there had been a data breach in all 312 cases, with about 29 percent of the attacks against these businesses traced to originating in the Russian Federation. However, a full 32.5 percent of the attacks had wholly unknown sources since they originated through Internet anonymity services.

Although the businesses hit by payment-card hackers claimed to be compliant with Payment Card Industry (PCI) security standards, in reality there were often gaps. The third-party vendor remote-access applications and VPNs used for systems maintenance were often the way attackers got in by stealing the simple, reusable passwords in use.

The Trustwave reports notes, "System logins require a username and password, and often these combinations are pitifully simple: administrator:password, guest:guest, and admin:admin were commonly found in our investigations. Many third-party IT service providers use standard passwords across their client base. In one 2011 case, more than 90 locations were compromised due to shared authentication credentials."

Percoco says the PCI standard for remote-access administration requires two-factor authentication, which wasn't being used. Percoco notes that these IT systems vendors at fault did have a price to pay. They were not only required to fix the issues identified, but also faced fines for noncompliance with the PCI standards and Percoco adds, ordered to "pay to recover the costs of the fraud."

The Trustwave report reveals some shocking statistics. Where it was an outside organization, rather than the business itself, that pushed for a forensics investigation, "analysis found that attackers had an average of 173.5 days within the victim's environment before detection occurred." Businesses that did so-called "self-detection" to detect attackers on their own did a little better -- the hackers only spent an average of 43 days inside their networks after the initial compromise.

And in a case from Europe last year in which a payment service provider was hacked and multiple servers and a wide-area network of more than 1,000 hosts were attacked, Trustwave says it identified the "single point of weakness as a legacy X.25 node."

The X.25 protocol, which was widely used in the 1980s to build wide-area networks, still finds use today with financial institutions for inter-bank data exchange, the report states. The attacker in this case "identified an internal development system and proceeded to re-write a well-known rootkit on the HP-UX operating system. The rootkit was then installed across a number of cardholder data processing servers to mask the presence of other malicious programs introduced by the attacker."

Trustwave says the "malicious scripts harvested cardholder data by terminating the legitimate instances of payment-processing software and then restarting the software with a Trojanized-debugger attached. The debugger captured all inter-process communications including unencrypted payment card data from within the system memory, which was otherwise encrypted when at rest on the disk and in transit on the network."

This attack went on from almost 18 months and the "attacker was only identified when a subtle flaw within their own customized malware alerted the payment service provider's operational staff to suspicious activity."