Author: Amira Armond, the president of Kieri Solutions – an authorized CMMC Third Party Assessment Organization (C3PAO) providing CMMC assessments, CMMC consulting, and Compliance Documentation packages designed for small/medium business.
This graphic depicts my personal experience talking with defense contractors about their 800-171 and CMMC compliance (and what score they entered in SPRS) over the last three years. This is non-scientific and is based on anecdotal evidence only.
The blue line is the “actual score” – what the company SHOULD be reporting for their compliance score in the Supplier Performance Risk System (SPRS).
The orange line is the “reported score” – what the company actually reports in SPRS.
What are the trends in 800-171 reporting?
At least half of defense contractors shouldn’t submit a SPRS score (technically “not applicable”).
According to the Department of Defense’s NIST SP 800-171 Assessment Methodology, a defense contractor should not report a score at all if they do not have a fully written System Security Plan.
The sad truth is that we have a huge cybersecurity skills shortage, especially for the topic of Governance, Risk, and Compliance (GRC). GRC experience is the baseline expected in order to write a System Security Plan. (want to become that GRC person? check our article and video on how to write a quality SSP)
How many defense contractors have someone on-staff with GRC experience? The answer is: almost none if they are small businesses. Not many if they are medium businesses. Only the large defense contractors have been consistently throwing resources at this problem.
What % of defense contractors are small? 67% according to Defense.gov.
These statistics match up with my own personal experience talking with hundreds of defense contractors over the last three years.
The reported SPRS scores become more accurate as real scores improve
There are two reasons why reported scores start matching real scores as cybersecurity improves: use of GRC specialists and incentives to falsely report.
Governance, Risk, and Compliance (GRC) Specialists
The first reason why scores become more accurate is that companies doing 800-171 / CMMC correctly will have that GRC expertise that I just mentioned above.
Companies need someone who understands GRC in order to determine if you are correctly performing the cybersecurity requirements. Many of the requirements have deep expectations that aren’t obvious to an untrained eye. Having a non-cybersecurity business owner or salesperson perform their own assessment and scoring typically results in MUCH higher scores than should be assigned.
If a company uses a GRC specialist to assess their compliance and calculate a score, it is much more likely to be accurate.
If a company doesn’t have a GRC specialist involved, their real score will go down (because they don’t know how to be compliant) and their reported score will go up (because they incorrectly think that they are performing the requirements).
Incentives to report a false SPRS score
Let’s look at two imaginary contractors:
WidgetsUSA reports a score of “110” – a score that indicates they are doing 100% of the requirements.
FreedomCorp reports a score of “3” – a score that indicates they are doing about 60% of the requirements.
Which company is actually more secure?
Which company doesn’t know what they are doing, or is falsely reporting?
Hopefully you are thinking like I think – that the company with a lower score is being more accurate, and likely is more secure in reality.
However, large primes are treating this in the opposite way. If you (a subcontractor) report a low score to your large prime, they will penalize you by investigating your company and potentially removing you from contracts. The contractors that report a perfect score get an easy pass.
The system incentivizes ignoring cybersecurity responsibilities
Contracting companies know that the odds they will be audited by the DoD are extremely low (not even 1 %). When they calculate the cost of being compliant (sometimes more than the yearly revenue of the entire company), a significant number decide to take their chances.
And know what? Those companies that are falsely attesting without paying for cybersecurity are the ones that win the contracts.
Allison Giddens from WinTech (a small defense contractor), explains it best in this YouTube clip:
Highly compliant companies report lower SPRS scores than they deserve
800-171 and CMMC requirements have a “lower bar” than similar requirements used within DoD and federal networks. Controlled Unclassified Information is not SECRET information.
Contractors are allowed to define their risk tolerance for many requirements. For example, in Federal networks, there are specific requirements to lock accounts after 3 bad password attempts, and force interaction with the helpdesk before unlocking the account. In contrast, defense contractors are allowed to lock accounts at any threshold they think is appropriate (5? 10? 15?) and automatic unlocking over time is allowed.
Contractors who hire strong Governance, Risk, and Compliance (GRC) talent, are normally hiring people who used to work on Federal networks. These specialists often set the bar too high for their own organizations because they are used to Federal requirements. This often results in the GRC specialist marking the requirement NOT MET when it would be considered MET by a typical CMMC assessor.
Having compliance documentation dramatically improves SPRS scores
About 70% of the CMMC assessment objectives are met with policy, procedure, and other documentation-centric activities.
Contractors that focus on “set it and forget it” technical security systems and ignore their documentation and manual procedures will fail almost all requirements. This is because each 800-171 / CMMC requirement typically includes at least one assessment objective that requires documentation. (Each CMMC requirement has one-to-many assessment objectives)
There is a steep drop in real security scores between the companies that have their documentation in order, and companies that are ignoring documentation. This line rapidly drops to a “not applicable” score for the companies that don’t even have a system security plan.
This is just my personal experience. How does it match with what you’ve seen?
Do you think Defense Contractors are acting rationally when they report perfect 110 SPRS scores, even if their real score should be in the negative numbers?
How can we fix this issue, other than requiring third party assessments?
Do you think that companies reporting a perfect 110 should be specifically targeted for audits?
Do you think that companies reporting a non-perfect score should be targeted instead?
Do you think companies choose to report close-to-perfect scores (like 105) in order to reduce their risk of audit?
Author: Amira Armond, President of Kieri Solutions, an authorized CMMC Third Party Assessment Organization (C3PAO). Kieri Solutions provides assessment services, high-quality CMMC consulting, and an easy to use compliance documentation package geared toward small and medium businesses.
“This is just my personal experience. How does it match with what you’ve seen?”
Matches my experience, as does the linked video describing how companies who spend little time or money on cybersecurity have more time and money to spend on business development.
“Do you think Defense Contractors are acting rationally when they report perfect 110 SPRS scores, even if their real score should be in the negative numbers?”
No, but I am sympathetic. There is is a huge discrepancy between how those knowledgeable with DoD cybersecurity policy and those who are not knowledgeable interpret the controls and the required documentation. That leads to misrepresentation of the labor and cost of becoming compliant– especially for new companies. So I do not think it is a surprise that there are so many artificially high scores. For DoD work you have to enter *something* in SPRS on day one. And for a new company with no paid cybersecurity or policy expert on staff it is simply impossible to absorb hundreds or thousands of pages of policy, documentation and discussion intantaneously. (Something that I think the cybersecurity industry counts on and that the policy makers overlook.) My initial score was high and has only moved lower as I learn more and find more cases where I cannot possibly be compliant to the degree that is actually expected.
There’s also a pychological anchoring effect, where “110” is set as the anchor, and a self assessor will start from there and subtract points. So maybe a good starting point would be to direct companies with no DoD contracts to enter “0” or the most negative score, and then adjust the score *upwards* over time. That would bias scores towards the low side versus the high side.
“How can we fix this issue, other than requiring third party assessments?”
Make the controls and doc requirements explicit and clear for small or new businesses, and then permit flexibility for those who know how to execute it correctly. It’s the opposite right now, with small/new companies having to adapt to well established policy written by enterprise-grade policy and cyber experts.
Also, publish the stats on actual incidents, include company size and the specific control failures. The vulnerabilities of a microcompany are probably not the same as a Fortune 100 company. Related: I was just on a site that suggested that not having a SIEM makes me noncompliant with DFARS. That would cost me points in SPRS– if I didn’t have a SIEM. But I do– for one laptop, a firewall appliance and NAS (with an additional VM running the SIEM). A SIEM is a total waste of time, money and resources in my case. Those resources would have been better spent on weaknesses that are more likely to be exploited or on improving my documentation.
“Do you think that companies reporting a perfect 110 should be specifically targeted for audits?”
It depends on the objective. If the objective is to prevent that company from working with CUI or in the DIB, then “yes.” The message should be to protect information, not to maximize compliance metrics. (Like my example of having a SIEM for a minimal system. Adds to my SPRS score, but unnecessary to correlate events across two devices and a computer on a triple NAT’d network accessible by exactly one person. I actually see it as a liability. But it’s only now that I am comfortable making that trade.)
I think if you combine a score of 110 with contract size in dollars and CUI sensitivity/volume, that would direct audits at companies that lean more towards dishonesty and away from those who are simply naive or new to DoD cybersecurity requirements.
“Do you think that companies reporting a non-perfect score should be targeted instead?”
I think anyone who answers “yes” to this should be audited.
“Do you think companies choose to report close-to-perfect scores (like 105) in order to reduce their risk of audit?”
I wonder if high but non-perfect scores (like mine) are more a symptom of knowing some cybersecurity, but not enough to operate at the DoD or enterprise level: Knowing practical cybersecurity– the hardware and practice– maybe from employment in the DIB, but not the specifics of crafting policy and documentation.
I think my SPRS score is higher than it should be. But primarily because I skimp on documentation prose whose only audience (to date) is me. I have an SSP, but it’s diagrams, inventories and notes to myself on how my system addresses the controls. So my SPRS score reflects how well I address the controls implementation, with less emphasis on documentating for someone else to read, since it is such a small system with only one physical user (me). CMMC L2 will change the documentation audience of course. But I can’t justify the cost of pursuing CMMC L2 as an individual.