I am thrilled to present this second video Q&A session about the CMMC Assessment process.
This CMMC Q&A is not technical
Unlike many topics on this website, this discussion is relevant to ALL people interested in CMMC. Business owners, IT professionals, cybersecurity professionals, CMMC assessors, CMMC assessment organizations, etc.
We purposefully try to use CMMC level 1 examples where possible, so this discussion doesn’t get very technical.
CMMC Q&A about Assessments discussed in Part 2:
- Assessing different sensitivities of systems within the same company.
- Does the client or assessor identify the scope of assessment?
- Assessment planning process.
- Why do we use the word “assessment” rather than “audit”?
Show notes for the interview will be added below the video. Make sure to scroll down for additional information and links.
I think this is extremely valuable content for the entire DIB. Thanks to Jeff Dalton and the CMMC-AB for their efforts. Please share this with other companies and sign up for our newsletter. This series will hopefully save contractors a lot of angst about the assessment process. – Amira
Text version of interview
Note: This section is paraphrased by Amira Armond. If anything was misquoted, it is Amira’s fault. Please let us know if you see any major problems. For the most accurate version, watch the video!
How does a company assess multiple CMMC Levels?
Amira: Let’s assume we are talking about an organization that is going for CMMC Level 3.
Amira: In a lot of cases, these organizations will do something called “segmenting their network” where they separate their CUI systems from the rest of the network. For example, they will have a corporate email system or a corporate Salesforce system, and then they will have separated file servers with CUI on the file servers.
Amira: How do assessments work for those companies which have FCI (CMMC Level 1 protected data) on systems that aren’t CMMC Level 3?
Jeff: At a big picture level, it will be rare for entire companies to be assessed at a single CMMC Maturity Level. If you take a very big company as an example, like Northrup Grumman, it is very unlikely that the entire company will get a single assessment. It is possible, but highly unlikely, due to the sheer scope and complexity.
Jeff: The same thing applies to smaller companies, in that the entire company is generally not subject to CMMC ML3. The contract they are responding to is only applicable to a [division] of the company. Unless they are very small, or the entire company only performs [DoD contracts], in which case it could make sense to assess the entire company.
Introduction of the term “Host Unit”
Jeff: The “host unit” is the method we are asking C3PAOs to use [to define organizational scope]. In that case, [the organizational scope] could be the whole company. In other certification frameworks (ISO, ITIL, CMMI) there will be a segmentation of the assessment itself. The hierarchy is “organization” > “host unit” > “enclave”.
Jeff: Not all companies will have an enclave. They will just have a host unit. Others will have corporate and enclave, but no host unit.
The company (client) needs to define scope
Jeff: An organization needs to clearly define the scope of their assessment to only include the segment of the organization that is required to be CMMC Level 3. They have to provide that information as part of the assessment planning process. They can provide network diagrams, organization charts, divisional boundaries, management boundaries, charge code, CAGE code, or any combination of these things, to clearly define a boundary around any of these things.
The scope needs to make sense to the assessor
Jeff: It will be up to the assessor to validate that those complexities are solved. It will be challenging to identify how data flows. Data flow diagrams of some sort will need to be provided to convince the assessor that the [scope] is true and real and accurate. It will be a negotiation between the assessor and the company.
Jeff: The company needs to convince the assessor that the segmentation is true and real and accurate. The upside of a company doing it this way (identifying a smaller scope with segmentation) is that it reduces the overall implementation and assessment costs. The downside is that the company is limited to only that segmentation. If they bid on another contract, or other parts of the network need to be included, they will need to un-do the segmentation to accomplish that [adding scope].
Segmentation needs to be realistic and functional
Jeff: You need to consider your goals and objectives. What is the company trying to accomplish? I hear a lot, “We want to skinny down our assessment to the smallest possible network segmentation”. Well, you can do that. You are allowed to do that. But I don’t recommend that you do that because the utility, the utility of your certification, is limited when you do that. So companies should consider their goals and objectives in relation to CMMC as they consider this.
How does it work if FCI is segmented separately from CUI?
Amira: When I first started helping clients prepare for CMMC Level 3, at first I was very focused on “OK, here are our level 3 systems, let’s secure them.” It wasn’t until I got deep into this effort that I realized there was a big question about the level 1 systems, the FCI systems, where most clients can’t be limited to just level 3 capable systems. If they are capable, then it is great, you just do one assessment at level 3. But in a lot of cases, especially for organizations above 200 people, especially organizations that are not just DoD, they will have FCI on systems that can’t be certified to Level 3. Literally not capable of performing the security requirements within budget.
Amira: Has that come up as a discussion point? Do these companies need a Level 1 assessment that is separate from Level 3? Can an assessor say “OK, here is our Level 1 scope, and here is our Level 3 scope” and the assessor applies different requirements to each system?
Jeff: It has been discussed, and a company can certainly do that. There is no preclusion for a company saying they want Level 3 on this host unit and we want to do Level 1 on other host units at the same time. That is just a planning question. You can do multiple assessments at the same time. The question remains, what is the utility of doing that? I think you described a good use case for that. We want to have FCI over there, but not CUI, and I think there is utility for that.
Jeff: The question is a business question. Will companies want to do that? Will they expand the boundaries of their assessment to other parts of the organization? And I think they should. It is an excellent thing to do.
If FCI for a contract is outside of the segmented enclave, the scope needs to increase
Jeff: If history is any guide, we will see a laser-like focus on the contract requirements. If the contract says that “this division” or “this host unit” needs to be Level 3, then companies will really focus on that. Now if the FCI for the host unit extends out into other parts of the organization, then they need to be part of the scope. That is a little different use case. Then they need to be part of the scope. This is what I was talking about before, about the data flow. You need to follow where the data goes. So that means that those networks, if they were segmented in the way that you described, would be subject to the appraisal as well.
Amira: OK. We should probably talk about this in depth at some point…
Jeff: Yes, that is also a good Regan Edens question, because he has done a lot with scoping, so he could be brought into this discussion as well.
Jeff: This has never been done, right? A lot of people are asking these questions. Whatever use case we account for, there will be other use cases that weren’t accounted for. That will just be the situation for the first couple of years. This happened with CMMI as well, every year I am still seeing new use cases. Some people will push the envelope, some people will ask. It will be an interesting couple of years.
Amira: Next topic then. The audit charter… assessment plan… what is the term?
Jeff: We call it assessment plan. The assessment plan is a very detailed artifact that is an iterative document. In other words, it isn’t a Microsoft project schedule. It is a document that defines: schedule, cost, stakeholders, sponsor, points of contact, host unit information, who we are going to interview, what types of data we will review, who will be on the assessment team. Basically everything logistically about the assessment and everything we are going to do, so the defense contractor has a clear understanding of exactly what will happen when we walk in the door.
“Verification” type assessments for CMMC
Jeff: This might be interesting to you. There are basically three different types of assessments in the industry. There is the “discovery assessment”, which has almost no planning done except for scheduling. The assessor walks in the door and starts asking to see things. A “verification” assessment is when everything is identified in advance. The assessor tells the client exactly what they want to see / test / interview, and asks for it to be ready before they come. This reduces confusion and stress and hassle and time for the onsite appraisal. Then there is a hybrid which is called “managed discovery” . We do some in advance (such as reviewing documents) and then when we come onsite, we discover more.
Jeff: We recommend the “verification” technique which is very in-depth planning up front so that there are no surprises, and so that we can limit time onsite with the larger team. Reduces time and effort.
How is assessment scope identified?
Amira: You answered some of my questions there. One of my questions is “What is the client’s role in identifying scope, and is there a recommended way to identify scope to your assessment organization?
Jeff: We have processes with the resulting artifacts. We have the “intake form” which is basic demographics information about the client. What their networks are like, who the CEO and directors are. Basic information.
Assessment sponsor, point of contact
Jeff: Then we have the planning process, where scoping takes place. We require an assessment sponsor, normally a senior leader in the company, and an assessment point of contact, to provide that scoping information. It starts initially with the [DoD] contract – what does the contract require? Is the scope sufficiently covered? Then it will be a conversation between the assessor, C3PAO, and the defense contractor.
Scope starts with where FCI and CUI for the contract(s) exists
Jeff: It will be a similar conversation to what we are having now. Where is the FCI? The CUI? The [provisional] assessors have gone through some training on this. We said, “You need to keep your eye out for accuracy of scoping”. What happens a lot in the industry, is the company being assessed will have an idea in their mind of what the scope should be. They might say “We’ve picked the three contracts that will be the subject of the scope”, and we ask “Well, how many contracts total do you have?” “Oh, we have 500”. Well, we need to re-examine the scope then because it isn’t credible. That is an extreme example.
Jeff: It will be a combination of contract scope, the sponsor’s desire for the utility of the assessment, the point of contact’s knowledge of the network and the organization, and the assessor’s knowledge of the requirements and what needs to be looked at. It will be a discussion based on those characteristics. Then the sponsor needs to sign off on the scope. Basically everything in the plan needs to be signed off by the senior executive at the defense contractor. So they need to take responsibility for the scoping and the evidence and the entire assessment, because they are the ones conducting it, and we can’t take responsibility because we aren’t part of the company. So if anything was mis-stated or mis-understood, then the sponsor owns that process for changing the plan.
What about certification before the defense contractor has the work?
Amira: Let me channel my “inner OSC”. And in case anyone doesn’t know, OSC stands for Organization Seeking Certification – the defense contractor seeking CMMC.
Jeff: Right now, yeah, generally.
Amira: So say they are getting a CMMC Level 1 certification. The idea is they will be bidding on a DoD contract that requires CMMC Level 1. And as part of the bid, they will be saying they have a CMMC certificate, they identify the information system that is certified, and that they plan to use this information system to perform this contract. Is that correct?
Jeff: Yes, that is what the host unit is about. The information system will be identified by the organization somehow: “this network inside of Infrastructure Services” – whatever the host unit is, that is described, will be defined by the defense contractor based on contract requirements.
Amira: If the contractor doesn’t have the contract already, if they aren’t performing the work, is the defense contractor saying “This is what I would use, if I won?”
Jeff: Right. “This is what we plan to use for this.” “This is our projected use case.” “This is the system and people that will be performing the work.” They will need to scope it that way. The defense contractor won’t have the contract yet, they will have the RFP. Just like any other technology RFP, they will be describing how they plan to do the work.
Jeff: That will be part of their discussion with the assessor. The assessor will want to see what the projected plan is, and whether that aligns with their scope.
Mixing Level 1 and Level 3 in the same assessment
Amira: If there is a desired maturity level that is higher than Level 1… meaning the client wants to get CMMC Level 2, or Level 3, or higher… When the client is identifying their scope, should they say “on this system, I want Level 1, on that system, I want Level 3”?
Jeff: If you are getting a Level 3 assessment on a system, then you are also getting Level 1 on a system. But if you are saying “I want network A, or organization A, to be Level 3” and “I want organization B and network B to be Level 1”, that is a restating of your earlier question. I think that is possible. But it is important for people to understand that if you conduct a Level 3 assessment on a system, you are also performing a Level 1 assessment on that system.
Does the C3PAO re-state their understanding in the assessment plan? (as opposed to using a client statement unaltered)
Amira: When the C3PAO comes back with the assessment plan, would that plan re-iterate the data locations and the systems that are considered in-scope by the C3PAO?
Jeff: Yes, it should. A big part of the plan is scoping. Three parties need to agree to the scoping plan: Assessor, C3PAO, and executive sponsor at the defense contractor. They need to sign off on the scope so that everyone is exactly on the same page.
Assessment plans need to be signed by Assessors, C3PAO, and executive sponsor (defense contractor)
Jeff: I like to say on my plans, “Let’s all sign off on this so that we can remain friends.” Assessments can be emotional events. If you haven’t been through a formal assessment, you may not have a feel for it… but people don’t like other people judging them. At all. Especially technology people, they seem to have real trouble with it. It is difficult. So there is often disagreement with the findings. “You guys don’t have this.” “Oh yes we do.” So it is really important that all of this is signed off and agreed upon in advance, in case of emotion. And assessors get emotional too, it isn’t just defense contractors. So we need a very fact-based process.
Does the assessment plan go to the individual practice level?
Amira: Does the assessment plan get into great detail about scoping for individual requirements or practices? For example, would you say “For this practice, we will evaluate this set of systems, and for this other practice we will evaluate this other set of systems.”? Or is it more ad-hoc?
Jeff: There are two tools for that. The plan is one of them. The current plan, which is just a template at this point, asks a little more generically. “We want to look at this network, or we want to look at these things.” It isn’t a list of 500 things. It is a smaller list. We have also given provisional assessors a tool for this. So far we’ve given them [provisional assessors] an excel workbook that has all the practices, and some examples of what is expected. This is a great thing to use for planning, and it could be given to the customer to help explain what is expected for each practice. Is it required? No. That is part of the managed discovery view where we want the assessors to do a lot of planning upfront, but not do everything upfront. So some of them are going to be using that tool. They could say “Here’s the inventory we might look at, but we’re going to find out when I get there, exactly what the answer is for specific practices.”
About purchasing pre-made policy and procedure templates
Jeff: There is a fine line with what we give [the assessors]. If we publish a document with exactly the kinds of work products that we are looking for, a lot of people would applaud and say it is helpful. The downside is that enterprising consultants will get a copy, go to the clients, and tell them they need to do exactly these things and they will be compliant. To me, I think this is backwards. We want to give suggestions, but we don’t want companies copying these things exactly. The processes need to live in their organization as part of their normal operations.
Jeff: I’ve seen this on LinkedIn already. I’ve seen companies advertising CMMC Compliant templates. Templates by themselves don’t get you there. This is about people, processes, and technology. Not just templates.
Amira: I can echo you on this, I’ve heard offhand comments from people working toward Certified Assessor, that if they found a [well known] policy set at one of the defense contractors being assessed, that the assessor would dig deeper to make sure the company is actually using them.
Jeff: I could do a whole session on templates. I think “templates are interesting, but not relevant.” They are an example of what one person thinks is the answer. If you are struggling to understand what a practice is about, a template could help you by showing you what one person thinks it is about. But I promise you, if you buy another company’s templates, it will be different. I’ve already experienced this with the working groups, where there was disagreement on the examples that were created. Different companies have different use cases. This proves out the idea that CMMC is an “assessment”, not an “audit”. This needs to be context driven, and this will be challenging.
Why is “assessment” used instead of “audit” for CMMC?
Amira: I can’t help but follow up on this. One of my questions is, “What is the difference between an assessment and an audit?” Because apparently there is a difference.
Jeff: I think there is a difference. It isn’t just me, the industry thinks there is a difference. Katie Arrington thinks there is a difference. The DoD thinks there is a difference.
Jeff: An assessment is about context. For the most part, the practices in the model are descriptive, not prescriptive. They don’t say exactly what to do, they tell you the kinds of thing you should do. Your implementation of that particular practice is contextual, based on your company and your culture and the way you do work. So for example, a policy at Lockheed Martin [a very large defense contractor] is a very serious thing that is reviewed by Legal, Management, working groups. Revisions are done, everyone signs off and adheres to the process. And that makes sense when you have 70,000 people in the company. But when you have a 10 person company, I don’t need that level of rigor. The CEO writes a policy, posts it on a portal, and tells everyone to do it. There is more to it than that, but that is the basic delivery mechanism.
Jeff: So if I had an audit, I would have [very specific questions]. Reviewed by a working group? Signed off by… Done by…? Whatever those characteristics are, if we are doing an audit, it needs to be a Yes or No.
Jeff: There is sort of a “high trust” and “low trust” paradigm here. The audit model is very low trust. The people who wrote it know what the correct answer is. It is either Yes or No. An IRS audit has no negotiation. They know the answer and you need to comply. With an assessment, we are saying “we are looking for these kinds of things. For your culture, your use case, tell us how you are satisfying that.” It is a context driven answer versus a yes or no.
Jeff: There are some things that are yes or no. “Do you have a lock on the door?” Yes or no. But even then, with everyone working from home, what does a lock on the door mean? In some communities, it could be a screen door and it works. In other communities, I could need three deadbolts.
Jeff: It comes down to high trust or low trust. An audit is low trust. The ones that wrote the audit method know best and we need to comply with that. An assessment is high trust. An assessment method is much more high trust. The writers did research, created examples, said they think that it should be done something like this. Then the defense contractor implements it in a way that is best for them, and the assessors are trained to make the connection between those two things.
Jeff: So to me, I think they are very different. To me, the high trust version, the assessment, is healthier and better for the organization. It helps stickiness. Stickiness is when people actually keep doing the things what you asked them to do. So if I audit you, threatening you with a stick, and say you must do this and this and this… I’m pretty sure that as soon as the auditor leaves, the company will stop doing it. So I think it is a lot better.
Does the “assessment” philosophy give some wiggle room to pass despite a missed Assessment Objective?
Amira: The DoD assessment guides [for CMMC] do seem to follow-up on what you say. They look for characteristics of compliance, or characteristics of secure practice. They don’t say “do this exact thing.” With this philosophy (and this could get you in trouble…) could an assessor pick out an Assessment Objective and say “Well, this one isn’t met, but let’s call this overall practice good because I don’t think it really matters for this organization.”
Jeff: Well, I’m not going to comment on the DoD’s position on things. They own the policy on this topic. An assessment is about “are they meeting the intent”, and the objectives and the examples are designed to clarify the intent of the practice. So if I’m an assessor, then my goal is to clarify the intent, using the assessment guide to inform me as to the intent of the authors of the CMMC model. So if I believe, and the assessment team agrees, that the defense contractor has met the intent of the practice, then I’d say that you’ve met the intent of the practice and that is a pass. There are so many objectives and so many examples that I can’t really comment on whether one or two or others aren’t there.
Jeff: I’ll take a wild stab and say that there are objectives in there that shouldn’t be in there, and there are objectives that should be in there, that aren’t in there. Because there are a lot of them, and it is version 1.0. It is a little like the template example that we talked about before.
Jeff: What makes me uncomfortable with objectives is that will be the only thing that people look at. So [consultants] will take the assessment guide, take that to clients, and tell them to just do the exact things that are listed there. There is history here. With CMMI, back in 2001, they used to list work products for each practice. So a bunch of companies started doing the work products, that was it. And those work products had nothing to do with their businesses or their goals. All they had to do was reverse-engineer it. And then some enterprising consultant came out with a book with one page per document. All you had to do was fill it out by hand, send it to the assessor, and you are level 3. It’s like the templates. That’s just crazy! All of those artifacts [don’t match your company].
Jeff: I don’t know all of the objectives. I don’t have them all memorized. But I would say that it is about the intent of the practice. That is what we are trying to validate, the intent of the practice.
Amira: I’ve been through a few self-assessments with my own company, or with clients. I’m literally going through those assessment objectives one by one, Met, Not Met, one by one. I can tell you that there isn’t a lot of wiggle room for them. But I do appreciate the intent idea. And this seems to be supported by the CMMC Model, where it says you can have a plan of action for temporary issues. For example, one of the requirements is that you have antivirus on all computers. And the day before the assessment, one of your computers stopped reporting it’s antivirus. I would hope that you wouldn’t fail your assessment based on that one computer, as long as you’ve got a process to notice the problem, track it, and fix it pretty quickly.
Jeff: I’m not going to address individual yes or no’s… whether I would or would not pass that. But I think you are articulating the overall intent of the model, which is improvement, and improving performance. So to me, as an assessor, if we failed someone based on that scenario, I’d wonder if we were improving performance. But I won’t get into specific scenarios because each one is very unique.
Is the assessment plan expected to be performed as part of the bid process? Or is it paid?
Amira: I have one last question. Is the assessment plan built before there is a contract with the defense contractor? As part of the bidding process? Because that is a lot of work! Or should we expect to have a contract then build the assessment plan?
Jeff: This is a question that has come up before. It is less of a CMMC-AB or DoD question than one about how to run your business. I love questions like these, because they aren’t policy. I can’t get in trouble with questions like this.
Jeff: What I encouraged provisional assessors to consider during the class, is that this is a business. If you are going to do work on behalf of a client, you should be paid for it. That is how the system works. I would encourage any C3PAO that is speaking with a client, to discuss the minimum amount of time involved in creating the scope of the assessment. The scope of work. And I would establish that if any new information comes to light, that the scope will change. So many of the headaches we see in the consulting business is that we don’t adequately prepare our customers for change. We don’t say “It’s going to be $1, but if what you told me isn’t actually correct, it will cost more.” What I do with my own company, for this type of engagement, is we split this into phases. Have a phase 1 for planning, which is a required engagement in the model. Since it has to happen, why not have an engagement for just the planning piece?
Jeff: I tell my clients, “I don’t know anything about you, other than this brief discussion we’ve had over the phone.” It is kind of like going to a builder and saying “I want a house, how much will it cost?”. The builder says, “Lets have an architecture drawing and go through options, and that will cost $100, and then I’ll tell you what it will cost. And the customer says “OK”. So I’d encourage everyone to follow generally accepted business practices and have a planning period that is funded, and just do minimal planning before a contract.
Jeff: We don’t want to see assessors quoting an engagement for $12,000 and then the defense contractor says, “oh by the way, I didn’t mention it, but we are Northrup Grumman.” We don’t want to see that sort of thing happening. We think the planning is big enough that it should be an engagement by itself.
<finish out, discuss next interview plans>
Thanks for the read, all!
Please share this with Defense Industrial Base contractors who are interested in getting CMMC certified. This series of interviews should be very helpful in understanding the assessment process.
Thanks again to the CMMC-AB for authorizing this interview and helping the ecosystem stay on track.
V. Amira Armond (CISSP, CISA, PMP, MBA) is a computer systems architect, cyber-security consultant, and owner of Kieri Solutions LLC.
Kieri Solutions LLC is in progress to become a CMMC assessment organization and has several Registered Practitioners and Certified Assessor candidates on staff. Amira is also the chief editor for cmmcaudit.org, a public resource for news and informational articles about the Cybersecurity Maturity Model Certification.