Assessment Troubleshooting

Resolve issues with compliance assessments including failures, errors, and unexpected results.

Assessment Won't Start

Connection Required

Symptoms:

• "Connect Microsoft 365 first" message
• Run button disabled
• No frameworks available

Solutions:

1. Complete M365 setup

   • Go to Settings → Integrations
   • Follow app registration process
   • Test connection successfully

2. Verify connection status

   • Check for green "Connected" indicator
   • Test connection if uncertain

Permission Issues

Symptoms:

• "Insufficient permissions" error
• Assessment starts but fails immediately

Solutions:

1. Check M365 permissions

   • Review required permissions
   • Grant admin consent
   • Wait for propagation (up to 24 hours)

2. Verify app registration

   • App registration troubleshooting

Assessment Fails Mid-Run

Partial Failures

Symptoms:

• Some controls pass, some show errors
• "Assessment completed with errors"
• Red error indicators on specific controls

Causes:

• Missing permissions for specific resources
• Resources don't exist in tenant
• Temporary API errors

Solutions:

1. Review error details

   • Click on failed controls
   • Read specific error messages
   • Identify pattern (all policy controls, etc.)

2. Check permissions

   • Missing permission often affects groups of controls
   • Add required permissions
   • Re-run assessment

3. Re-run assessment

   • Temporary errors may resolve
   • API throttling can cause failures

Complete Failure

Symptoms:

• Assessment stops with error
• No results generated
• "Assessment failed" status

Causes:

• Connection lost
• App registration invalid
• Service outage

Solutions:

1. Test M365 connection

   • Settings → Integrations → Test
   • Fix any connection issues first

2. Check Microsoft status

   • status.office365.com
   • Wait for service restoration

3. Try again later

   • Network issues may be temporary
   • Microsoft API may be throttled

Unexpected Results

Scores Lower Than Expected

Symptoms:

• Score dropped from previous assessment
• Controls failing that should pass
• Unexpected non-compliant findings

Causes:

• Configuration actually changed
• Different assessment scope
• Framework updated
• Permission issues

Solutions:

1. Compare to previous run

   • View assessment history
   • Click on changed controls
   • Review what specifically changed

2. Verify configuration

   • Check actual M365 settings
   • Confirm expected state
   • May have been changed

3. Check scope differences

   • Same framework version?
   • Same controls selected?
   • Exclusions changed?

Controls Show "Unknown"

Symptoms:

• Controls with gray/unknown status
• "Unable to determine" messages
• No evidence collected

Causes:

• Permission not granted for that data
• Resource type not configured
• API error during evaluation

Solutions:

1. Check required permissions

   • Different controls need different permissions
   • Add missing permissions

2. Verify resource exists

   • Some controls require specific features
   • Feature may not be enabled in tenant

3. Review control requirements

   • Check control documentation
   • Verify prerequisites

False Positives

Symptoms:

• Control marked non-compliant incorrectly
• Settings appear correct but control fails

Causes:

• Evaluation logic edge case
• Configuration nuance
• Assessment timing

Solutions:

1. Review evidence

   • Check actual values captured
   • Compare to M365 portal
   • Identify discrepancy

2. Create exclusion

   • If you've verified it's compliant
   • Document reason for exclusion
   • Review periodically

3. Report issue

   • Contact support with details
   • Include evidence screenshots

Performance Issues

Assessment Takes Too Long

Symptoms:

• Assessment stuck on loading
• Running for hours
• Progress not advancing

Causes:

• Large tenant with many resources
• API rate limiting
• Network issues

Solutions:

1. Be patient for large tenants

   • First assessment may take longer
   • Subsequent runs use caching

2. Check progress

   • Assessments process in batches
   • Some stages take longer

3. Try during off-peak

   • Run during low-usage periods
   • Avoid times with other M365 activity

Frequent Timeouts

Symptoms:

• Assessment times out repeatedly
• "Request timeout" errors

Solutions:

1. Check network stability

   • Verify consistent connection
   • Check for proxy issues

2. Try smaller scope

   • Assess fewer controls at once
   • Split into multiple runs

3. Contact support

   • May need backend optimization
   • Provide tenant size info

Specific Control Errors

Policy Controls Failing

Symptoms:

• All policy-related controls fail
• "Unable to read policies" errors

Solutions:

1. Grant Policy.Read.All permission
2. Grant admin consent
3. Wait for propagation
4. Re-run assessment

User Controls Failing

Symptoms:

• User-related controls fail
• "Unable to read users" errors

Solutions:

1. Grant User.Read.All permission
2. Grant Directory.Read.All permission
3. Admin consent required

Security Controls Failing

Symptoms:

• Security controls show errors
• No security data available

Solutions:

1. Grant SecurityEvents.Read.All
2. May need E5 or ATP license
3. Some features require specific plans

Assessment History Issues

History Not Showing

Symptoms:

• Previous assessments not listed
• History appears empty

Causes:

• No previous assessments run
• Different organization selected
• Data retention period

Solutions:

1. Check organization context

   • Verify correct org selected
   • Assessment history is per-org

2. Check date range

   • History may be filtered
   • Expand date range

Can't Compare Assessments

Symptoms:

• Compare feature unavailable
• Different frameworks can't compare

Solutions:

1. Use same framework

   • Comparisons require same framework
   • Same version preferred

2. Have multiple runs

   • Need at least 2 assessments
   • Run assessment again to compare

Error Codes

Common Assessment Errors

Best Practices

Before Running

1. Verify connection is healthy
2. Check permissions are granted
3. Review scope of assessment
4. Note current score for comparison

After Running

1. Review failures for patterns
2. Check evidence on failed controls
3. Document exceptions if needed
4. Plan remediation for real issues

Regular Assessments

1. Schedule recurring assessments
2. Track trends over time
3. Investigate score changes
4. Update exclusions as needed

What's Next?

• Drift Detection Troubleshooting - Monitoring issues
• App Registration Issues - Connection problems
• Running Assessments - Assessment guide