Small Population Compliance Samples

My last post, Compliance Sample Size, demonstrated the set up of an efficient sample method for compliance tests in case of large populations.

What if population size is relatively small ?, some actuaries asked me....

In this case you can ( instead of the beta distribution) make use of the hypergeometric distribution for calculating confidence levels.

Here's the same example as I used in my blog 'Compliance Sample Size', but now for a population of 100 .


'Compliance Check' Example (N=100)
As you probably know, pension advisors have to be compliant and  meet strict federal, state and local regulations.

On behave of the employee, the sponsoring employer as well as the insurer or pension fund, all have a strong interest that the involved 'Pension Advisor' actually is, acts and remains compliant.

PensionAdvice
A professional local Pension Advisor firm, 'PensionAdvice' (fictitious name), wants 'compliance' to become a 'calling card' for  their company. Target is that 'compliance' will become a competitive advantage over its rivals.

You, as an actuary, are asked to advise on the issue of how to verify PensionAdvice's compliance....... What to do?

• Step 1 : Compliance Definition
  First you ask the board of PensionAdvice  what compliance means.
  After several discussions compliance is in short defined as:
  
  
  1. Compliance Quality
     Meeting the regulator's (12 step)  legal compliance requirements
     ('Quality Advice Second Pillar Pension')
     
     
  2. Compliance Quantity
     A 100% compliance target of PensionAdvice's portfolio, with a 5% non-compliance rate (error rate) as a maximum on basis of a 95% confidence level.
  .
  
• Step 2: Check on the prior believes of management
  On basis of earlier experiences, management estimates the actual NonCompliance rate at 8% with 90% confidence that the actual NonCompliance rate is 8% or less:
  
  If management would have no idea at all, or if you would not (like to) include management opinion, simply estimate both (NonCompliance rate and confidence) at 50% (= indifferent) in your model.
  
  
• Step 3: Define Management Objectives
  After some discussion, management defines the (target) Maximum acceptable NonCompliance rate at 5% with a 95% confidence level (=CL).
  
  
• Step 4: Define population size
  In this case it's simple. PensionAdvice management knows for sure the portfolio they want to check for compliance, consists of 100 files: N=100.
  
  This is how step 2 to 4 look in your spreadsheet...
  
  
  
  
  
  
• Step 5 : Define Sample Size
  Now we get to the testing part....
  
  
  
  
  
  Before you start sampling, please notice how prior believes of management are rendered into a fictitious sample (test number = 0) in the model:

• In this case prior believes match a fictitious sample of size 25 with zero noncompliance observations. 
• This fictitious sample corresponds to a confidence level of 77% on basis of a maximum (population) noncompliance rate of 5%.

[ If you think the rendering is to optimistic, you can change the fictitious number of noncompliance observations from zero into 1, 2 or another number (examine in the spreadsheet what happens and play around).]


To lift the 77% confidence level to 95%, it would take an additional sample size of 20 - with zero noncompliance outcomes (you can check this in the spreadsheet).
As sampling is expensive, your employee Jos runs a first test (test 1) with a sample size of 10 with zero noncompliance outcomes. This looks promising!
The cumulative confidence level has risen from 76% to over 89%.


You decide to take another limited sample with a sample size of 10. Unfortunately this sample contains one noncompliant outcome. As a result, the cumulative confidence level drops to almost 75% and another sample of size 20 with zero noncompliant outcomes is necessary to reach the desired 95% confidence level.

You decide to go on and after a few other tests you finally arrive at the intended 95%cumulative confidence level. Mission succeeded!

Evaluation
The interesting aspects of this method are:

1. Prior (weak or small) samples or beliefs about the true error rate and confidence levels, can be added in the model in the form of an (artificial) additional (pre)sample.
   
   
2. As the sample size increases, it becomes clear whether  the defined confidence level will be met or not and if adding more samples is appropriate and/or cost effective.
   

This way unnecessary samples are avoided, sampling becomes as cost effective as possible and auditor and client can dynamically develop a grip on the distribution. Enough talk, let's demonstrate how this works.

Another great advantage of this incremental sampling method is that if noncompliance shows up in an early stage, you can

• stop sampling, without having made major sampling cost
• Improve compliance of the population by means of additional measures on basis of the learnings from the noncompliant outcomes
• start sampling again (from the start) 

If - for example -  test 1 would have had 3 noncompliant outcomes instead of zero, it would take an additional test of size 57 with zero noncompliant outcomes tot achieve a 95% confidence level.  It's clear that in this case it's better to first learn from the 3 noncompliant outomes, what's wrong or needs improvement, than to go on with expensive sampling against your better judgment.

D. Conclusions

On basis of a prior believe that - with 90% confidence - the population is  8% noncompliant, we can now conclude that after an additional total sample of size 40, PensionAdvice's noncompliance rate is 5% or less with a 95% confidence level.

If we want to be 95% sure without 'prior believe', we'll have to take an additional sample of size 25 with zero noncompliant outcomes as a result.

E. Check out: DOWNLOAD EXCEL
You can download the next Excel spreadsheets to check the Demo or tot set up your own compliance test:

- Small population Compliance test DEMO
- Small population Compliance test BLANK
- Large population Compliance test

Enjoy!