Medical Device Auditing Sampling Methodologies
A subject that I think needs way more attention
Introduction
One of the most common inconsistencies seen across the medical-device industry is in sampling methodology.
A strong sampling methodology can really be the difference between a super effective audit that gives you a comprehensive understanding of the entire population and an audit that doesn’t really scrape the surface at all within overall amount of records.
When auditing, you need to be able to digest huge amounts of information within very short and most importantly restricted periods of time.
Unless your FDA - you probably aren’t able to camp at site or run your inspection for weeks on end.
So, this is where sampling becomes criticalI.
Although sampling is understood largely to be for time-saving, it is far more than a time-saving tool; it’s the statistical foundation of audit credibility.
In this article, we’ll walk through a practical lifecycle of sampling:
What sampling actually is.
Why it matters.
How it connects to validations and quality control.
The types of sampling available.
When to use each type.
What regulators — including Notified Bodies, MDSAP, and FDA — expect to see.
What Is Sampling?
Sampling is the process of selecting a subset of records, products, or activities to draw conclusions about the whole.
If you know me by now, you know I don’t like to invent or recreate things, especially when standards committees have done so for us.
ISO 19011: 2018 - Guidelines for Auditing Management Systems puts forward a comprehensive description of sampling below:
Audit sampling takes place when it is not practical or cost effective to examine all available information during an audit, e.g. records are too numerous or too dispersed geographically to justify the examination of every item in the population.
Audit sampling of a large population is the process of selecting less than 100 % of the items within the total available data set (population) to obtain and evaluate evidence about some characteristic of that population, in order to form a conclusion concerning the population.
The objective of audit sampling is to provide information for the auditor to have confidence that the audit objectives can or will be achieved.
The risk associated with sampling is that the samples may not be representative of the population from which they are selected. Thus, the auditor’s conclusion may be biased and be different from that which would be reached if the whole population was examined. There may be other risks depending on the variability within the population to be sampled and the method chosen.
In practice, your “lot” might be:
All CAPAs closed in a year.
All supplier evaluations in a quarter.
All complaint files within a product family.
Sampling converts an overwhelming population into manageable, defensible evidence.
Why Sampling Is Important
Without structured sampling, audits drift into two extremes:
Too shallow → token checks that miss systemic issues.
Too deep → days lost reviewing dozens of records with diminishing value potentially reducing the linkages to other elements or lost time on other elements within the audit plan.
A documented, risk-based sampling plan:
Demonstrates efficiency and proportionality.
Enables repeatability between auditors.
Provides traceable logic if challenged by a Notified Body or FDA investigator.
Protects the auditor from “why didn’t you look at X?” moments.
✅ Auditing is a confidence exercise — sampling defines how much confidence you can legitimately claim.
Types of Sampling
ISO 19011 within A.6 breaks sampling into two distinct methodologies:
Judgement-based sampling
Judgement-based sampling relies on the competence and experience of the audit team (see Clause 7).
For judgement-based sampling, the following can be considered:
a) previous audit experience within the audit scope;
b) complexity of requirements (including statutory and regulatory requirements) to achieve the audit objectives;
c) complexity and interaction of the organization’s processes and management system elements;
d) degree of change in technology, human factor or management system;
e) previously identified significant risks and opportunities for improvement;
f) output from monitoring of management systems.
A drawback to judgement-based sampling is that there can be no statistical estimate of the effect of uncertainty in the findings of the audit and the conclusions reached.
Statistical-based sampling
If the decision is made to use statistical sampling, the sampling plan should be based on the audit objectives and what is known about the characteristics of overall population from which the samples are to be taken.
Statistical sampling design uses a sample selection process based on probability theory. Attribute-based sampling is used when there are only two possible sample outcomes for each sample (e.g. correct/incorrect or pass/fail). Variable-based sampling is used when the sample outcomes occur in a continuous range.
The sampling plan should take into account whether the outcomes being examined are likely to be attribute-based or variable-based. For example, when evaluating conformity of completed forms to the requirements set out in a procedure, an attribute-based approach could be used. When examining the occurrence of hazard safety incidents or the number of cybersecurity breaches, a variable-based approach would likely be more appropriate.
Elements that can affect the audit sampling plan are:
a) the context, size, nature and complexity of the organization;
b) the number of competent auditors;
c) the frequency of audits;
d) the time of individual audit;
e) any externally required confidence level;
f) the occurrence of undesirable and/or unexpected events.
When a statistical sampling plan is developed, the level of sampling risk that the auditor is willing to accept is an important consideration. This is often referred to as the acceptable confidence level. For example, a sampling risk of 5 % corresponds to an acceptable confidence level of 95 %. A sampling risk of 5 % means the auditor is willing to accept the risk that 5 out of 100 (or 1 in 20) of the samples examined will not reflect the actual values that would be seen if the entire population was examined.
When statistical sampling is used, auditors should appropriately document the work performed. This should include a description of the population that was intended to be sampled, the sampling criteria used for the evaluation (e.g. what is an acceptable sample), the statistical parameters and methods that were utilized, the number of samples evaluated and the results obtained.
To build upon the foundation set by ISO 19011: 2018 - we can’t talk about sampling without mentioning ISO 2859-1 - Sampling procedures for inspection by attributes - Part 1: Sampling schemes indexed by acceptance quality limit (AQL) for lot-by-lot inspection.
🧠 Pro Tip: ISO 2859-1 General Inspection Level II, AQL 2.5 % is a strong baseline for most audit sampling.
For high-risk systems, tighten to AQL 1.0 % or 0.65 %.
Types of statistical-based sampling
Full Population (100% Inspection) - Every unit in the population or lot is inspected or tested. This approach eliminates sampling uncertainty and is typically used for critical or high-risk characteristics where failure cannot be tolerated (e.g., sterility, labeling of implantable devices). While resource-intensive, it ensures complete verification when the risk of defect is unacceptable.
Block / Cluster Sampling - The population is divided into clusters or groups (e.g., by production batch, machine, or time period), and entire clusters are selected for inspection rather than individual random samples. Useful when products are produced in batches or lots with expected within-cluster uniformity (e.g., same shift or same sterilization cycle). Reduces sampling costs but assumes minimal variation between clusters.
Haphazard Sampling - Samples are taken without formal randomization, but without conscious bias. It’s a non-statistical method used for exploratory checks, early process verification, or when formal random selection isn’t practical. It is not statistically valid for conformance verification but may be acceptable for informal in-process reviews or audits.
Attribute Sampling - Each unit is inspected for conformance/nonconformance against defined criteria (pass/fail, go/no-go). Commonly used in visual inspections, labeling, dimensional checks, and packaging integrity tests. The data are qualitative, and sampling plans are often based on ISO 2859-1 (AQL tables).
Random Sampling - Each unit in the population has an equal and independent chance of being selected. Ensures unbiased representation of the population and supports statistical validity of conclusions. Widely used in acceptance sampling, process validation, and stability studies. Typically implemented using random number generators or randomisation tables.
Sampling with Standards - Sampling conducted according to a recognized statistical standard, such as ISO 2859 (for attributes) or ISO 3951 (for variables). Provides defined sampling plans, acceptance/rejection criteria, and statistical confidence levels. This approach ensures regulatory defensibility and consistency in inspection methods.
Stratified Sampling - The population is divided into subgroups (strata) that share similar characteristics (e.g., production line, operator, shift, material lot), and random samples are drawn from each stratum. Improves representativeness and ensures all key process variables are covered. Commonly used in process validation and supplier quality audits.
Systematic Sampling - Items are selected at regular intervals (e.g., every 10th unit) after a random start point. Provides good coverage with practical efficiency. Effective for in-process inspections and continuous manufacturing, but only valid if there’s no hidden periodicity in the process.
Sampling Situation Matrix
I’ve tried to pull together a bit of a matrix with some example situations mapped with potential ways of utilising sampling for each scenario:
For small populations (≤30 records) or immature QMS processes, use judgement-based full population sampling, meaning you review 100% of all records because sampling adds no statistical value.
When on-site audit time is limited — such as during surveillance audits, use judgement-based block or cluster sampling, where you select all records from a defined week, month, shift, or batch to gain meaningful depth within a shorter audit window.
For human-factor-dependent or operator-driven processes, use judgement-based haphazard sampling, which involves picking records without formal randomisation, avoiding patterns while providing a broad exploratory check.
When auditing a process with known previous nonconformities, apply statistical-based sampling with standards, intentionally selecting records linked to known weak points or prior NC trends to assess recurrence and control effectiveness (ISO 2859-1 Level II, AQL .65%).
For binary or pass/fail conformity checks—such as signatures, approvals, or IFU completeness—use statistical attribute sampling, where sample size is determined using ISO 2859-1 Level II, AQL 2.5% and each record is assessed as conforming or nonconforming.
For extremely large electronic datasets—such as audit trails, LIMS logs, or software event data—use statistical random sampling, selecting records using random number generators or automated randomisation tools to ensure unbiased coverage.
For large, stable populations within a mature QMS, use statistical sampling with standards such as ISO 2859-1 Level II, AQL 2.5%, determining the sample size directly from the AQL tables to achieve approximately 95% confidence.
For multi-site, multi-product, or multi-strata organisations, apply statistical stratified sampling, where the population is divided into strata such as site, product line, or shift, and samples are taken proportionally from each stratum to ensure representation.
For uniform datasets with no known clustering, use statistical systematic sampling, selecting every nth record (e.g., every 10th) after a random starting point to achieve efficient and consistent coverage across the dataset.
What Regulators Expect
FDA
Per the FDA’s Sampling Plans Guide:
Inspectors select tables based on required confidence (e.g., 95 % vs 99 %).
For populations ≤ 30, full review is recommended.
When a sample is used, the lot size, table, row, and sample size must be recorded.
Samples should be chosen at random whenever possible.
If objectionable conditions are found, inspectors may expand to 100 % review.
Notified Bodies
Use representative, risk-based sampling for both technical documentation and QMS audits in line with ISO 13485 and ISO 19011.
Expect manufacturers to mirror this logic in internal audits: define the population, risk, and rationale.
MDSAP
The selection of samples during audits in order to obtain evidence of conformity or nonconformity with MDSAP audit criteria can be either statistically based or judgement based. Judgement based sampling using audit trails from one task or process to inform the selection of samples in other tasks or processes is preferred. Where possible, auditors should select samples of records representing all participating MDSAP jurisdictions applicable to the audit.
The audit of the processes and the sampling should focus on the following (based on risk):
- new or modified designs and new products
- previously identified potential and existing nonconformities
- new or modified processes
- areas not sufficiently covered during the surveillance period.
Bottom Line
Regulators all operate on the same principle:
risk-based representativeness, documented rationale, and expansion if findings appear.
Key Takeaways
Sampling = statistical confidence, not guesswork.
Ensure to define and document your audit sampling approach within audit plans, and confirm whether the sampling plan was covered within the audit report.
Quote ISO 2859 and FDA Sampling Plans to show quantitative logic.
Always state population, rationale, and expansion criteria.