Did the programme cause the change? Designs and field data that can answer honestly
Bharat Survey is an impact evaluation agency in India for NGOs, CSR foundations, social enterprises, government programmes and academic research teams. We help set out the theory of change, choose a design that can support a causal claim, and collect the household, beneficiary and facility data ourselves with GPS-verified field teams. Where the design cannot prove impact, the report says so plainly.
- Theory of change and logframe before tools
- Counterfactual designs matched to budget and timing
- Field partner for randomised trials led by researchers
- Quantitative surveys plus qualitative interviews
- In areaInside approx. boundary
- Accuracy±12 m
- Time09:42 · fresh fix
- Mock GPSNot detected
- Interview length11 min
An impact evaluation compares what happened with a programme against a credible estimate of what would have happened without it. Everything else, however useful, is monitoring or outcome tracking.
The hard part is the counterfactual. Participants often differ from non-participants before the programme even starts: they are closer to the road, better connected or more motivated. Comparing the two groups afterwards mixes those differences with the programme's effect. A proper design removes or measures that bias, through random assignment, a phased roll-out, a well-matched comparison group or a before-and-after comparison with a control.
We are direct about what each design can claim. A before-and-after survey of beneficiaries alone shows change, not impact. If that is all the budget allows, we will still run it well, and we will label it accurately in the report.
Where evaluation fits in a programme cycle
The best time to plan an impact evaluation is before the programme starts, when a baseline can be collected and a comparison group can be identified. Evaluations commissioned after the end usually have to rely on recall and weaker comparisons.
Typical deliverables
- Theory of change and evaluation matrix
- Design and sample size note
- Baseline and end-line datasets
- Qualitative interview summaries
- Impact estimates with confidence intervals
- Report and board presentation
Tools written without a theory of change measure whatever is easy. These two steps decide what is worth measuring.
Theory of change workshop facilitation
A half-day or full-day session with programme staff, and ideally community representatives, to map activities to outputs, outcomes and long-term impact, and to write down the assumptions that must hold for each step to work.
- Causal pathway diagram
- Assumptions and risks listed
- Evaluation questions agreed
Logical framework indicators design
Each outcome gets indicators that are specific, measurable in the field, and comparable between rounds, with a clear data source, frequency and responsibility.
- Indicator definitions
- Means of verification
- Baseline and target columns
- Comes regularly
- Comes sometimes
- Rarely / never
- Don't know
Evaluation matrix
Every evaluation question is linked to indicators, data sources, sample and analysis method, so gaps show up before fieldwork rather than in the draft report.
- Question-by-question plan
- Data source per indicator
- Analysis plan
The right design depends on how the programme was rolled out, not on preference.
| Design | Works when | What it can claim | Main risk |
|---|---|---|---|
| Randomised assignment | Eligible units can be randomly assigned before roll-out | Causal impact, with strongest internal validity | Ethical and practical limits on who is left out |
| Phased roll-out | Programme reaches areas in stages | Impact while later areas wait | Later areas may change in the meantime |
| Difference-in-differences | Baseline and end-line exist for both groups | Impact if trends would have been parallel | Groups on different trends beforehand |
| Matched comparison | Comparable non-participants can be found | Impact net of observed differences | Unobserved differences remain |
| Before and after only | No comparison group is possible | Change over time, not impact | Other causes of change |
Design descriptions follow the World Bank's Impact Evaluation in Practice (2nd edition).
Our position on attribution
Field operations for researcher-led trials
Many trials in India are designed by university or institute researchers who need a field partner to list households, run baseline and end-line surveys, and track attrition. We follow the pre-analysis plan and protocol, keep treatment status away from interviewers where possible, and report attrition by arm.
Randomisation itself is normally done by the principal investigator's team. We implement the assignment and document any deviation from it.
- Listing and consent
- Blinded interviewer assignments where feasible
- Attrition and compliance reports by arm
- Ethics committee documentation support
Tracking the same households
Panel studies stand or fall on re-contact. Households get stable IDs, GPS points and consented contact details, so end-line teams can find them years later.
- Stable household IDs
- GPS for re-visit
- Re-contact rate reported
Surveys tell you how much changed. Interviews tell you why, and for whom it did not.
Quantitative strand: surveys and data
- Household, beneficiary and facility surveys
- Sample sized for the expected effect
- Impact estimates with confidence intervals
- Sub-group analysis where sample allows
Qualitative strand: interviews and groups
- In-depth interviews with participants and staff
- Focus groups with women, youth or farmers
- Case studies of what did and did not work
- Themes coded and linked to survey findings
Funders usually specify a results framework. We build logframes and indicators in the format the funder uses and design data collection to report against it. We do not claim relationships with any funder; the notes below describe how their evaluations are commonly set up.
UNICEF evaluation consultant India requirements
UNICEF-supported evaluations generally follow UNEG norms and standards, with ethics review and attention to children and women's safeguarding. Field partners are expected to handle consent, child protection and sensitive modules carefully.
World Bank survey firm India expectations
World Bank-financed projects hire survey firms through the borrower under the Bank's procurement regulations. Terms usually ask for CAPI data collection, detailed field protocols and raw data delivery, in line with published guidance such as the DIME Wiki.
Gates Foundation evaluation partner India considerations
Foundation-funded evaluations often combine large household surveys with programme data. Grantees typically need a field partner who can deliver clean, well-documented data on tight timelines and share it under the grant's data policy.
Multiple indicator cluster survey partner methods
UNICEF's Multiple Indicator Cluster Surveys (MICS) publish standard questionnaires and manuals. Where a programme wants comparability with MICS-style indicators on child health, nutrition or education, we can adapt those modules with attribution. That is not the same as being an official MICS implementing partner, which we are not.
Not the right fit if
- The programme is too new or too small for an effect to be detectable
- No baseline or comparison is possible and you need a causal claim
- The answer is needed before outcomes could have changed
- You want a report that confirms results in advance
A good fit if
- Programmes planning a baseline before roll-out
- Researchers needing a field partner for a trial
- CSR teams answering board questions on impact
- Studies that need both survey numbers and field stories
NGO and CSR solutions
Surveys and software for development programmes.
Learn moreImpact evaluation methods guide
Designs explained in more depth.
Learn moreBaseline and end-line survey
Before-and-after data collection.
Learn moreMonitoring and evaluation
Ongoing M&E support.
Learn moreNGO data collection app
Run your own field surveys.
Learn moreNGO M&E software
Track indicators across programmes.
Learn moreStraight answers to the questions people ask most about this topic.
What is the difference between impact evaluation and impact assessment?
The terms are often used interchangeably in India. Strictly, an impact evaluation estimates causal effect against a counterfactual, while impact assessment is broader and can include descriptive before-and-after studies. We state in the proposal which one a given design supports.
How large a sample does an impact evaluation need?
It depends on the size of effect you expect, variation in the outcome, how clustered the sample is and how many sub-groups need results. We do a power calculation and share it before quoting, because an underpowered study cannot detect a real effect.
Can you evaluate a programme that has already ended?
Yes, with limits. Without a baseline we rely on recall, programme records and comparison areas, which weakens causal claims. We will explain what the available data can support and label findings accordingly.
Do you design RCTs yourselves?
We support design discussions, but most randomised trials we would work on are led by academic researchers who own the design and randomisation. Our role is listing, surveys, tracking and data quality in the field.
Share the programme description, locations, timeline and what your board or funder needs to know. We will propose a design and a data collection plan.