Quantitative Research Methods experimental

Quantitative Research Methods Doing quantitative research that survives scrutiny: sourcing, dataset construction, signal extraction and honest reporting. Sources and evidence. Sources and evidence. Vet a primary source and record provenance.. Explain the difference between primary and secondary sourcing, and apply a vetting checklist to assess whether a given source can support a stated claim.. Primary vs secondary, Source vetting, Provenance records, Bias in sourcing Primary vs secondary A primary source is a record generated at the point of observation: a sensor log, a signed survey response, a court transcript, a lab notebook entry, an API pull timestamped at capture. A secondary source restates, aggregates, or interprets primary material: a review article, a news summary, a dashboard someone else built from data you cannot see. Quantitative work leans on primary sources wherever possible because every layer of restatement is a place where units get silently converted, caveats get dropped, and rounding compounds. When only secondary sources are available, the researcher's job is to chase the citation chain back as far as it goes and to note explicitly where the chain goes cold. A source that cannot name where its numbers came from is not evidence, it is an assertion wearing evidence's clothes. Source vetting Vetting a source means asking, before you use a single number from it, who collected this, with what instrument or method, under what incentive, and what would have to be true for this number to be wrong. Check the collection method against the claim being made, since a convenience sample cannot support a population-level claim. Check the incentives of whoever produced the data, such as a vendor benchmarking its own product or an agency measuring its own program's success. Check internal consistency, meaning whether subtotals sum to the stated total and units match across tables. Check external consistency, meaning whether the number roughly agrees with independent estimates. A source that fails vetting is not automatically discarded, but it must be downgraded and its limitations carried forward into every downstream claim. Provenance records Provenance is the documented chain of custody for a datum: where it came from, when it was captured, who touched it, and what transformations were applied before it reached you. A provenance record should let a stranger reconstruct the datum's history without asking you anything. In practice this means recording, at minimum, the original source identifier, the retrieval date, the exact query or extraction method, and a hash or checksum where feasible. Provenance is not bureaucratic overhead; it is the mechanism that turns a number into a falsifiable claim. Work that cannot produce provenance on demand cannot be audited, and work that cannot be audited cannot be trusted at Level 4. Bias in sourcing Every corpus of sources has a shape determined by what was easy to collect, what someone wanted collected, and what survived to be collectable. Survivorship bias hides the cases that dropped out. Selection bias skews toward whoever was reachable, willing, or funded. Publication bias favors positive, novel, or dramatic findings over null results. Availability bias favors digitized, English-language, recent material over everything else. None of these biases are solved by having more sources of the same kind; they are addressed by actively seeking disconfirming and out-of-band evidence, and by stating in the write-up exactly whose voice or data is systematically absent from the evidence base. Corroboration and triangulation A single source, however well vetted, is one witness with one vantage point; triangulation means seeking a second and third source that would have to be wrong in the same specific way for the claim to be false, ideally sources with different collection methods, different incentives, or different institutional origins, since sources sharing a method or an incentive can fail together in a way that looks like independent confirmation but is not. A claim corroborated by two sources with genuinely different failure modes, for example an administrative record and an independent survey, is meaningfully stronger evidence than the same claim repeated by three outlets that all cite the same original press release. Documenting source limitations Every source has limitations, and naming them explicitly is not a weakness in a write-up, it is what separates a calibrated claim from an overconfident one. A limitation section should state the known gaps in coverage, the population the source does and does not represent, the time window the data covers, and any known incentive the source's producer has to shade the numbers a particular way. Downstream readers who inherit a claim without its limitations will apply it more broadly than the evidence supports, which is exactly how a narrow finding from one population quietly becomes treated as a general truth several citations later. Related CCI capabilities Computer Architecture (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/computer-architecture/). Optics Primer Series (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/optics/). Maths Refresher Series, Finance (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/maths-finance/). System Dynamics (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/system-dynamics/). CCI Lab: Run it, build with it, read the thinking, reuse the data. (https://www.cambridgecyberinternational.com/en/insights/lab/) Dataset construction. Dataset construction. Build a traceable dataset.. Explain why implicit units and undisclosed cleaning decisions corrupt a dataset, and construct a versioned, traceable schema for a given data-collection scenario.. Schema and units, Cleaning decisions, Traceability, Versioning Schema and units A schema is a contract: it fixes the name, type, unit, and allowed range of every field before data starts landing in it. Quantitative research fails quietly and often when units are implicit. A column called weight that silently mixes kilograms and pounds, a date field that mixes ISO 8601 with locale-dependent day and month ordering, a rate that is sometimes per year and sometimes per quarter: each of these produces numbers that compute cleanly and mean nothing. A defensible schema states the unit in the field name or metadata, states the valid range, states what a null means, whether missing, not applicable, or genuinely zero, and states the grain of each row, such as one row per event, per subject, or per subject-day. Changing any of these after data collection has started is a breaking change and must be versioned, not silently patched. Cleaning decisions Cleaning is not a neutral, mechanical step; every cleaning decision is a modeling choice and must be logged as one. Deciding to drop rows with an out-of-range value, to cap outliers at a percentile, to impute a missing field with a mean, or to merge two categories that were coded inconsistently: each of these changes the data-generating story being told. At Level 4 no cleaning step is applied silently. Each transformation is written down with its rationale, its effect size, meaning how many rows or what proportion of values it touched, and a note on whether it was decided before or after looking at the analysis results. Cleaning decisions made after seeing preliminary results, and chosen because they produce a more favorable outcome, are a form of researcher degrees of freedom and must be disclosed as such, not hidden inside a generic cleaning step. Traceability A traceable dataset lets any single cell be traced back to the exact raw record it came from and the exact sequence of transformations applied to reach its current value. This is usually implemented as a lineage column or companion log: a cell identifier, the source record identifier, an ordered list of transformation steps with parameters, and a timestamp for each step. Traceability is what allows an auditor, a peer reviewer, or a future version of yourself to answer the question, why does this cell say 42, without re-running the entire pipeline from memory. Datasets without traceability accumulate silent forks, where two people's copies diverge and nobody can say which is authoritative. Versioning A dataset is not a single artifact; it is a sequence of artifacts, each one an immutable snapshot with a version identifier, a changelog entry, and a reason for the change. Versioning means never overwriting a prior release in place. If a cleaning rule changes, a new version is cut, the old version remains retrievable, and any analysis built on the old version records which version it used. This discipline is what makes replication possible: a claim tied to dataset v3 can be checked against exactly the rows and transformations that existed at v3, independent of whatever the dataset has become since. Codebooks and metadata A codebook is the human-readable companion to a schema: for every field, it states what the field means in plain language, how each coded value maps to its real-world meaning, and any known quirks in how the field was historically populated, such as a coding convention that changed partway through data collection. Without a codebook, a schema's formal type and range constraints tell a future analyst what values are legal but not what they mean, which is exactly how a category originally meant to capture one thing gets silently reused to capture something adjacent but different, corrupting every downstream analysis that assumes the original meaning still holds. Reproducible pipelines A reproducible pipeline runs from raw input to final output through a recorded, re-executable sequence of steps, rather than a series of manual, undocumented actions in a spreadsheet or notebook that cannot be replayed exactly. This matters because a pipeline that exists only as a memory of what someone clicked cannot be audited, cannot be handed to a collaborator, and cannot be re-run against a corrected input when an upstream error is discovered. Version-controlling the pipeline code alongside the dataset versions it produces closes the loop: a given dataset version is not just a file, it is the output of a specific, inspectable, re-runnable set of transformation steps. Related CCI capabilities Computer Architecture (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/computer-architecture/). Optics Primer Series (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/optics/). Maths Refresher Series, Finance (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/maths-finance/). System Dynamics (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/system-dynamics/). CCI Lab: Run it, build with it, read the thinking, reuse the data. (https://www.cambridgecyberinternational.com/en/insights/lab/) Signal from noise. Signal from noise. Separate signal from noise and missingness.. Distinguish missingness mechanisms and noise models, and evaluate whether a given estimator or imputation choice is appropriate for a stated missingness regime.. Noise models, Missing data, Robust estimation, Overfitting Noise models Before extracting a signal from data, a researcher must state a model for what the noise looks like, because the correct estimator depends on it. Additive Gaussian noise, heavy-tailed noise with occasional large excursions, multiplicative noise that scales with the signal magnitude, and structured noise correlated across observations, such as sensor drift or repeated-measures autocorrelation, all call for different handling. Assuming Gaussian noise by default because that is what the textbook estimator expects, without checking the residual distribution, is one of the most common silent errors in applied quantitative work. A stated noise model is falsifiable: it predicts a residual shape that can be checked against the data, and it should be revised when the check fails. Missing data Data go missing for reasons, and those reasons determine what can be safely done about it. Missing completely at random means the missingness carries no information and simple deletion is defensible. Missing at random means missingness depends on observed variables and can be modeled or imputed conditional on those variables. Missing not at random means missingness depends on the unobserved value itself, such as high earners declining to report income, and this is the dangerous case, because no imputation scheme recovers the true distribution without an explicit model of the missingness mechanism. The obligation on the researcher is to state which regime is assumed, justify it, and show what changes under a less convenient assumption. Robust estimation Robust estimators are chosen to limit the influence any single observation, or small cluster of observations, can have on the final estimate. A mean can be dragged arbitrarily far by one extreme value; a median cannot. Trimmed means, M-estimators, and rank-based methods trade a small amount of efficiency under ideal conditions for a large amount of protection against contamination and model misspecification. Robustness is not an excuse to discard inconvenient data; it is a principled acknowledgment that real data contains a mixture of the process being studied and unrelated contamination, and the estimator should reflect that mixture honestly rather than pretending the data are pristine. Overfitting Overfitting occurs when a model captures the idiosyncrasies of the specific sample rather than the underlying process, producing excellent in-sample fit and poor out-of-sample performance. Its signatures include an estimate that changes drastically when a handful of data points are removed, an in-sample error much lower than a held-out or cross-validated error, and a model whose complexity was chosen after looking at how well earlier, simpler versions performed on the same data. Guarding against it requires a held-out test set decided before analysis, complexity penalties or regularization, and reporting cross-validated performance as the headline number rather than the fit statistic from the full sample. Sensitivity analysis Sensitivity analysis asks how much a conclusion changes as an assumption is varied across its plausible range, rather than reporting a single result under one fixed assumption as if it were the only possible one. For an unmeasured confounder, this typically means computing how strong the confounder would need to be, in its association with both the treatment and the outcome, to fully explain away the observed effect, and then judging whether a confounder of that strength is plausible given domain knowledge. A conclusion that survives a wide range of reasonable assumptions is far more trustworthy than one that flips as soon as a single debatable choice, such as an outlier threshold or an imputation method, is changed. Multiple comparisons Testing many hypotheses against the same dataset inflates the chance that at least one test crosses a significance threshold purely by chance, even if every individual null hypothesis is true; running twenty independent tests at a 5% threshold yields roughly a 64% chance of at least one false positive. Corrections such as the Bonferroni adjustment or control of the false discovery rate exist specifically to address this, but the more durable fix is discipline at the design stage: pre-specifying which comparisons are primary before the data are seen, and clearly labeling any additional comparison as exploratory rather than confirmatory in the final report. Related CCI capabilities Computer Architecture (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/computer-architecture/). Optics Primer Series (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/optics/). Maths Refresher Series, Finance (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/maths-finance/). System Dynamics (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/system-dynamics/). CCI Lab: Run it, build with it, read the thinking, reuse the data. (https://www.cambridgecyberinternational.com/en/insights/lab/) Validity and reporting. Validity and reporting. Design against validity threats and report honestly.. Distinguish internal from external validity threats, and design a pre-registration plan that closes off after-the-fact analytic choices for a stated hypothesis.. Internal and external validity, Confounding, Pre-registration mindset, Honest reporting Internal and external validity Internal validity asks whether the study design actually isolates the causal effect it claims to measure, within the sample studied. Threats include confounding, selection into treatment, attrition that differs by condition, and measurement that changes over the course of the study. External validity asks whether a finding that holds internally generalizes beyond the specific sample, setting, and time period studied. A tightly controlled lab result can be perfectly internally valid and nearly worthless externally if the population, context, or incentives of the real setting differ sharply from the study conditions. Every study report should state both, separately, rather than treating a strong internal result as automatically generalizable. Confounding A confounder is a variable that influences both the presumed cause and the presumed effect, creating an association that has no direct causal link. Age confounds many health and behavioral associations; regional wealth confounds many social and infrastructure associations. Confounding is addressed through randomization when feasible, through statistical adjustment when the confounder is measured, and through design features such as instrumental variables, difference-in-differences, or regression discontinuity when randomization is not feasible. The dangerous case is an unmeasured confounder, which cannot be adjusted away and can only be reasoned about through sensitivity analysis and domain knowledge about how strong such a confounder would need to be to explain away the observed effect. Pre-registration mindset Pre-registration means committing, before seeing the outcome data, to the hypothesis, the primary analysis method, the sample size or stopping rule, and the criteria for what counts as a positive result. The value of this discipline is not the paperwork; it is that it closes off the large space of after-the-fact choices, such as which subgroup to report, which covariates to include, or which outlier rule to apply, that can turn a null result into an apparently significant one purely by search. Even where formal pre-registration is not required, adopting the mindset, meaning writing an analysis plan before looking at the results and disclosing any deviation from it, is the practical defense against p-hacking and hypothesizing after results are known. Honest reporting Honest reporting discloses what was tried and did not work, not only what worked; states effect sizes and uncertainty intervals rather than only significance thresholds; distinguishes pre-specified analyses from exploratory ones; and reports the exact sample, exclusions, and their justification. A results section that only shows the analysis that produced the cleanest story, without acknowledging the other analyses attempted, is not lying in any single sentence, but it is dishonest in aggregate. The standard to hold work to is whether a skeptical peer, given the full record, would reach the same confidence in the conclusion. Replication A replication attempts to reproduce a prior finding using new data collected under the same or a closely matched design, distinct from reproducibility, which merely re-runs the original analysis on the original data. A successful replication in a new sample is stronger evidence for a claim's robustness than any amount of re-analysis of the original dataset, because it tests whether the effect exists in the world, not just whether the original computation was performed correctly. A field that treats replication as low-status, unpublishable work creates exactly the incentive structure that let the replication crisis go undetected for as long as it did. Registered reports A registered report is a publication format in which a study's introduction, hypotheses, and methods are peer-reviewed and provisionally accepted before the data are collected or analyzed, with the eventual publication decision based on whether the pre-registered plan was followed rather than on whether the result was positive or novel. This directly removes the incentive to selectively report only the analyses that produced a favorable result, since the outlet has already committed to publishing the outcome regardless of its direction. Registered reports have shown markedly higher rates of null results reaching publication compared to the traditional literature, which is itself evidence of how strong publication bias is under the conventional model. Related CCI capabilities Computer Architecture (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/computer-architecture/). Optics Primer Series (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/optics/). Maths Refresher Series, Finance (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/maths-finance/). System Dynamics (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/system-dynamics/). CCI Lab: Run it, build with it, read the thinking, reuse the data. (https://www.cambridgecyberinternational.com/en/insights/lab/)