R: Statistical Computing experimental

R: Statistical Computing Statistical computing for people who must defend the number: wrangling, inference, regression, and reproducible analysis in R. R and the tidyverse. R and the tidyverse. Manipulate data frames with dplyr and tidyr verbs.. Explain the difference between wide and long (tidy) data, and construct a dplyr/tidyr pipeline that reshapes and summarises a stated dataset correctly.. Vectors and data frames, dplyr verbs, tidyr reshaping, The pipe Vectors and data frames Everything in R is a vector, even a single number: x <- 5 is a length-1 numeric vector. Vectors are atomic (one type: double, integer, character, logical) and recycle silently when lengths mismatch in arithmetic, which is a common source of quiet bugs. A data frame is a list of equal-length vectors bundled as columns, and a tibble (the tidyverse's data frame) refines this with saner printing, no silent partial matching, and no automatic string-to-factor conversion. Import real data with readr::read_csv() rather than base read.csv() so column types are reported explicitly and you can audit them before analysis begins. dplyr verbs Analysis is a small vocabulary applied repeatedly: filter() keeps rows matching a logical condition, select() keeps or drops columns, mutate() creates or changes columns, arrange() reorders rows, and summarise() collapses rows to a single value per group. group_by() makes any downstream verb operate within groups instead of across the whole table -- group_by(region) then summarise(mean_x = mean(x)) gives one mean per region, not one overall mean. Forgetting to ungroup() afterwards is a classic mistake that silently corrupts later steps. tidyr reshaping Tidy data means one variable per column, one observation per row, one value per cell. Real data rarely arrives this way. pivot_longer() gathers wide columns (for example, separate columns per year) into key-value pairs; pivot_wider() does the reverse, spreading a key column into several columns. separate() and unite() split or combine columns. Tidying is not cosmetic -- untidy data forces bespoke code for every analysis, while tidy data lets the same dplyr verbs and ggplot2 calls work unchanged on any dataset shaped this way. The pipe The pipe (%>% from magrittr, or base R's native |>) takes the expression on its left and feeds it as the first argument of the function on its right, letting you write a data transformation as a left-to-right sequence of verbs instead of nested function calls read inside-out. A pipeline such as filter, then group_by, then summarise reads as a sentence. Each step should do one clear thing; a pipeline that needs a comment explaining what it does as a whole is usually a sign it should be broken into named intermediate objects for a reader -- including the you of six months from now. Joins across tables Real analyses rarely live in one table: a customer table, an orders table, and a products table each hold a piece of the story, and dplyr's join functions combine them along a shared key. left_join() keeps every row of the left table and attaches matching columns from the right table, filling unmatched rows with NA, which is the right default when you want to know which customers never ordered anything. inner_join() keeps only rows that match in both tables, dropping unmatched rows entirely, which silently discards data if used where left_join() was intended, a common source of a report's totals mysteriously shrinking after a seemingly innocuous join is added to a pipeline. NA handling and its traps R's NA represents a genuinely missing value, and it propagates: any arithmetic or comparison involving NA typically returns NA rather than a guessed value, which is deliberate but easy to trip over. sum(x) on a vector containing NA returns NA unless na.rm = TRUE is passed explicitly, and a filter like df %>% filter(x > 5) silently drops rows where x is NA, since NA > 5 evaluates to NA rather than TRUE or FALSE, not to a value the filter treats as a match. A pipeline that reports a suspiciously round or suspiciously reduced row count is often hiding an unhandled NA somewhere upstream, which is why checking is.na() coverage on key columns before analysis is a standard first step, not an afterthought. Related CCI capabilities Computer Architecture (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/computer-architecture/). Optics Primer Series (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/optics/). Maths Refresher Series, Finance (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/maths-finance/). System Dynamics (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/system-dynamics/). CCI Lab: Run it, build with it, read the thinking, reuse the data. (https://www.cambridgecyberinternational.com/en/insights/lab/) Distributions and inference. Distributions and inference. Select a test and state its assumptions.. Explain what a p-value and a confidence interval do and do not mean, and apply the correct hypothesis test to a stated data scenario given its assumptions.. Sampling distributions, Confidence intervals, Hypothesis tests, p-values honestly Sampling distributions A statistic (a mean, a proportion, a slope) computed from a sample varies from sample to sample; the sampling distribution describes that variation across all possible samples of a given size. The Central Limit Theorem says that for reasonably large n, the sampling distribution of the mean is approximately normal regardless of the population's shape, with standard deviation equal to the population standard deviation divided by the square root of n -- the standard error. This is why larger samples give tighter estimates: the standard error shrinks with the square root of n, not with n itself, so quadrupling a sample only halves the uncertainty. Confidence intervals A 95% confidence interval is the result of a procedure that, if repeated across many samples, would produce intervals containing the true parameter about 95% of the time. It is a statement about the long-run behaviour of the procedure, not a 95% probability that this particular interval contains the true value -- the true value either is or is not in this interval, full stop. In R, t.test(x)$conf.int returns one for a mean; a wider interval signals more uncertainty, from a smaller sample, more variable data, or both. Always report the interval alongside the point estimate -- a mean without its uncertainty is half an answer. Hypothesis tests A hypothesis test starts from a null hypothesis (typically no effect or no difference) and asks how surprising the observed data would be if the null were true. Match the test to the data: one-sample or paired data suggests a t-test; two independent groups with roughly normal, equal-variance data suggest t.test(y ~ group); count data in categories suggests a chi-squared test; non-normal or ordinal data suggest a rank-based test like Wilcoxon. Every test carries assumptions -- independence, distributional shape, variance homogeneity -- and checking them (with a plot or Levene's test) is not optional decoration; violating them invalidates the p-value that follows. p-values honestly A p-value is the probability of seeing a result at least as extreme as the observed one, if the null hypothesis were exactly true. It is not the probability that the null is true, not the probability of replication, and not a measure of effect size -- a tiny effect in a huge sample can produce a minuscule p-value. A conventional 0.05 threshold is a convention, not a law of nature, and running many tests inflates the chance that at least one is 'significant' by chance alone, which is why multiple-comparison corrections (Bonferroni, Benjamini-Hochberg) matter when testing many hypotheses at once. Report the effect size and confidence interval alongside the p-value; the p-value alone answers a narrower question than most readers assume. Statistical power and sample size A test's power is the probability of correctly detecting a real effect of a given size, given the sample size and significance threshold used; a low-powered study can fail to find a real, meaningful effect not because the effect doesn't exist but because the sample was too small to detect it reliably. Power depends jointly on effect size, sample size, variance, and the chosen alpha level, and R's pwr package (or a manual power calculation) lets a researcher determine, before collecting data, how large a sample is needed to detect an effect of a stated minimum size with a stated probability. Skipping this step and simply collecting 'as much data as convenient' risks either wasting resources on an overpowered study or reaching an underpowered null result that cannot distinguish 'no effect' from 'not enough data to tell.' Effect sizes alongside significance A p-value answers whether an effect is distinguishable from zero given the data; an effect size answers how large that effect actually is, and the two questions are independent. Cohen's d for a difference in means, or r-squared for variance explained, translate a result into units a reader can judge for practical importance, independent of the sample size that produced the p-value. Reporting only 'p < 0.05, significant' without an accompanying effect size and confidence interval leaves a reader unable to tell a tiny, practically irrelevant effect measured in a huge sample from a large, actionable effect measured in a small one, even though both can produce an identical p-value threshold crossing. Related CCI capabilities Computer Architecture (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/computer-architecture/). Optics Primer Series (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/optics/). Maths Refresher Series, Finance (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/maths-finance/). System Dynamics (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/system-dynamics/). CCI Lab: Run it, build with it, read the thinking, reuse the data. (https://www.cambridgecyberinternational.com/en/insights/lab/) Regression. Regression. Fit and diagnose a linear model.. Explain what each regression diagnostic plot reveals about a model's assumptions, and evaluate whether a fitted model's diagnostics support trusting its coefficients.. Least squares, Diagnostics, Interpretation, Multiple regression Least squares Ordinary least squares finds the line (or hyperplane) that minimises the sum of squared residuals -- the squared vertical distances between observed and predicted values. In R, lm(y ~ x, data = df) fits it in one call. Squaring residuals rather than using absolute values makes the problem solvable in closed form and penalises large errors more than small ones, which is a modelling choice with consequences: a single extreme outlier can pull the fitted line noticeably toward itself. The fitted coefficients are the values that make the residuals, on average, uncorrelated with each predictor -- not a guarantee of causal accuracy, only a description of the best linear summary of the association in this sample. Diagnostics A regression's coefficients are only as trustworthy as its assumptions: linearity, independence of errors, constant variance (homoscedasticity), and approximately normal residuals for small-sample inference. plot(model) in R produces four diagnostic panels: Residuals vs Fitted (curved pattern signals non-linearity), Q-Q plot (departure from the line signals non-normal residuals), Scale-Location (fanning signals heteroscedasticity), and Residuals vs Leverage (points beyond Cook's distance contours are unduly influential). Diagnostics are read together, not cherry-picked; a model can look fine on one panel and be badly misspecified on another. Interpretation A slope coefficient is the expected change in the response for a one-unit increase in the predictor, holding other predictors in the model constant -- it is an association within the fitted model, not automatically a causal effect. The intercept is the predicted response when all predictors are zero, which is sometimes meaningless (nobody has an age of zero) and best left uninterpreted in those cases. R-squared is the proportion of variance in the response explained by the model; it always increases (or stays flat) when more predictors are added, which is why adjusted R-squared, penalising for the number of predictors, is the fairer number to compare across models. Multiple regression Adding predictors with lm(y ~ x1 + x2 + x3, data = df) lets you estimate each variable's association while holding the others fixed, which is essential when predictors are correlated with a shared confound. But correlated predictors (multicollinearity) inflate the standard errors of the coefficients involved, making individual estimates unstable even when the overall model fits well; the variance inflation factor (VIF) flags this. Interaction terms (x1:x2) let one predictor's slope depend on the level of another, and should be included deliberately, not by default -- an uninterpreted interaction term is worse than none at all. Transformations for non-linearity When a Residuals vs Fitted plot shows a clear curved pattern, the fix is often not a more complex model but a transformation of the existing variables. A log transformation of a right-skewed response variable frequently linearizes a relationship that looked curved on the raw scale, and it has the added benefit of making a coefficient interpretable as an approximate percentage change rather than a raw unit change. Polynomial terms (poly(x, 2)) can also capture curvature directly, but they extrapolate poorly outside the range of the observed data and are harder to interpret than a transformation chosen for a specific, understood reason, so a transformation grounded in the variable's actual behaviour is generally preferable to an arbitrary higher-degree polynomial fit purely to reduce residual curvature. Train/test evaluation beyond R-squared R-squared and adjusted R-squared describe how well a model fits the data it was trained on, but they say nothing about how well the model predicts new, unseen data, which is usually the actual question of interest. Splitting data into a training set (used to fit the model) and a held-out test set (used only to evaluate predictions) exposes overfitting that in-sample R-squared cannot: a model with many predictors can achieve a high training R-squared by fitting noise specific to that sample, then perform poorly on the test set because that noise does not generalize. Reporting only in-sample fit statistics for a model intended to predict future data, rather than a held-out evaluation metric such as test-set root mean squared error (RMSE), is a common way an analysis overstates how useful the fitted model actually is. Related CCI capabilities Computer Architecture (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/computer-architecture/). Optics Primer Series (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/optics/). Maths Refresher Series, Finance (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/maths-finance/). System Dynamics (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/system-dynamics/). CCI Lab: Run it, build with it, read the thinking, reuse the data. (https://www.cambridgecyberinternational.com/en/insights/lab/) Reproducible analysis. Reproducible analysis. Produce a reproducible, visualised report.. Explain what makes an analysis reproducible, and construct a literate R Markdown or Quarto report whose figures are traceable to their generating code and data.. Literate analysis, ggplot2, Provenance of each figure, Pitfalls Literate analysis Literate analysis interleaves narrative, code, and output in one document -- an R Markdown or Quarto file -- so the report and the computation that produced it cannot drift apart. Every number a reader sees traces back to a code chunk they can re-run; there is no separate 'analysis' happening in an untracked console or spreadsheet. knitr::knit() or quarto render executes the chunks fresh and inserts their output, meaning a report that fails to knit is a report you cannot trust, because it means the narrative has outrun the code. Setting a seed with set.seed() before any random step (simulation, bootstrap, train/test split) is required for the document to reproduce identical output on every run. ggplot2 ggplot2 builds a plot in layers from a grammar of graphics: ggplot(data, aes(x, y)) declares the data and the mapping from variables to visual properties, and geoms (geom_point(), geom_line(), geom_boxplot()) add the visual representation. facet_wrap() and facet_grid() split one plot into a panel per category instead of forcing colour to carry too much information at once. Every plot destined for a report needs axis labels with units, a title that states the finding rather than just the variable names, and a caption noting the data source -- a chart that requires the surrounding paragraph to be intelligible has failed as a chart. Provenance of each figure A defensible figure has a traceable chain: raw data file (with a checksum or version noted), the exact filtering and transformation steps applied, the code chunk that produced the plot, and the session information (R version, package versions) at the time it ran. sessionInfo() or the renv lockfile pins package versions so a figure regenerated next year uses the same computational environment. When someone challenges a number, the answer should never be 'let me re-derive that' -- it should be 'here is the exact chunk, on line N, using this data snapshot.' Pitfalls Common failures include: leaving hardcoded numbers in prose that no longer match updated code output; filtering or excluding rows silently mid-script without comment; using library(dplyr) without noting the version, so a future run behaves differently after an update; and copy-pasting a plot image instead of embedding the generating chunk, breaking the link between figure and code. The discipline is straightforward even when tedious: run the whole document from a clean environment before submitting it, and if it doesn't reproduce cleanly, the analysis isn't finished. Parameterized reports A single R Markdown or Quarto document can be rendered multiple times with different inputs by declaring parameters at the top of the document (params: region: "North") and referencing params$region inside the analysis, rather than hardcoding a specific value and duplicating the file for every region or time period a report needs to cover. Rendering the same parameterized template once per region, driven by a small loop over region names, guarantees every regional report used identical logic, differing only in the input data slice, which eliminates an entire class of bug where a manually copy-pasted report for one region quietly retains a filter or calculation left over from the region it was copied from. Version control as a reproducibility layer A reproducible report's code and data provenance still depends on knowing which version of the code produced a given result, which is what a version control system such as Git provides on top of set.seed() and sessionInfo(). Committing the analysis script alongside the rendered report, and recording the commit hash the report was generated from, closes the last gap in the provenance chain: even a report that knits cleanly today, from pinned package versions, cannot be traced back to the exact code that generated a past figure without knowing which commit was checked out at the time. Tagging or committing at the point a report is finalized, rather than only when convenient, is what makes 're-run the exact analysis from six months ago' an achievable request rather than an archaeology project. Related CCI capabilities Computer Architecture (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/computer-architecture/). Optics Primer Series (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/optics/). Maths Refresher Series, Finance (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/maths-finance/). System Dynamics (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/system-dynamics/). CCI Lab: Run it, build with it, read the thinking, reuse the data. (https://www.cambridgecyberinternational.com/en/insights/lab/)